Last Updated: January 28, 2019
·
49.43K
· blockjon

Save your company's ass - Launch a static "emergency mode" site on S3

We were just featured in an email that was sent to a million people. It went great but we were a little concerned about what would happen if we had an unexpected problem in our application at the last minute. If our site went down on the day that thousands of people were all trying to get to RockThePost, we would have squandered the opportunity and I would have have been completely embarrassed.

We decided to launch our "hot spare" website. It's a special lite version of RockThePost which allows you to see some of the startups we are currently featuring. You can even sign up.

To get an idea of how it works, first take a look at our regular website:

https://rockthepost.com/

Now, take a look at our emergency "holy shit we're in trouble" version of RockThePost:

http://maintenance.rockthepost.com/

Just in case the RockThePost has massive problems, my recovery plan is as follows:

1) I remove the normal web servers from the load balancer

2) I launch my "emergency" Apache redirect server and put it in the pool

My emergency web server simply mod_rewrites all of the requests to http://maintenance.rockthepost.com/

Our maintenance website is hosted directly on S3 as a static site. As far as I'm concerned, S3 static file website serving is completely indestructible. I only need one bland Apache server to bump requests over to it.

We knew the maintenance site needed to be able to accomplish two things: We wanted to allow people to continue to register for our service and also see current ventures. In order to accomplish this, we signed up with http://www.wufoo.com/ to do the user data collection form processing and we wrote a periodic Jenkins job to rewrite the ventures which are displayed on the emergency maintenance site.

Now, even in the worst of cases, I have some realistic hope of keeping our brand on the internet and continuing to collect user registrations even if we have a holy-shit-we-are-down day.

15 Responses
Add your response

Great idea! However, S3 isn't built to serve static assets. You should probably be using CloudFront instead.

over 1 year ago ·

Those docs were written before Amazon CloudFront supported root domain hosting. Check this link out:

http://aws.typepad.com/aws/2013/06/custom-ssl-domain-names-root-domain-hosting-for-amazon-cloudfront.html

over 1 year ago ·

That is actually not correct. Static website CAN be served from S3 and it's been that way for a while...

over 1 year ago ·

Wouldn't it be easier to setup an "origin-pull" scenario via CloudFront?

over 1 year ago ·

Use Amazon Route53 Failover so everything is automated (you don't have to remove the apache from the load balancer pool manually)

over 1 year ago ·

That works as long as Amazon itself doesn't take a nosedive. Maybe host it somewhere else and deal with the redirect using DNS?

over 1 year ago ·

What how many and what size servers did you have running for your million person email blast? Just curious

over 1 year ago ·

Pineappleio - We only needed 1 c1.xlarge server but I ran 5 of them just in case. See my other post on how I set it up: https://coderwall.com/p/__z9ia

over 1 year ago ·

navigatetech - Agreed. In our case, it is my opinion that that we're too early to use a strategy where we're on multiple hosts.

over 1 year ago ·

This is great advice! Set up your static files on S3 and serve them with Cloudfront. I have had this sort of backup going for quite a while, and since I already store my images this way, it's a natural transition. The cost of S3+cloudfront (a few bucks a month for my blog) is well worth it.

Thanks for the tips!

over 1 year ago ·

If you are really concerned with an entire region going down ( unlikely ), run that maintenance page in a different region, and use Route 53's failover abilities to help do that failover for you! http://aws.typepad.com/aws/2013/02/create-a-backup-website-using-route-53-dns-failover-and-s3-website-hosting.html

over 1 year ago ·

Great idea (we also run ours cross-region/cross-datacenter for redundancy), but you are basically losing the scale advantages of S3 with the Apache server in the critical path (I believe Apache or nginx can serve static file requests basically as quickly as 302s, though I haven't run the benchmarks myself).

You can eliminate the LB, Apache server, and redirect by just using DNS CNAMEs if you change your "common path" URL (eg. the one in your email) to be http://www.rockthepost.com (you still need the LB & webserver for http://rockthepost.com since you can't reasonably CNAME root records). Then on failure, just point the DNS for www to your S3 site.

over 1 year ago ·

I've visited sites that used CloudFlare, & when they were down CF served the cached site. Doesn't seem to be listed specifically https://www.cloudflare.com/overview

over 1 year ago ·

Yeah, that CloudFlare feature is really great. It's called "Always Online": https://www.cloudflare.com/always-online

over 1 year ago ·