Save your company's ass - Launch a static "emergency mode" site on S3
We were just featured in an email that was sent to a million people. It went great but we were a little concerned about what would happen if we had an unexpected problem in our application at the last minute. If our site went down on the day that thousands of people were all trying to get to RockThePost, we would have squandered the opportunity and I would have have been completely embarrassed.
We decided to launch our "hot spare" website. It's a special lite version of RockThePost which allows you to see some of the startups we are currently featuring. You can even sign up.
To get an idea of how it works, first take a look at our regular website:
Now, take a look at our emergency "holy shit we're in trouble" version of RockThePost:
http://maintenance.rockthepost.com/
Just in case the RockThePost has massive problems, my recovery plan is as follows:
1) I remove the normal web servers from the load balancer
2) I launch my "emergency" Apache redirect server and put it in the pool
My emergency web server simply mod_rewrites all of the requests to http://maintenance.rockthepost.com/
Our maintenance website is hosted directly on S3 as a static site. As far as I'm concerned, S3 static file website serving is completely indestructible. I only need one bland Apache server to bump requests over to it.
We knew the maintenance site needed to be able to accomplish two things: We wanted to allow people to continue to register for our service and also see current ventures. In order to accomplish this, we signed up with http://www.wufoo.com/ to do the user data collection form processing and we wrote a periodic Jenkins job to rewrite the ventures which are displayed on the emergency maintenance site.
Now, even in the worst of cases, I have some realistic hope of keeping our brand on the internet and continuing to collect user registrations even if we have a holy-shit-we-are-down day.
Written by Jonathan Block
Related protips
15 Responses
Great idea! However, S3 isn't built to serve static assets. You should probably be using CloudFront instead.
dlo - take a look here:
http://docs.aws.amazon.com/AmazonS3/latest/dev/WebsiteHosting.html
Those docs were written before Amazon CloudFront supported root domain hosting. Check this link out:
That is actually not correct. Static website CAN be served from S3 and it's been that way for a while...
Wouldn't it be easier to setup an "origin-pull" scenario via CloudFront?
Use Amazon Route53 Failover so everything is automated (you don't have to remove the apache from the load balancer pool manually)
That works as long as Amazon itself doesn't take a nosedive. Maybe host it somewhere else and deal with the redirect using DNS?
What how many and what size servers did you have running for your million person email blast? Just curious
Pineappleio - We only needed 1 c1.xlarge server but I ran 5 of them just in case. See my other post on how I set it up: https://coderwall.com/p/__z9ia
navigatetech - Agreed. In our case, it is my opinion that that we're too early to use a strategy where we're on multiple hosts.
This is great advice! Set up your static files on S3 and serve them with Cloudfront. I have had this sort of backup going for quite a while, and since I already store my images this way, it's a natural transition. The cost of S3+cloudfront (a few bucks a month for my blog) is well worth it.
Thanks for the tips!
If you are really concerned with an entire region going down ( unlikely ), run that maintenance page in a different region, and use Route 53's failover abilities to help do that failover for you! http://aws.typepad.com/aws/2013/02/create-a-backup-website-using-route-53-dns-failover-and-s3-website-hosting.html
Great idea (we also run ours cross-region/cross-datacenter for redundancy), but you are basically losing the scale advantages of S3 with the Apache server in the critical path (I believe Apache or nginx can serve static file requests basically as quickly as 302s, though I haven't run the benchmarks myself).
You can eliminate the LB, Apache server, and redirect by just using DNS CNAMEs if you change your "common path" URL (eg. the one in your email) to be http://www.rockthepost.com (you still need the LB & webserver for http://rockthepost.com since you can't reasonably CNAME root records). Then on failure, just point the DNS for www to your S3 site.
I've visited sites that used CloudFlare, & when they were down CF served the cached site. Doesn't seem to be listed specifically https://www.cloudflare.com/overview
Yeah, that CloudFlare feature is really great. It's called "Always Online": https://www.cloudflare.com/always-online
