68ozza
43.86K
· July 2013 ·
Jackjon

Save your company's ass - Launch a static "emergency mode" site on S3

We were just featured in an email that was sent to a million people. It went great but we were a little concerned about what would happen if we had an unexpected problem in our application at the last minute. If our site went down on the day that thousands of people were all trying to get to RockThePost, we would have squandered the opportunity and I would have have been completely embarrassed.

We decided to launch our "hot spare" website. It's a special lite version of RockThePost which allows you to see some of the startups we are currently featuring. You can even sign up.

To get an idea of how it works, first take a look at our regular website:

https://rockthepost.com/

Now, take a look at our emergency "holy shit we're in trouble" version of RockThePost:

http://maintenance.rockthepost.com/

Just in case the RockThePost has massive problems, my recovery plan is as follows:

1) I remove the normal web servers from the load balancer

2) I launch my "emergency" Apache redirect server and put it in the pool

My emergency web server simply mod_rewrites all of the requests to http://maintenance.rockthepost.com/

Our maintenance website is hosted directly on S3 as a static site. As far as I'm concerned, S3 static file website serving is completely indestructible. I only need one bland Apache server to bump requests over to it.

We knew the maintenance site needed to be able to accomplish two things: We wanted to allow people to continue to register for our service and also see current ventures. In order to accomplish this, we signed up with http://www.wufoo.com/ to do the user data collection form processing and we wrote a periodic Jenkins job to rewrite the ventures which are displayed on the emergency maintenance site.

Now, even in the worst of cases, I have some realistic hope of keeping our brand on the internet and continuing to collect user registrations even if we have a holy-shit-we-are-down day.

Sign in or sign up to add your response.

15 Responses

7810
Bd0391d7225d10455d35de0fb5485bea

Great idea! However, S3 isn't built to serve static assets. You should probably be using CloudFront instead.

over 1 year ago ·
7811
Jackjon
7813
Bd0391d7225d10455d35de0fb5485bea

Those docs were written before Amazon CloudFront supported root domain hosting. Check this link out:

http://aws.typepad.com/aws/2013/06/custom-ssl-domain-names-root-domain-hosting-for-amazon-cloudfront.html

over 1 year ago ·
7815

That is actually not correct. Static website CAN be served from S3 and it's been that way for a while...

over 1 year ago ·
7816

Wouldn't it be easier to setup an "origin-pull" scenario via CloudFront?

over 1 year ago ·
7819
D7b0ba8a19490cbab513b172020f50a3

Use Amazon Route53 Failover so everything is automated (you don't have to remove the apache from the load balancer pool manually)

over 1 year ago ·
7820

That works as long as Amazon itself doesn't take a nosedive. Maybe host it somewhere else and deal with the redirect using DNS?

over 1 year ago ·
7821

What how many and what size servers did you have running for your million person email blast? Just curious

over 1 year ago ·
7822
Jackjon

Pineappleio - We only needed 1 c1.xlarge server but I ran 5 of them just in case. See my other post on how I set it up: https://coderwall.com/p/__z9ia

over 1 year ago ·
7824
Jackjon

navigatetech - Agreed. In our case, it is my opinion that that we're too early to use a strategy where we're on multiple hosts.

over 1 year ago ·
7825
4222f8d291b7439928d29c2c368e41b7

This is great advice! Set up your static files on S3 and serve them with Cloudfront. I have had this sort of backup going for quite a while, and since I already store my images this way, it's a natural transition. The cost of S3+cloudfront (a few bucks a month for my blog) is well worth it.

Thanks for the tips!

over 1 year ago ·
7828
39a7068dd91c30c33dea4db353d6be02

If you are really concerned with an entire region going down ( unlikely ), run that maintenance page in a different region, and use Route 53's failover abilities to help do that failover for you! http://aws.typepad.com/aws/2013/02/create-a-backup-website-using-route-53-dns-failover-and-s3-website-hosting.html

over 1 year ago ·
7829

Great idea (we also run ours cross-region/cross-datacenter for redundancy), but you are basically losing the scale advantages of S3 with the Apache server in the critical path (I believe Apache or nginx can serve static file requests basically as quickly as 302s, though I haven't run the benchmarks myself).

You can eliminate the LB, Apache server, and redirect by just using DNS CNAMEs if you change your "common path" URL (eg. the one in your email) to be http://www.rockthepost.com (you still need the LB & webserver for http://rockthepost.com since you can't reasonably CNAME root records). Then on failure, just point the DNS for www to your S3 site.

over 1 year ago ·
8005
E992a2ec5fca727a0e14cc9593f5c731

I've visited sites that used CloudFlare, & when they were down CF served the cached site. Doesn't seem to be listed specifically https://www.cloudflare.com/overview

over 1 year ago ·
8628
3642f9a4541eda45f6e81226741e170b

Yeah, that CloudFlare feature is really great. It's called "Always Online": https://www.cloudflare.com/always-online

over 1 year ago ·
Featured Programming Job

Product Manager
·
San Jose
·
Full Time
Search all programming jobs