Where developers come to connect, share, build and be inspired.

0

Save your company's ass - Launch a static "emergency mode" site on S3

42551 views


We were just featured in an email that was sent to a million people. It went great but we were a little concerned about what would happen if we had an unexpected problem in our application at the last minute. If our site went down on the day that thousands of people were all trying to get to RockThePost, we would have squandered the opportunity and I would have have been completely embarrassed.

We decided to launch our "hot spare" website. It's a special lite version of RockThePost which allows you to see some of the startups we are currently featuring. You can even sign up.

To get an idea of how it works, first take a look at our regular website:

https://rockthepost.com/

Now, take a look at our emergency "holy shit we're in trouble" version of RockThePost:

http://maintenance.rockthepost.com/

Just in case the RockThePost has massive problems, my recovery plan is as follows:

1) I remove the normal web servers from the load balancer

2) I launch my "emergency" Apache redirect server and put it in the pool

My emergency web server simply mod_rewrites all of the requests to http://maintenance.rockthepost.com/

Our maintenance website is hosted directly on S3 as a static site. As far as I'm concerned, S3 static file website serving is completely indestructible. I only need one bland Apache server to bump requests over to it.

We knew the maintenance site needed to be able to accomplish two things: We wanted to allow people to continue to register for our service and also see current ventures. In order to accomplish this, we signed up with http://www.wufoo.com/ to do the user data collection form processing and we wrote a periodic Jenkins job to rewrite the ventures which are displayed on the emergency maintenance site.

Now, even in the worst of cases, I have some realistic hope of keeping our brand on the internet and continuing to collect user registrations even if we have a holy-shit-we-are-down day.

Comments

  • Bd0391d7225d10455d35de0fb5485bea

    Great idea! However, S3 isn't built to serve static assets. You should probably be using CloudFront instead.

  • 4222f8d291b7439928d29c2c368e41b7

    This is great advice! Set up your static files on S3 and serve them with Cloudfront. I have had this sort of backup going for quite a while, and since I already store my images this way, it's a natural transition. The cost of S3+cloudfront (a few bucks a month for my blog) is well worth it.

    Thanks for the tips!

  • D7b0ba8a19490cbab513b172020f50a3

    Use Amazon Route53 Failover so everything is automated (you don't have to remove the apache from the load balancer pool manually)

  • User-avatar

    Great idea (we also run ours cross-region/cross-datacenter for redundancy), but you are basically losing the scale advantages of S3 with the Apache server in the critical path (I believe Apache or nginx can serve static file requests basically as quickly as 302s, though I haven't run the benchmarks myself).

    You can eliminate the LB, Apache server, and redirect by just using DNS CNAMEs if you change your "common path" URL (eg. the one in your email) to be http://www.rockthepost.com (you still need the LB & webserver for http://rockthepost.com since you can't reasonably CNAME root records). Then on failure, just point the DNS for www to your S3 site.

  • Jackjon
  • Bd0391d7225d10455d35de0fb5485bea

    Those docs were written before Amazon CloudFront supported root domain hosting. Check this link out:

    http://aws.typepad.com/aws/2013/06/custom-ssl-domain-names-root-domain-hosting-for-amazon-cloudfront.html

  • User-avatar

    That is actually not correct. Static website CAN be served from S3 and it's been that way for a while...

  • User-avatar

    Wouldn't it be easier to setup an "origin-pull" scenario via CloudFront?

  • User-avatar

    That works as long as Amazon itself doesn't take a nosedive. Maybe host it somewhere else and deal with the redirect using DNS?

  • User-avatar

    What how many and what size servers did you have running for your million person email blast? Just curious

  • Jackjon

    Pineappleio - We only needed 1 c1.xlarge server but I ran 5 of them just in case. See my other post on how I set it up: https://coderwall.com/p/__z9ia

  • Jackjon

    navigatetech - Agreed. In our case, it is my opinion that that we're too early to use a strategy where we're on multiple hosts.

  • 39a7068dd91c30c33dea4db353d6be02

    If you are really concerned with an entire region going down ( unlikely ), run that maintenance page in a different region, and use Route 53's failover abilities to help do that failover for you! http://aws.typepad.com/aws/2013/02/create-a-backup-website-using-route-53-dns-failover-and-s3-website-hosting.html

  • E992a2ec5fca727a0e14cc9593f5c731

    I've visited sites that used CloudFlare, & when they were down CF served the cached site. Doesn't seem to be listed specifically https://www.cloudflare.com/overview

  • 3642f9a4541eda45f6e81226741e170b

    Yeah, that CloudFlare feature is really great. It's called "Always Online": https://www.cloudflare.com/always-online

Add a comment