We were just featured in an email that was sent to a million people. It went great but we were a little concerned about what would happen if we had an unexpected problem in our application at the last minute. If our site went down on the day that thousands of people were all trying to get to RockThePost, we would have squandered the opportunity and I would have have been completely embarrassed.
We decided to launch our "hot spare" website. It's a special lite version of RockThePost which allows you to see some of the startups we are currently featuring. You can even sign up.
To get an idea of how it works, first take a look at our regular website:
Now, take a look at our emergency "holy shit we're in trouble" version of RockThePost:
Just in case the RockThePost has massive problems, my recovery plan is as follows:
1) I remove the normal web servers from the load balancer
2) I launch my "emergency" Apache redirect server and put it in the pool
My emergency web server simply mod_rewrites all of the requests to http://maintenance.rockthepost.com/
Our maintenance website is hosted directly on S3 as a static site. As far as I'm concerned, S3 static file website serving is completely indestructible. I only need one bland Apache server to bump requests over to it.
We knew the maintenance site needed to be able to accomplish two things: We wanted to allow people to continue to register for our service and also see current ventures. In order to accomplish this, we signed up with http://www.wufoo.com/ to do the user data collection form processing and we wrote a periodic Jenkins job to rewrite the ventures which are displayed on the emergency maintenance site.
Now, even in the worst of cases, I have some realistic hope of keeping our brand on the internet and continuing to collect user registrations even if we have a holy-shit-we-are-down day.