tl;dr - Apply a free monitoring service to keep your site from "sleeping".
If you have a free account on AppHarbor, you're only given one web worker. I have found that hitting my site for the first time in awhile is very slow. My guess is that the application pool is suspended after no activity.
I have seen much better performance from my site after signing up and applying 5 minute monitoring tests with pingdom. If my site had lots of traffic, this would become less of a problem, but if that was the case, I might be able to up my subscription on AppHarbor.