fprnhg
18.62K
· March 2013 ·
E6b625008c816ab3d8d742cc3eddbb00

Free background jobs on heroku

Rails + Sidekiq with a single heroku dyno

Sidekiq is a gem for background jobs and a great alternative to Resque or DelayedJob. To set it up with a rails app on heroku normally requires an extra worker process.

You can avoid the need for an extra dyno by using unicorn as your webserver. Unicorn lets you spawn your sidekiq workers within your web dyno.

Sidekiq your web dyno!

if your Profile looks like this

web: bundle exec unicorn -p $PORT -E $RACK_ENV -c ./config/unicorn.rb

then just add something like the following to your config/unicorn.rb

before_fork do |server, worker|
   @sidekiq_pid ||= spawn("bundle exec sidekiq -c 2")
end

Redis connection limitation

If you are using sidekiq with heroku's redis-to-go nano addon you might also need to tweak you sidekiq concurrency and connection size to not max out nano's 10 connections limit. (see this blog post for details)

Here are some settings I use on a small app in production on heroku with three unicorn worker processes, sidekiq concurrency set to two (see bundle exec sidekiq -c 2 above) and redis-to-go's nano plan. I did not really fiddle around with it much - it works without problems so far for me.

config/unicorn.rb

worker_processes 3
after_fork do |server, worker|
  Sidekiq.configure_client do |config|
    config.redis = { :size => 1 }
  end
  Sidekiq.configure_server do |config|
    config.redis = { :size => 5 }
  end
end

Alternative

You can also use something like sucker_punch. It does all the work within your rails process itself whereas Sidekiq runs as a separate long-running process, independent from your Rails process.

Sign in or sign up to add your response.

17 Responses

5066
C7b8c0b4ef05e68e2c92cd912eca414b
5067
E6b625008c816ab3d8d742cc3eddbb00

@linjunpop - I mentioned sucker_punch in the last paragraph "Alternatives". Anything particular you wanted to say?

over 1 year ago ·
5068
C7b8c0b4ef05e68e2c92cd912eca414b

@dommmel Oh, Sorry. I guess I had Evernote an old version of this tip.

over 1 year ago ·
6191
881c97b0386c0871903d9cd8da27d0c9

Very nice! I was wondering if there was an alternative to the two dynos on heroku. I'll give a try

over 1 year ago ·
6328
Profile picture may 2012

How would you handle if the process dies? Heroku won't notice & bounce your dyno and Unicorn doesn't notice either.

over 1 year ago ·
6331
E6b625008c816ab3d8d742cc3eddbb00

@jalada Good question. I guess you could use Process.waitpid in some way... haven't done it myself

over 1 year ago ·
6332
Profile picture may 2012

I actually implemented it. Time will tell if it works for sure. This kills the parent process as well, which on Heroku will bounce your entire dyno (which is what I wanted):

@worker_pid = spawn('bundle exec sidekiq')
t = Thread.new {
  Process.wait(@worker_pid)
  puts "Worker died. Bouncing unicorn."
  Process.kill 'QUIT', Process.pid
}
# Just in case
t.abort_on_exception = true

The idea of using Thread.new { Process.wait() } comes from the implementation of Process.detach, which is essentially the same.

over 1 year ago ·
6334
E6b625008c816ab3d8d742cc3eddbb00

@jalada Cool - please keep us updated on how it works out

over 1 year ago ·
7037
De8b71442a3187222718bd993f1eb835

Hi guys,
@jalada @dommmel did it work?

over 1 year ago ·
9940
London bridge sq

Hi, how can this be done with Puma? Concurrency noob here

over 1 year ago ·
10818
0 iykox7k9 md63p7nwxpmx2vz fow3xxnwgkvx2lbyoomore9bs86eulwstynczysoxtqiwrzrqss

looks like it to me - here is my puma.rb using 2 workers. Not tested under load just yet - and make sure you have you don't hit your db and redis connection limits.

environment ENV['RACK_ENV']
workers 2
threads 2,5 # 4-6
preload_app!

on_worker_boot do
  @sidekiq_pid ||= spawn("bundle exec sidekiq -c 2")
end
over 1 year ago ·
11226
Be8c56bdb448f4331038a776c14bf0ad

Thanks for the tip. You mentioned sucker punch as an alternative but am I right in thinking that with sucker punch, I wouldn't be able to start a job from a rake task (heroku scheduler, for instance) or some other scheduling job and run it in the background of one of the existing heroku dynos?

over 1 year ago ·
11227

Does worker perform even when the Heroku app goes to sleep?

over 1 year ago ·
13029

@matixmatix Set up the New Relic plugin to ping your server periodically and it will always be awake!

over 1 year ago ·
15161
Twitter icon normal

or you can use dynopoker gem https://github.com/kubenstein/dynopoker
its part of lil page i crafted http://wakemydyno.com

over 1 year ago ·
17148
Efe9dbb42b0ca3be02f1e2bea66fde3e

great tip!

over 1 year ago ·
17479
2a9b8f5273d934fe57daa8cf54c3a017

HERE BE DRAGONS!

I did this and it caused connections to my postgres database to build up on deploys and then took my site down. Use with caution!

over 1 year ago ·
Featured Programming Job

Front End Developer
·
Atlanta, GA
·
Full Time
Search all programming jobs