Rails, Heroku, Unicorn & Resque - how to choose the number of web workers / workers resque?
I just switched to using Unicorn on Hereka. I am also going to switch to resque from delayed_job and use the setting described in http://bugsplat.info/2011-11-27-concurrency-on-heroku-cedar.html
From this I don't understand how config / unicorn.rb:
worker_processes 3
timeout 30
@resque_pid = nil
before_fork do |server, worker|
@resque_pid ||= spawn("bundle exec rake " + \
"resque:work QUEUES=scrape,geocode,distance,mailer")
end
means:
"This will actually lead to six processes in each web dyno: 1 unicorn host, 3 unicorn web workers, 1 resque worker, 1 resque child worker when he actually processes the job.
How many workers will actually process background jobs? 1 or 2?
Suppose I wanted to increase the number of worker reskies - what would I change?
source to share
I think if you run this block, you already have your unicorn master, plus the 3 webmasters you specify at the top of the file, and then the block below starts one Resque worker if not already running.
My guess is that Resque launches the child worker on its own when it actually does the job.
It would seem that if you wanted another Resque worker, you could just do
worker_processes 3
timeout 30
@resque_pid = nil
@resque_pid2 = nil
before_fork do |server, worker|
@resque_pid ||= spawn("bundle exec rake " + \
"resque:work QUEUES=scrape,geocode,distance,mailer")
@resque_pid2 ||= spawn("bundle exec rake " + \
"resque:work QUEUES=scrape,geocode,distance,mailer")
end
In my experience with Resque, this is as easy as starting another process as above. The only uncertainty I have is with Heroku and how it decides to deal with giving you more workers.
source to share