Starting multiple DelayedJob workers w/ specific queues via Capistrano tasks

delayed-job, ruby, ruby-on-rails, ruby-on-rails-3

Solution

I split my jobs into two queues with one worker isolated to each queue with this setup in my `deploy.rb` file:

namespace :delayed_job do
  task :start, roles: :app do
    run "cd #{current_path}; #{rails_env} bundle exec script/delayed_job -i queue_one --queue=one start"
    run "cd #{current_path}; #{rails_env} bundle exec script/delayed_job -i queue_two --queue=two start"
  end

  task :stop, roles: :app do
    run "cd #{current_path}; #{rails_env} bundle exec script/delayed_job -i queue_one --queue=one stop"
    run "cd #{current_path}; #{rails_env} bundle exec script/delayed_job -i queue_two --queue=two stop"
  end

  task :restart, roles: :app do
    run "cd #{current_path}; #{rails_env} bundle exec script/delayed_job -i queue_one --queue=one restart"
    run "cd #{current_path}; #{rails_env} bundle exec script/delayed_job -i queue_two --queue=two restart"
  end
end

The `-i name` part of the command is very important. That's the part that allows multiple `delayed_job` instances to run.

If you want to add workers to specific queues, then you would expand them out like this (where I have two workers exclusively on queue one, and one worker exclusively on queue two):

namespace :delayed_job do
  task :start, roles: :app do
    run "cd #{current_path}; #{rails_env} bundle exec script/delayed_job -i one1 --queue=one start"
    run "cd #{current_path}; #{rails_env} bundle exec script/delayed_job -i one2 --queue=one start"
    run "cd #{current_path}; #{rails_env} bundle exec script/delayed_job -i two --queue=two start"
  end

  task :stop, roles: :app do
    run "cd #{current_path}; #{rails_env} bundle exec script/delayed_job -i one1 stop"
    run "cd #{current_path}; #{rails_env} bundle exec script/delayed_job -i one2 stop"
    run "cd #{current_path}; #{rails_env} bundle exec script/delayed_job -i two stop"
  end

  task :restart, roles: :app do
    run "cd #{current_path}; #{rails_env} bundle exec script/delayed_job -i one1 --queue=one restart"
    run "cd #{current_path}; #{rails_env} bundle exec script/delayed_job -i one2 --queue=one restart"
    run "cd #{current_path}; #{rails_env} bundle exec script/delayed_job -i two --queue=two restart"
  end
end

Problem

I'm looking into using queues with delayed_job. I've found this page which outlines various ways of starting workers, however I'd like to keep my currently Capistrano method: ``` set :delayed_job_args, "-n 2 -p ecv2.production" after "deploy:start", "delayed_job:start" ... ``` I was wondering how I could modify the delayed_job_args to handle spawning 1 worker with a specific queue, and 1 worker for every other job. So far, all I have is overriding each task like so: ``` namespace :delayed_job do task :restart, :roles => :app do run "cd #{current_path}; RAILS_ENV=#{rails_env} script/delayed_job -p ecv2.production --queue=export restart" run "cd #{current_path}; RAILS_ENV=#{rails_env} script/delayed_job -p ecv2.production restart" end end ``` ... But that's no fun. Any suggestions?

Original source