multiple workers and multiple queues on celery django daemon

celery, celeryd, daemon, django, django-celery

Solution

You can use `CELERYD_OPTS` option passing -Q parameter similar to those examples from Celery reference:

# Advanced example starting 10 workers in the background:
#   * Three of the workers processes the images and video queue
#   * Two of the workers processes the data queue with loglevel DEBUG
#   * the rest processes the default' queue.
$ celery multi start 10 -l INFO -Q:1-3 images,video -Q:4,5 data
    -Q default -L:4,5 DEBUG

# You can show the commands necessary to start the workers with
# the 'show' command:
$ celery multi show 10 -l INFO -Q:1-3 images,video -Q:4,5 data
    -Q default -L:4,5 DEBUG

Problem

I am using celery-django to queue tasks on my site backend. I am trying to create a setup where I have two queues named "low" and "high" and two workers W1 and W2. I want them to consume the tasks from the queue in the following way: W1 <-- low, high W2 <-- high Normally it can be done like this. Open terminal 1 and enter $`celery worker -n W1 -Q low,high` Open terminal 2 and enter $`celery worker -n W2 -Q high` However I am trying to do the same via celeryd daemon. I am following the steps given in the link: http://celery.readthedocs.org/en/latest/tutorials/daemonizing.html#example-configuration But the available options don't seem enough to fit the requirement. Please help me with some configs that I am unaware of which could make it possible. I would prefer not to run multiple daemons or use additional tools like supervisord unless really necessary (Maybe you could advice me on this as well).

Original source