Celery creating a new connection for each task

Question

I'm using Celery with Redis to run some background tasks, but each time a task is called, it creates a new connection to Redis. I'm on Heroku and my Redis to Go plan allows for 10 connections. I'm quickly hitting that limit and getting a "max number of clients reached" error. How can I ensure that Celery queues the tasks on a single connection rather than opening a new one each time? EDIT - including the full traceback ``` File "/app/.heroku/venv/lib/python2.7/site-packages/django/core/handlers/base.py", line 111, in get_response response = callback(request, *callback_args, **callback_kwargs) File "/app/.heroku/venv/lib/python2.7/site-packages/newrelic-1.4.0.137/newrelic/api/object_wrapper.py", line 166, in __call__ self._nr_instance, args, kwargs) File "/app/.heroku/venv/lib/python2.7/site-packages/newrelic-1.4.0.137/newrelic/hooks/framework_django.py", line 447, in wrapper return wrapped(*args, **kwargs) File "/app/.heroku/venv/lib/python2.7/site-packages/django/views/decorators/csr…

Accepted Answer

I ran into the same problem on Heroku with CloudAMQP. I do not know why, but I had no luck when assigning low integers to the `BROKER_POOL_LIMIT` setting.

Ultimately, I found that by setting `BROKER_POOL_LIMIT=None` or `BROKER_POOL_LIMIT=0` my issue was mitigated. According to the Celery docs, this disables the connection pool. So far, this has not been a noticeable issue for me, however I'm not sure if it might be for you.

Link to relevant info: http://celery.readthedocs.org/en/latest/configuration.html#broker-pool-limit

Celery creating a new connection for each task

Solution

Problem