Celery Heartbeat Not Working
celery, django, python, rabbitmq
Solution
the heartbeat of celery worker is application level heartbeat, not AMQP protocol's heartbeat. Each worker periodically send heartbeat event message to "celeryev" event exchange in BROKER. The heartbeat event is forwarded back to worker such worker can know the health status of BROKER. If number of loss heartbeat exceeding a threshold, the worker can do some reconnect action to BROKER.
For the rest of detail, you may check this page The section: BROKER_FAILOVER_STRATEGY describes the actions you may do for dropping from a BROKER.
Problem
I have set heartbeat in Celery settings: ``` BROKER_HEARTBEAT = 10 ``` I have also set this configuration value in RabbitMQ config: ``` 'heartbeat' => '10', ``` But somehow heartbeats are still disabled: ``` ubuntu@sync1:~$ sudo rabbitmqctl list_connections name timeout Listing connections ... some_address:37781 -> other_address:5672 0 some_address:37782 -> other_address:5672 0 ...done. ``` Any ideas what am I doing wrong? UPDATE: So now I get: ``` ubuntu@sync1:/etc/puppet$ sudo rabbitmqctl list_connections name timeout Listing connections ... some_address:41281 -> other_address:5672 10 some_address:41282 -> other_address:5672 10 some_address:41562 -> other_address:5672 0 some_address:41563 -> other_address:5672 0 some_address:41564 -> other_address:5672 0 some_address:41565 -> other_address:5672 0 some_address:41566 -> other_address:5672 0 some_address:41567 -> other_address:5672 0 some_address:41568 -> other_address:5672 0 ...done. ``` I have 3 servers: - RabbitMQ broker - RESTful API server - Remote Worker server It appears the remote demonised Celery workers send heartbeats correctly. The RESTful API server using Celery to remotely process tasks is not using heartbeat for some reason.