celery missed heartbeat (on_node_lost)
celery, django, python
Solution
Celery 3.1 added in the new mingle and gossip procedures. I too was getting a ton of missed heartbeats and passing --without-gossip to my workers cleared it up.
https://docs.celeryproject.org/en/3.1/whatsnew-3.1.html#mingle-worker-synchronization
Mingle: Worker synchronization
The worker will now attempt to synchronize with other workers in the same cluster.
Synchronized data currently includes revoked tasks and logical clock.
This only happens at startup and causes a one second startup delay to collect broadcast responses from other workers.
You can disable this bootstep using the --without-mingle argument.
https://docs.celeryproject.org/en/3.1/whatsnew-3.1.html#gossip-worker-worker-communication
Gossip: Worker <-> Worker communication
Workers are now passively subscribing to worker related events like heartbeats.
This means that a worker knows what other workers are doing and can detect if they go offline. Currently this is only used for clock synchronization, but there are many possibilities for future additions and you can write extensions that take advantage of this already.
Some ideas include consensus protocols, reroute task to best worker (based on resource usage or data locality) or restarting workers when they crash.
We believe that although this is a small addition, it opens amazing possibilities.
You can disable this bootstep using the --without-gossip argument.
Problem
I just upgraded to celery 3.1 and now I see this i my logs :: ``` on_node_lost - INFO - missed heartbeat from celery@queue_name for every queue/worker in my cluster. ``` According to the docs `BROKER_HEARTBEAT` is off by default and I haven't configured it. Should I explicitly set `BROKER_HEARTBEAT=0` or is there something else that I should be checking?