How to handle long asynchronous requests with pyramid and celery?
amqp, celery, pyramid, rabbitmq
Solution
RabbitMQ has flow control built into the QoS. If RabbitMQ cannot handle the publishing rate it will adjust the TCP window size to slow down the publishers. In the event of too many messages being sent to the server it will also overflow to disk. This will allow your consumer to be a bit more naive although if you restart the connection on error and flood the connection you can cause problems.
I've always decided to spend more time making sure the publishers/consumers could work with multiple queue servers instead of trying to make them more intelligent about a single queue server. The benefit is that if you are really overloading a single server you can just add another one (or another pair if using RabbitMQ HA. There is a useful video from Pycon about Messaging at Scale using Celery and RabbitMQ that should be of use.
Problem
I'm setting up a web service with pyramid. A typical request for a view will be very long, about 15 min to finish. So my idea was to queue jobs with celery and a rabbitmq broker. I would like to know what would be the best way to ensure that bad things cannot happen. Specifically I would like to prevent the task queue from overflow for example. A first mesure will be defining quotas per IP, to limit the number of requests a given IP can submit per hour. However I cannot predict the number of involved IPs, so this cannot solve everything. I have read that it's not possible to limit the queue size with celery/rabbitmq. I was thinking of retrieving the queue size before pushing a new item into it but I'm not sure if it's a good idea. I'm not used to good practices in messaging/job scheduling. Is there a recommended way to handle this kind of problems ?