Maximize throughput with RabbitMQ
rabbitmq
Solution
For best performance in RabbitMQ, follow the advice of its creators. From the RabbitMQ blog:
RabbitMQ's queues are fastest when they're empty. When a queue is empty, and it has consumers ready to receive messages, then as soon as a message is received by the queue, it goes straight out to the consumer. In the case of a persistent message in a durable queue, yes, it will also go to disk, but that's done in an asynchronous manner and is buffered heavily. The main point is that very little book-keeping needs to be done, very few data structures are modified, and very little additional memory needs allocating.
If you really want to dig deep into the performance of RabbitMQ queues, this other blog entry of theirs goes into the data much further.
Problem
In our project, we want to use the RabbitMQ in "Task Queues" pattern to pass data. On the producer side, we build a few TCP server(in node.js) to recv high concurrent data and send it to MQ without doing anything. On the consumer side, we use JAVA client to get the task data from MQ, handle it and then ack. So the question is: To get the maximum message passing throughput/performance( For example, 400,000 msg/second) , How many queues is best? Does that more queue means better throughput/performance? And is there anything else should I notice? Any known best practices guide for using RabbitMQ in such scenario? Any comments are highly appreciated!!