What are appropriate settings for ZMQ when sending 500K 64 byte messages?
messaging, zeromq
Solution
Let's break this down.
First, why the HWM isn't "working":
The HWM is not an exact limit, since internal buffers are filled and emptied by two separate threads, and the count of available space can lag quite a lot when there's a lot of activity. The 0MQ zmq_setsockopt man page says, "0MQ does not guarantee that the socket will accept as many as ZMQ_SNDHWM messages, and the actual limit may be as much as 60-70% lower depending on the flow of messages on the socket."
Second, why are you losing messages:
As you dump 0.5M messages (x 20) into the sockets' buffers, you will randomly hit the HWM and the PUB socket's behaviour then is to drop the messages it can't queue.
Third, how to solve this:
There's zero reason to break the state into separate messages; the only rationale for this would be if the state did not fit into memory, which it does easily. Send as multipart (ZMQ_SNDMORE); this creates a single effective message that takes 1 slot in the outgoing buffer.
Then, remove your 500K HWM limit and revert to the default (1000) which will be more than sufficient.
Fourth, how to get better performance:
Obviously, profile and improve your publisher and subscriber code as possible; these are the usual bottlenecks.
Then, consider some form of compression on the message if it is sparse and you can do that without too much CPU cost. At 20 subscribers you will usually gain more from network overhead than you will lose from CPU cost.
Finally, if you grow to more subscribers and it's a critical system, look at PGM multicast, which will effectively remove the network costs.
Problem
I have an application that periodically needs to send out a snapshot of its current state, which currently would be represented by about 500,000 64 byte messages. I've been having difficulty getting this many messages sent and received quickly and reliably using ZMQ. I've been using PUB/SUB over tcp to do this currently, but I'm not wedded to either the pattern or the protocol as long as it will get the job done. In my experiments I've focused on playing around with the send and receive high water mark, send and receive buffer settings, and adding some sleeps to the send loop to try to slow it down a bit. With settings that seemed quite generous to me (500K HWM, 10MB buffers) and using only a loopback connection the messages still aren't all being received consistently. I'm interested in what are appropriate settings for these or other tuning parameters, and more broadly in how to reason about the effect various settings will have. Some further details that may help provide an appropriate answer: The distribution is one to many. The expected number of recipients is around 20. Each message represents a set of information about a different financial instrument, all observed at the same time. In my mind arguments can be made for both combining them into one big message (the set of all messages logically makes up one complete snapshot) and for keeping them separate (clients may potentially be interested only in some instruments, and I think this would help filter them out more easily). The intended frequency of messages is basically no faster than every 20 milliseconds, and no slower than 5 seconds. Where I actually land will probably be influenced by performance considerations (ie, how fast my server can actually pump the messages out and what kind of data rate would prove overwhelming to clients).