ZeroMQ buffer size v/s High Water Mark

Question

In the `zeromq` socket options, we have flags for both a high water mark and a buffer size. For sending, it's `ZMQ_SNDHWM` and `ZMQ_SNDBUF`. Can someone explain the difference between these two?

Accepted Answer

Each one controls some other thing:

`ZMQ_SNDBUF`: Set kernel transmit buffer size The `ZMQ_SNDBUF` option shall set the underlying kernel transmit buffer size for the socket to the specified size in bytes. A ( default ) value of -1 means leave the OS default unchanged.

where `man 7 socket` says: ( credits go to @Matthew Slatery )

```
[...]

SO_SNDBUF
          Sets or gets the maximum socket send buffer in bytes.  The  ker-
          nel doubles this value (to allow space for bookkeeping overhead)
          when it is set using setsockopt(), and  this  doubled  value  is
          returned  by  getsockopt().   The  default  value  is set by the
          wmem_default sysctl and the maximum allowed value is set by  the
          wmem_max sysctl.  The minimum (doubled) value for this option is
          2048.
[...]

NOTES
   Linux assumes that half of the send/receive buffer is used for internal
   kernel structures; thus the sysctls are twice what can be  observed  on
   the wire.
[...]
```

whereas

`ZMQ_SNDHWM`: Set high water mark for outbound messages The `ZMQ_SNDHWM` option shall set the high water mark for outbound messages on the specified socket. The high water mark is a hard limit on the maximum number of outstanding messages ØMQ shall queue in memory for any single peer that the specified socket is communicating with. A ( non-default ) value of zero means no limit. If this limit has been reached the socket shall enter an exceptional state and depending on the socket type, ØMQ shall take appropriate action such as blocking or dropping sent messages. Refer to the individual socket descriptions in zmq_socket(3) for details on the exact action taken for each socket type. ØMQ does not guarantee that the socket will accept as many as `ZMQ_SNDHWM` messages, and the actual limit may be as much as 60-70% lower depending on the flow of messages on the socket.

Verba docent, Exempla trahunt:

Having an infrastructure setup, where `zmq.PUB` side leaves all the settings in their default values, and such sender will have about 20, 200, 2000 `zmq.SUB` abonents listening to the sender, there would be soon depleted the default, unmodified O/S kernel buffer space, as each `.bind()/.connect()` relation ( each abonent ) will try to "stuff" as much as about the total cummulative sum of `~ 1000 * aVarMessageSIZE [Bytes]` of data, once having been broadcast in a `.send( ..., ZMQ_DONTWAIT )` manner & if the O/S would not have provided a sufficient buffer space -- there we go --

"Houston, we have a problem..." If the message cannot be queued on the socket, the `zmq_send()` function shall fail with `errno` set to `EAGAIN`.

Q.E.D.

ZeroMQ buffer size v/s High Water Mark

Solution

Problem

Related problems