select and accept delay in Linux

c, linux, select, sockets

Solution

UPDATE/WARNING: while trying to prove this answer applies, I found that maybe it doesn't. I ran the test and got delays without max_fd ever getting higher than 300. And I got delays with poll() too. So I tried tcpdump and there were retransmissions. It looks like even 127.0.0.1 can drop packets when you throw them at it this fast. Leaving the answer here because it is a real issue, even if it's not the most pressing one.

So this involves a lot of file descriptors, and it works with poll but not select. With those clues I can see the explanation: you've gone over the `FD_SETSIZE` limit.

The official pronouncement from POSIX is (referring to `FD_ZERO`/`FD_SET`/`FD_CLR`/`FD_ISSET`):

The behavior of these macros is undefined if the fd argument is less than 0 or greater than or equal to FD_SETSIZE, or if fd is not a valid file descriptor, or if any of the arguments are expressions with side-effects.

(from http://pubs.opengroup.org/onlinepubs/9699919799/functions/select.html)

To really understand what happened you have to look deeper than the official specification into the actual implementation of the `fd_set` type. It has a split personality. In the kernel, where `select` is implemented, it's treated as a variable-length array of bits. The first argument to `select` is used to decide where the array ends. If you call `select(2048, ...)` the kernel will expect each non-NULL `fd_set *` to point to an array of 256 bytes (2048 bits).

But in userspace, `fd_set` is a fixed-size struct. The size is `FD_SETSIZE` bits, which is 1024 on my system and probably yours too. `FD_SET` and the other macros are basically just doing assignments to elements of the array, only they're a little more complicated because they have to deal with the conceptual array elements being the individual bits. So if one of your file descriptors is 1024 and you try to `FD_SET` it, you've done the equivalent of

int array[1024];
array[1024] = 1;

In other words, you clobbered whatever was in memory after the `fd_set`, causing weird things to happen later.

There are ways around this. I've seen old code that does a `#define FD_SETSIZE somebignumber` before including the header that defines `fd_set`. I don't know what OSes that worked on; I just tried it and glibc seems to ignore it.

A better possibility is to do something like the old "struct hack" where you'd allocate a struct with more memory than its `sizeof`, and the extra memory would be usable as extra elements in the array that was the last member of the struct.

fd_set *rfds = malloc(128+sizeof *foo); /* can hold fds up to FD_SETSIZE+128*8-1 */

Now of course you need to remember to free it when you're done with it, and pass `rfds` instead of `&rfds` to `select` and the `FD_*` macros, and do your own `memset` instead of `FD_ZERO`, and hope that the kernel implementation doesn't change since you're now real chummy with it. But it works... for now.

Using poll is actually probably the correct answer.

Problem

Original source