fclose()/pclose() may block on some file pointers

c, fclose, popen, stdio, unix

Solution

Disappointed with the generality of the answers so far (I can RTFM, tyvm), I've investigated this thoroughly, by stepping through and reading the glibc source.

In glibc `pclose()` directly calls `fclose()` with no additional effect, so the 2 calls are same. In fact you could use `pclose()` and `fclose()` interchangeably. I'm sure this is purely a coincidence in the evolved implementation, and the use of `pclose()` to close a `FILE *` returned from `popen()` is still recommended.

The magic is in `popen()`. `FILE *`s in glibc contain a jump table with pointers to appropriate functions to handle such calls as `fseek()`, `fread()`, and of relevance `fclose()`. When calling `popen()`, a different jump table used than the one used by `fopen()`. The `close` member in this jump table points to a special function `_IO_new_proc_close`, which calls `waitpid()` on the pid stored in the region pointed to by `FILE *`.

Here's the relevant call stack in my version of glibc, which I've annotated with notes about what is going on:

// linux waitpid system call interface
#0  0x00f9a422 in __kernel_vsyscall ()
#1  0x00c38513 in __waitpid_nocancel () from /lib/tls/i686/cmov/libc.so.6

// removes fp from a chain of proc files
// and waits for the process of the stored pid to terminate
#2  0x00bff248 in _IO_new_proc_close (fp=0x804b008) at iopopen.c:357

// flushes the stream and calls close in its jump table
#3  0x00c09ff3 in _IO_new_file_close_it (fp=0x804b008) at fileops.c:175

// destroys the FILEs buffers
#4  0x00bfd548 in _IO_new_fclose (fp=0x804b008) at iofclose.c:62

// calls fclose
#5  0x00c017fd in __new_pclose (fp=0x804b008) at pclose.c:43

// calls pclose
#6  0x08048788 in main () at opener.c:34

So the short of it is, using `popen()`, the returned `FILE *` must not be closed, even if you `dup()` its file descriptor, because it will block until the child process terminates. Of course, after this you'll be left with a file descriptor to a pipe which will contain whatever the child process managed to write() to it before terminating.

By not `fread()`ing with the file pointer returned from `popen()`, the underlying pipe will not be touched, it's safe to use the file descriptor by `fileno()`, and finish up by calling `pclose()`.

Problem

Calling `fclose()` here after `dup()`ing its file descriptor blocks until the child process has ended (presumably because the stream has ended). ``` FILE *f = popen("./output", "r"); int d = dup(fileno(f)); fclose(f); ``` However by manually performing the `pipe()`, `fork()`, `execvp()` of the `popen()`, and then `dup()`ing the pipe's read file descriptor, closing the original does not block. ``` int p[2]; pipe(p); switch (fork()) { case 0: { char *argv[] = {"./output", NULL}; close(p[0]); dup2(p[1], 1); execvp(*argv, argv); } default: { close(p[1]); int d = dup(p[0]); close(p[0]); } } ``` Why does this occur, and how can I close the `FILE *` returned from `popen()` and use a file descriptor in its place? Update: I'm aware that the documentation says to use `pclose()`, however `fclose()` blocks as well. Furthermore, I poked around in the glibc code, and `pclose()` just calls `fclose()`. The behaviour is the same, whether `fclose()` or `pclose()` is used.

Original source