Resource temporarily unavailable error with subprocess module in Python

gnuplot, python, resources, subprocess

Solution

pid numbers, open file descriptors, memory are limited resources.

fork(2) manual says when `errno.EAGAIN` should happen:

[EAGAIN]  The system-imposed limit on the total number of processes under
          execution would be exceeded.  This limit is configuration-dependent.

[EAGAIN]  The system-imposed limit MAXUPRC () on the total number of processes
          under execution by a single user would be exceeded.

To reproduce the error more easily, you could add at the start of your program:

import resource

resource.setrlimit(resource.RLIMIT_NPROC, (20, 20))

The issue might be that all child processes are alive because you haven't called `p.stdin.close()` and gnuplot's stdin might be fully buffered when redirected to a pipe i.e., `gnuplot` processes might be stuck awaiting input. And/or your application uses too many file descriptors (file descriptors are inherited by child processes by default on Python 2.7) without releasing them.

If input doesn't depend on the output and the input is limited in size then use `.communicate()`:

from subprocess import Popen, PIPE, STDOUT

p = Popen("gnuplot", stdin=PIPE, stdout=PIPE, stderr=PIPE,
          close_fds=True, # to avoid running out of file descriptors
          bufsize=-1, # fully buffered (use zero (default) if no p.communicate())
          universal_newlines=True) # translate newlines, encode/decode text
out, err = p.communicate("\n".join(['set terminal gif;', contents]))

`.communicate()` writes all input and reads all output (concurrently, so there is no deadlock) then closes p.stdin, p.stdout, p.stderr (even if input is small and gnuplot's side is fully buffered; EOF flushes the buffer) and waits for the process to finish (no zombies).

`Popen` calls `_cleanup()` in its constructor that polls exit status of all known subprocesses i.e., even if you won't call `p.wait()` there shouldn't be many zombie processes (dead but with unread status).

Problem

In Python, I spawn a `gnuplot` process to generate gif images from a data set. ``` from subprocess import Popen, PIPE def gnuplotter(...) p = Popen([GNUPLOT], shell=False, stdin=PIPE, stdout=PIPE) p.stdin.write(r'set terminal gif;') ... p.stdin.write(contents) p.stdout.close() ``` It works fine when I use `gnuplotter()` one time, but when I launch the process multiple times, I got `Resource temporarily unavailable` error. ``` for i in range(54): gnuplotter(i, ... File "/Users/smcho/code/PycharmProjects/contextAggregator/aggregation_analyzer/aggregation_analyzer/gnuplotter.py", line 48, in gnuplotter p = Popen([GNUPLOT], shell=False, stdin=PIPE, stdout=PIPE) File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py", line 711, in __init__ errread, errwrite) File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py", line 1205, in _execute_child self.pid = os.fork() OSError: [Errno 35] Resource temporarily unavailable ``` What's wrong, and how can I close gnuplot process before spewing another one?

Original source