Python - Limit amount of data subprocess.Popen can produce

Question

I found lots of similar questions asking size of an object at run time in python. Some of the answers suggests to set a limit on amount of memory of sub-process. I do not want to set a limit on memory of sub-process. Here is what I want -- I'm using `subprocess.Popen()` to execute an external program. I can, very well, get standard output and error with `process.stdout.readlines()` and `process.stderr.readlines()` after the process is complete. I have a problem when an erroneous program gets into an infinite loop and keeps producing output. Since `subprocess.Popen()` stores output data in memory this infinite loop quickly eats up entire memory and program slows down. One solution is that I can run the command with timeout. But programs take variable time to complete. Large timeout, for a program taking small time and having an infinite loop, defeats the purpose of having it. Is there any simple way where I can put an upper limit say 200MB on amount of data the command can produce? If…

Accepted Answer

First: It is not `subprocess.Popen()` storing the data, but it is the pipe between "us" and "our" subprocess.

You shouldn't use `readlines()` in this case as this will indefinitely buffer the data and only at the end return them as a list (in this case, it is indeed this function which stores the data).

If you do something like

```
bytes = lines = 0
for line in process.stdout:
    bytes += len(line)
    lines += 1
    if bytes > 200000000 or lines > 10000:
        # handle the described situation
        break
```

you can act as wanted in your question. But you shouldn't forget to kill the subprocess afterwards in order to stop it producing further data.

But if you want to take care of `stderr` as well, you'd have to try to replicate `process.communicate()`'s behaviour with `select()` etc., and act appropriately.

Python - Limit amount of data subprocess.Popen can produce

Solution

Problem