Python read from subprocess stdout and stderr separately while preserving order
python, stderr, stdout, subprocess
Solution
Here's a solution based on `selectors`, but one that preserves order, and streams variable-length characters (even single chars).
The trick is to use `read1()`, instead of `read()`.
import selectors
import subprocess
import sys
p = subprocess.Popen(
["python", "random_out.py"], stdout=subprocess.PIPE, stderr=subprocess.PIPE
)
sel = selectors.DefaultSelector()
sel.register(p.stdout, selectors.EVENT_READ)
sel.register(p.stderr, selectors.EVENT_READ)
while True:
for key, _ in sel.select():
data = key.fileobj.read1().decode()
if not data:
exit()
if key.fileobj is p.stdout:
print(data, end="")
else:
print(data, end="", file=sys.stderr)
If you want a test program, use this.
import sys
from time import sleep
for i in range(10):
print(f" x{i} ", file=sys.stderr, end="")
sleep(0.1)
print(f" y{i} ", end="")
sleep(0.1)
Problem
I have a python subprocess that I'm trying to read output and error streams from. Currently I have it working, but I'm only able to read from `stderr` after I've finished reading from `stdout`. Here's what it looks like: ``` process = subprocess.Popen(command, stdout=subprocess.PIPE, stderr=subprocess.PIPE) stdout_iterator = iter(process.stdout.readline, b"") stderr_iterator = iter(process.stderr.readline, b"") for line in stdout_iterator: # Do stuff with line print line for line in stderr_iterator: # Do stuff with line print line ``` As you can see, the `stderr` for loop can't start until the `stdout` loop completes. How can I modify this to be able to read from both in the correct order the lines come in? To clarify: I still need to be able to tell whether a line came from `stdout` or `stderr` because they will be treated differently in my code.