How to replicate tee behavior in Python when using subprocess?

Question

I'm looking for a Python solution that will allow me to save the output of a command in a file without hiding it from the console. FYI: I'm asking about tee (as the Unix command line utility) and not the function with the same name from Python intertools module. Details - Python solution (not calling `tee`, it is not available under Windows) - I do not need to provide any input to stdin for called process - I have no control over the called program. All I know is that it will output something to stdout and stderr and return with an exit code. - To work when calling external programs (subprocess) - To work for both `stderr` and `stdout` - Being able to differentiate between stdout and stderr because I may want to display only one of the to the console or I could try to output stderr using a different color - this means that `stderr = subprocess.STDOUT` will not work. - Live output (progressive) - the process can run for a long time, and I'm not able to wait for it to finish. - Python 3…

Accepted Answer

If requiring python 3.6 isn't an issue there is now a way of doing this using `asyncio`. This method allows you to capture stdout and stderr separately but still have both stream to the tty without using threads. Here's a rough outline:

```
class RunOutput:
    def __init__(self, returncode, stdout, stderr):
        self.returncode = returncode
        self.stdout = stdout
        self.stderr = stderr

async def _read_stream(stream, callback):
    while True:
        line = await stream.readline()
        if line:
            callback(line)
        else:
            break

async def _stream_subprocess(cmd, stdin=None, quiet=False, echo=False) -> RunOutput:
    if isWindows():
        platform_settings = {"env": os.environ}
    else:
        platform_settings = {"executable": "/bin/bash"}
    if echo:
        print(cmd)
    p = await asyncio.create_subprocess_shell(
        cmd,
        stdin=stdin,
        stdout=asyncio.subprocess.PIPE,
        stderr=asyncio.subprocess.PIPE,
        **platform_settings
    )
    out = []
    err = []

def tee(line, sink, pipe, label=""):
        line = line.decode("utf-8").rstrip()
        sink.append(line)
        if not quiet:
            print(label, line, file=pipe)

await asyncio.wait(
        [
            _read_stream(p.stdout, lambda l: tee(l, out, sys.stdout)),
            _read_stream(p.stderr, lambda l: tee(l, err, sys.stderr, label="ERR:")),
        ]
    )

return RunOutput(await p.wait(), out, err)

def run(cmd, stdin=None, quiet=False, echo=False) -> RunOutput:
    loop = asyncio.get_event_loop()
    result = loop.run_until_complete(
        _stream_subprocess(cmd, stdin=stdin, quiet=quiet, echo=echo)
    )

return result
```

The code above was based on this blog post: https://kevinmccarthy.org/2016/07/25/streaming-subprocess-stdin-and-stdout-with-asyncio-in-python/

How to replicate tee behavior in Python when using subprocess?

Solution

Problem

Related problems