live output from subprocess command

error-handling, logging, python, shell, subprocess

Solution

TLDR for Python 3:

import subprocess
import sys

with open("test.log", "wb") as f:
    process = subprocess.Popen(your_command, stdout=subprocess.PIPE)
    for c in iter(lambda: process.stdout.read(1), b""):
        sys.stdout.buffer.write(c)
        f.buffer.write(c)

You have two ways of doing this, either by creating an iterator from the `read` or `readline` functions and do:

import subprocess
import sys

# replace "w" with "wb" for Python 3
with open("test.log", "w") as f:
    process = subprocess.Popen(your_command, stdout=subprocess.PIPE)
    # replace "" with b'' for Python 3
    for c in iter(lambda: process.stdout.read(1), ""):
        sys.stdout.write(c)
        f.write(c)

or

import subprocess
import sys

# replace "w" with "wb" for Python 3
with open("test.log", "w") as f:
    process = subprocess.Popen(your_command, stdout=subprocess.PIPE)
    # replace "" with b"" for Python 3
    for line in iter(process.stdout.readline, ""):
        sys.stdout.write(line)
        f.write(line)

Or you can create a `reader` and a `writer` file. Pass the `writer` to the `Popen` and read from the `reader`

import io
import time
import subprocess
import sys

filename = "test.log"
with io.open(filename, "wb") as writer, io.open(filename, "rb", 1) as reader:
    process = subprocess.Popen(command, stdout=writer)
    while process.poll() is None:
        sys.stdout.write(reader.read())
        time.sleep(0.5)
    # Read the remaining
    sys.stdout.write(reader.read())

This way you will have the data written in the `test.log` as well as on the standard output.

The only advantage of the file approach is that your code doesn't block. So you can do whatever you want in the meantime and read whenever you want from the `reader` in a non-blocking way. When you use `PIPE`, `read` and `readline` functions will block until either one character is written to the pipe or a line is written to the pipe respectively.

Problem

I'm using a python script as a driver for a hydrodynamics code. When it comes time to run the simulation, I use `subprocess.Popen` to run the code, collect the output from `stdout` and `stderr` into a `subprocess.PIPE` --- then I can print (and save to a log-file) the output information, and check for any errors. The problem is, I have no idea how the code is progressing. If I run it directly from the command line, it gives me output about what iteration its at, what time, what the next time-step is, etc. Is there a way to both store the output (for logging and error checking), and also produce a live-streaming output? The relevant section of my code: ``` ret_val = subprocess.Popen( run_command, stdout=subprocess.PIPE, stderr=subprocess.PIPE, shell=True ) output, errors = ret_val.communicate() log_file.write(output) print output if( ret_val.returncode ): print "RUN failed\n\n%s\n\n" % (errors) success = False if( errors ): log_file.write("\n\n%s\n\n" % errors) ``` Originally I was piping the `run_command` through `tee` so that a copy went directly to the log-file, and the stream still output directly to the terminal -- but that way I can't store any errors (to my knowlege). My temporary solution so far: ``` ret_val = subprocess.Popen( run_command, stdout=log_file, stderr=subprocess.PIPE, shell=True ) while not ret_val.poll(): log_file.flush() ``` then, in another terminal, run `tail -f log.txt` (s.t. `log_file = 'log.txt'`).

Original source

Related problems