Python subprocess .check_call vs .check_output
bash, python, ssh, subprocess
Solution
`check_call()` returns as soon as `/bin/sh` process exits without waiting for descendant processes (assuming `shell=True` as in your case).
`check_output()` waits until all output is read. If `ssh` inherits the pipe then `check_output()` will wait until it exits (until it closes its inherited pipe ends).
`check_call()` code example:
#!/usr/bin/env python
import subprocess
import sys
import time
start = time.time()
cmd = sys.executable + " -c 'import time; time.sleep(2)' &"
subprocess.check_call(cmd, shell=True)
assert (time.time() - start) < 1
The output is not read; `check_call()` returns immediately without waiting for the grandchild background python process.
`check_call()` is just `Popen().wait()`. `Popen()` starts the external process and returns immediately without waiting for it to exit. `.wait()` collects the exit status for the process -- it doesn't wait for other (grandchildren) processes.
If the output is read (it is redirected and the grandchild python process inherits the stdout pipe):
start = time.time()
subprocess.check_output(cmd, shell=True)
assert (time.time() - start) > 2
then it waits until the background python process that inherited the pipe exits.
`check_output()` calls `Popen().communicate()`, to get the output. `.communicate()` calls `.wait()` internally i.e., `check_output()` also waits for the shell to exit and `check_output()` waits for EOF.
If the grandchild doesn't inherit the pipe then `check_output()` doesn't wait for it:
start = time.time()
cmd = sys.executable + " -c 'import time; time.sleep(2)' >/dev/null &"
subprocess.check_output(cmd, shell=True)
assert (time.time() - start) < 1
Grandchild's output is redirected to `/dev/null` i.e., it doesn't inherit the parent's pipe and therefore `check_output()` may exit without waiting for it.
Note: `&` at the end which puts the grandchild python process into background. It won't work on Windows where `shell=True` starts `cmd.exe` by default.
Problem
My python script (python 3.4.3) calls a bash script via subprocess: ``` import subprocess as sp res = sp.check_output("bashscript", shell=True) ``` The bashscript contains the following line: ``` ssh -MNf somehost ``` which opens a shared master connection to some remote host to allow some subsequent operations. When executing the python script, it will prompt for password for the `ssh` line but then it blocks after the password is entered and never returns. When I ctrl-C to terminate the script, I see that the connection was properly established (so `ssh` line was successfully executed). I don't have this blocking problem when using `check_call` instead of `check_output`, but `check_call` does not retrieve stdout. I'd like to understand what exactly is causing the blocking behavior for `check_output`, probably related to some subtlety with `ssh -MNf`.