Why is subprocess.Popen not waiting until the child process terminates?

mysql, python

Solution

`subprocess.Popen`, when instantiated, runs the program. It does not, however, wait for it -- it fires it off in the background as if you'd typed `cmd &` in a shell. So, in the code above, you've essentially defined a race condition -- if the inserts can finish in time, it will appear normal, but if not you get the unexpected output. You are not waiting for your first `run()`'d PID to finish, you are simply returning its `Popen` instance and continuing.

I'm not sure how this behavior contradicts the documentation, because there's some very clear methods on Popen that seem to indicate it is not waited for, like:

Popen.wait()
  Wait for child process to terminate. Set and return returncode attribute.

I do agree, however, that the documentation for this module could be improved.

To wait for the program to finish, I'd recommend using `subprocess`'s convenience method, `subprocess.call`, or using `communicate` on a `Popen` object (for the case when you need stdout). You are already doing this for your second call.

### START MAIN
# copy some rows from a source table to a destination table
# note that the destination table is empty when this script is run
cmd = 'mysql -u ve --skip-column-names --batch --execute="insert into destination (select * from source limit 100000)" test'
subprocess.call(cmd)

# check to see how many rows exist in the destination table
cmd = 'mysql -u ve --skip-column-names --batch --execute="select count(*) from destination" test'
process = subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE)
try: count = (int(process.communicate()[0][:-1]))
except: count = 0

Additionally, in most cases, you do not need to run the command in a shell. This is one of those cases, but you'll have to rewrite your command like a sequence. Doing it that way also allows you to avoid traditional shell injection and worry less about quoting, like so:

prog = ["mysql", "-u", "ve", "--execute", 'insert into foo values ("snargle", 2)']
subprocess.call(prog)

This will even work, and will not inject as you'd expect:

prog = ["printf", "%s", "<", "/etc/passwd"]
subprocess.call(prog)

Try it interactively. You avoid the possibilities of shell injection, particularly if you're accepting user input. I suspect you're using the less-awesome string method of communicating with subprocess because you ran into trouble getting the sequences to work :^)

Problem

Original source