piping together several subprocesses

Question

I have 5 processes `p1,p2,...,p5` where I want to write some data to stdin of p1, pipe p1 output to p2 stdin and finally read the final result from output of p5. What I have tried so far: ``` p1 = Popen(['p1'], stdin=PIPE, stdout=PIPE) p2 = Popen(['p2'], stdin=p1.stdout, stdout=PIPE) ... p5 = Popen(['p5'], stdin=p4.stdout, stdout=PIPE) # write data to stdin p1.stdin.write(indata) p1.stdin.close() # not sure in what order to close the pipes here, if at all # read output out = p5.stdout.read() print out ``` The last code snipped just hangs, because I must be doing the read/write operations incorrectly. I was able to get working single processes using `communicate()` and two processes without suppling any input to first one (example from Python docs): ``` output=`dmesg | grep hda` ==> p1 = Popen(["dmesg"], stdout=PIPE) p2 = Popen(["grep", "hda"], stdin=p1.stdout, stdout=PIPE) p1.stdout.close() # Allow p1 to receive a SIGPIPE if p2 exits. output = p2.communicate()[0] ``` But I cannot figu…

Accepted Answer

Maybe this can help:

```
import sys
import tempfile
from subprocess import Popen, PIPE

cmd = [sys.executable, '-c', 'print raw_input()']

# Using a temp file to give input data to the subprocess instead of stdin.write to avoid deadlocks.
with tempfile.TemporaryFile() as f:
    f.write('foobar')
    f.seek(0)  # Return at the start of the file so that the subprocess p1 can read what we wrote.
    p1 = Popen(cmd, stdin=f, stdout=PIPE)

p2 = Popen(cmd, stdin=p1.stdout, stdout=PIPE)
p3 = Popen(cmd, stdin=p2.stdout, stdout=PIPE)

# No order needed.
p1.stdout.close()
p2.stdout.close()

# Using communicate() instead of stdout.read to avoid deadlocks. 
print p3.communicate()[0]
```

Output:

```
$ python test.py
foobar
```

Hope this can be hepfull.

piping together several subprocesses

Solution

Problem

Related problems