Python's subprocessing with pipes and large files

Question

I'm trying to use python + ffmpeg + oggenc to convert any audiofile to ogg. The program works, almost. But for big files (i think > ~6mb) the ffmpeg process starts to sleep at pipe_wait. I don't know which pipe it waits for. If I kill the ffmpeg process, the oggenc process continues and I get a resulting ogg-file with about ~2:40 of all the sound. ``` #!/usr/bin/env python # -*- coding: utf-8 -*- from subprocess import Popen, PIPE from sys import argv ffmpeg = Popen([ "ffmpeg", "-i", argv[1], "-vcodec", "null", "-acodec", "pcm_s16le", "-ac", "2", "-ab", "44100", "-f", "wav", "-" ],stdout = PIPE,stderr = PIPE) oggenc = Popen([ "oggenc", "-", "--raw", "-q", "4", "-o", argv[2] ],stdin = ffmpeg.stdout,stderr = PIPE) oggenc.communicate() ffmpeg.communicate() ``` EDIT: Thought I might add that this works perfectly: ``` #!/bin/bash ffmpeg -i "$1" -vcodec null -acodec pcm_s16le -ac 2 -ab 44100 -f wav - | oggenc - --raw -q 4 -o "$2" ```

Accepted Answer

What exactly do you do with the `stderr` channels of the two pipes?

Encoders/decoders typically produce lots of stderr output, as status updates; this output is piped to your process, and buffers will become full. Perhaps you should add some dummy `ffmpeg.stderr.read()` call before the (useless, I think) `.communicate` calls, or even better, drop the `stderr=PIPE` arguments completely.

UPDATE

For the `>/dev/null` equivalent, do the following:

```
nulfp = open(os.devnull, "w")
…
… = subprocess.Popen(…, stderr=nulfp.fileno())
```

Obviously, you can re-use the same `nulfp` for all `stderr`s you want to ignore.

Python's subprocessing with pipes and large files

Solution

Problem