Difference between iterating over a file-like and calling readline
io, iterator, pipe, python, subprocess
Solution
The difference is purely in the implementation of iteration versus the `readline` method. File iteration reads in blocks (of 8 kilobytes, by default) and then splits up the buffer into lines as you consume them. The `readline` method, on the other hand, takes care never to read more than one line, and that means reading character by character. Reading in blocks is much more efficient, but it means you can't mix other operations on the file between reads. The expectation is that when you are iterating over the file, your intent is to read all lines sequentially and you will not be doing other operations on it. The `readline` method can't make that assumption.
As Sven Marnach hinted in his comment to your question, you can use `iter(f.readline, '')` to get an iterator that reads lines from the file without reading in blocks, at the cost of performance.
Problem
I always thought iterating over a file-like in Python would be equivalent to calling its `readline` method in a loop, but today I found a situation where that is not true. Specifically, I have a `Popen`'d process `p` where ``` list(itertools.takewhile(lambda x: x != "\n", p.stdout)) ``` hangs (presumably because `p` waits for input; both `stdin` and `stdout` are pipes to my Python process), while the following works: ``` list(itertools.takewhile(lambda x: x != "\n", iter(p.stdout.readline, ""))) ``` Can someone explain the difference?