Processing only non-blank lines

Question

I've got the following snippet of code ``` def send(self, queue, fd): for line in fd: data = line.strip() if data: queue.write(json.loads(data)) ``` Which of course works just fine, but I wonder sometimes if there is a "better" way to write that construct where you only act on non-blank lines. The challenge is this should use the iterative nature of the for the 'fd' read and be able to handle files in the 100+ MB range. UPDATE - In your haste to get points for this question you're ignoring an import part, which is memory usage. For instance the expression: ``` non_blank_lines = (line.strip() for line in fd if line.strip()) ``` Is going to buffer the whole file into memory, not to mention performing a strip() action twice. Which will work for small files, but fails when you've got 100+MB of data (or once in a while a 100GB). Part of the challenge is the following works, but is soup to read: ``` for line in ifilter(lambda l: l, imap(lambda l: l.strip(), fd)): queue.write(json.loads(line…

Accepted Answer

There's nothing wrong with the code as written, it's readable and efficient.

An alternative approach would be to write it as a generator comprehension:

```
def send(self, queue, fd):
    non_blank_lines = (line.strip() for line in fd if line.strip())
    for line in non_blank_lines:
        queue.write(json.loads(data))
```

This approach can be beneficial (terser) if you are applying a function that can take an iterator: e.g. python3 print

```
non_blank_lines = (line.strip() for line in fd if line.strip())
print(*non_blank_lines, file='foo')
```

To do away with the multiple calls to strip(), chain together generator comprehensions

```
stripped_lines = (line.strip() for line in fd)
non_blank_lines = (line for line in stripped_lines if line)
```

Note that generator expressions will not adversely affect memory as detailed in this pep.

For a more in depth look at this approach, and some performance bench marks, take a look at this set of notes.

Finally note that rstrip() will outperform strip() if you don't need the full behaviour of strip().

Processing only non-blank lines

Solution

Problem