Understanding list comprehension for flattening list of lists in Python
generator, list, list-comprehension, python
Solution
List Comprehension
When I first started with list comprehension, I read that like English sentences and I was able to easily understand them. For example,
[item for sublist in list_of_lists for item in sublist]
can be read like
for each sublist in list_of_lists and for each item in sublist add item
Also, the filtering part can be read as
for each sublist in list_of_lists and for each item in sublist add item only if it is valid
And the corresponding comprehension would be
[item for sublist in list_of_lists for item in sublist if valid(item)]
Generators
They are like land mines, triggered only when invoked with the `next` protocol. They are similar to functions, but till an exception is raised or the end of function is reached, they are not exhausted and they can be invoked again and again. The important thing is, they retain the state between the previous invocation and the current.
The difference between a generator and a function is that, generators use `yield` keyword to give the value back to the invoker. In case of a generator expression, they are similar to the list comprehension, the fist expression is the actual value being "yielded".
With this basic understanding, if we look at your expressions in the question,
[(item for sublist in list_of_lists) for item in sublist]
You are mixing list comprehension with the generator expressions. This will be read like this
for each item in sublist add a generator expression which is defined as, for every sublist in list_of_lists yield item
which is not what you had in your mind. And since the generator expression is not iterated, the generator expression object is added in the list as it is. Since they will not be evaluated without being invoked with the next protocol, they will not produce any error (if there are any, unless they have syntax error). In this case, it will produce runtime error as `sublist` is not defined yet.
Also, in the last case,
[item for sublist in (list_of_lists for item in sublist)]
for each sublist in the generator expression, add item and the generator expression is defined as for each item in sublist yield list_of_lists.
The for loop will iterate any iterable with the next protocol. So, the generator expression will be evaluated and the `item` will always be the last element in the iteration of the `sublist` and you are adding that in the list. This will also produce runtime error, since sublist is not defined yet.
Problem
I found this comprehension that works perfectly for flattening a list of lists: ``` >>> list_of_lists = [(1,2,3),(2,3,4),(3,4,5)] >>> [item for sublist in list_of_lists for item in sublist] [1, 2, 3, 2, 3, 4, 3, 4, 5] ``` I like this better than using `itertools.chain()`, but I just can't understand it. I've tried surrounding parts with parentheses, to see if I could reduce the complexity, but now I'm just more confused: ``` >>> [(item for sublist in list_of_lists) for item in sublist] [<generator object <genexpr> at 0x7ff919fdfd20>, <generator object <genexpr> at 0x7ff919fdfd70>, <generator object <genexpr> at 0x7ff919fdfdc0>] >>> [item for sublist in (list_of_lists for item in sublist)] [5, 5, 5] ``` I get this feeling that I'm having a hard time understanding, because I don't quite understand how generators work... I mean, I thought I did, but now I'm seriously in doubt. Like I said, I love how compact this idiom is, and it's exactly what I need, but I'm loathe to use code that I don't understand. What exactly is happening here?