(list|set|dict) comprehension containing a yield expression does not return a (list|set|dict)
generator, python
Solution
List (set, dict) comprehensions translate to a different code structure from generator expressions. Let's look at a set comprehension:
def f():
return {i for i in range(10)}
dis.dis(f.__code__.co_consts[1])
2 0 BUILD_SET 0
3 LOAD_FAST 0 (.0)
>> 6 FOR_ITER 12 (to 21)
9 STORE_FAST 1 (i)
12 LOAD_FAST 1 (i)
15 SET_ADD 2
18 JUMP_ABSOLUTE 6
>> 21 RETURN_VALUE
Compare to the equivalent generator expression:
def g():
return (i for i in range(10))
dis.dis(g.__code__.co_consts[1])
2 0 LOAD_FAST 0 (.0)
>> 3 FOR_ITER 11 (to 17)
6 STORE_FAST 1 (i)
9 LOAD_FAST 1 (i)
12 YIELD_VALUE
13 POP_TOP
14 JUMP_ABSOLUTE 3
>> 17 LOAD_CONST 0 (None)
20 RETURN_VALUE
You'll notice that where the generator expression has a `yield`, the set comprehension stores a value directly into the set it is building.
This means that if you add a `yield` expression into the body of a generator expression, it is treated indistinguishably from the `yield` that the language constructs for the generator body; as a result, you get two (or more) values per iteration.
However, if you add a `yield` to a list (set, dict) comprehension then the comprehension is transformed from a function building a list (set, dict) into a generator that executes the `yield` statements then returns the constructed list (set, dict). The `{None}` in the set comprehension result is the set built from each of the `None`s that the `yield` expressions evaluate to.
Finally, why does Python 3.3 not produce a `{None}`? (Note that previous versions of Python 3 do.) It's because of PEP 380 (a.k.a. `yield from` support). Prior to Python 3.3, a `return` in a generator is a `SyntaxError: 'return' with argument inside generator`; our `yield`ing comprehensions are therefore exploiting undefined behaviour, but the actual result of the `RETURN_VALUE` opcode is to just generate another (final) value from the generator. In Python 3.3, `return value` is explicitly supported; a `RETURN_VALUE` opcode results in a `StopIteration` being raised, which has the effect of stopping the generator without producing a final value.
Problem
Python 3.3 I've constructed this slightly cryptic piece of python 3.3: ``` >>> [(yield from (i, i + 1, i)) for i in range(5)] <generator object <listcomp> at 0x0000008666D96900> >>> list(_) [0, 1, 0, 1, 2, 1, 2, 3, 2, 3, 4, 3, 4, 5, 4] ``` If I use a generator comprehension inside a list constructor, I get a different result: ``` >>> list((yield from (i, i + 1, i)) for i in range(5)) [0, 1, 0, None, 1, 2, 1, None, 2, 3, 2, None, 3, 4, 3, None, 4, 5, 4, None] ``` Why isn't the list comprehension returning a list? Python 2.7 I can get a similarly odd effect in python 2 (using a set comprehension, because list comprehensions have odd scope): ``` >>> {(yield i) for i in range(5)} <generator object <setcomp> at 0x0000000004A06120> >>> list(_) [0, 1, 2, 3, 4, {None}] ``` And when using a generator comprehension: ``` >>> list((yield i) for i in range(5)) [0, None, 1, None, 2, None, 3, None, 4, None] ``` Where'd that `{None}` come from?