Is it possible to add a where clause with list comprehension?

Question

Consider the following list comprehension ``` [ (x,f(x)) for x in iterable if f(x) ] ``` This filters the iterable based a condition `f` and returns the pairs of `x,f(x)`. The problem with this approach is `f(x)` is calculated twice. It would be great if we could write like ``` [ (x,fx) for x in iterable if fx where fx = f(x) ] or [ (x,fx) for x in iterable if fx with f(x) as fx ] ``` But in python we have to write using nested comprehensions to avoid duplicate call to f(x) and it makes the comprehension look less clear ``` [ (x,fx) for x,fx in ( (y,f(y) for y in iterable ) if fx ] ``` Is there any other way to make it more pythonic and readable? Update Coming soon in python 3.8! PEP ``` # Share a subexpression between a comprehension filter clause and its output filtered_data = [y for x in data if (y := f(x)) is not None] ```

Accepted Answer

You seek to have `let`-statement semantics in python list comprehensions, whose scope is available to both the `___ for..in`(map) and the `if ___`(filter) part of the comprehension, and whose scope depends on the `..for ___ in...`.

Your solution, modified: Your (as you admit unreadable) solution of `[ (x,fx) for x,fx in ( (y,f(y) for y in iterable ) if fx ]` is the most straightforward way to write the optimization.

Main idea: lift x into the tuple (x,f(x)).

Some would argue the most "pythonic" way to do things would be the original `[(x,f(x)) for x in iterable if f(x)]` and accept the inefficiencies.

You can however factor out the `((y,fy) for y in iterable)` into a function, if you plan to do this a lot. This is bad because if you ever wish to have access to more variables than `x,fx` (e.g. `x,fx,ffx`), then you will need to rewrite all your list comprehensions. Therefore this isn't a great solution unless you know for sure you only need `x,fx` and plan to reuse this pattern.

Generator expression:

Main idea: use a more complicated alternative to generator expressions: one where python will let you write multiple lines.

You could just use a generator expression, which python plays nicely with:

```
def xfx(iterable):
    for x in iterable:
        fx = f(x)
        if fx:
            yield (x,fx)

xfx(exampleIterable)
```

This is how I would personally do it.

Memoization/caching:

Main idea: You could also use(abuse?) side-effects and make `f` have a global memoization cache, so you don't repeat operations.

This can have a bit of overhead, and requires a policy of how large the cache should be and when it should be garbage-collected. Thus this should only be used if you'd have other uses for memoizing f, or if f is very expensive. But it would let you write...

```
[ (x,f(x)) for x in iterable if f(x) ]
```

...like you originally wanted without the performance hit of doing the expensive operations in `f` twice, even if you technically call it twice. You can add a `@memoized` decorator to `f`: example (without maximum cache size). This will work as long as x is hashable (e.g. a number, a tuple, a frozenset, etc.).

Dummy values:

Main idea: capture fx=f(x) in a closure and modify the behavior of the list comprehension.

```
filterTrue(
    (lambda fx=f(x): (x,fx) if fx else None)() for x in iterable
)
```

where filterTrue(iterable) is filter(None, iterable). You would have to modify this if your list type (a 2-tuple) was actually capable of being `None`.

Is it possible to add a where clause with list comprehension?

Solution

Problem