itertools.product slower than nested for loops

Question

I am trying using the `itertools.product` function to make a segment of my code (in an isotopic pattern simulator) easier to read and hopefully faster as well (the documentation states that no intermediate results are created) , I have however tested both versions of the code against each other using the `cProfiling` library and noticed that the `itertools.product` was significantly slower than my nested for loops. Example values used for the testing: ``` carbons = [(0.0, 0.004613223957020534), (1.00335, 0.02494768843632857), (2.0067, 0.0673219412049374), (3.0100499999999997, 0.12087054681917497), (4.0134, 0.16243239687902825), (5.01675, 0.17427700732161705), (6.020099999999999, 0.15550695260604208), (7.0234499999999995, 0.11869556397525197), (8.0268, 0.07911287899598853), (9.030149999999999, 0.04677626606764402)] hydrogens = [(0.0, 0.9417611429667746), (1.00628, 0.05651245007201512)] nitrogens = [(0.0, 0.16148864310897554), (0.99703, 0.2949830688288726), (1.99406, 0.26887643366755537…

Accepted Answer

Your original itertool code spent a lot extra time in the needless `lambda`, and building lists of intermediate values by hand - a lot of this can be replaced with builtin functionality.

Now, the inner for loop does add quite a lot extra overhead: just try the following and the performance is very much on par with your original code:

```
for a in itertools.product(carbons,hydrogens,nitrogens,oxygens17,
                           oxygens18,sulfurs33,sulfurs34,sulfurs36):
    i, j, k, l, m, n, o, p = a
    totals.append((i[0]+j[0]+k[0]+l[0]+m[0]+n[0]+o[0]+p[0],
                   i[1]*j[1]*k[1]*l[1]*m[1]*n[1]*o[1]*p[1]))
```

The following code runs as much as possible in the CPython builtin side, and I tested it to be equivalent to with code. Notably the code uses `zip(*iterable)` to unzip each of the product results; then uses the `reduce` with `operator.mul` for product, and `sum` for summing; 2 generators for going through the lists. The for loop still beats slightly, but being hardcoded it probably is not what you can use in the long run.

```
import itertools
from operator import mul
from functools import partial

prod = partial(reduce, mul)
elems = carbons, hydrogens, nitrogens, oxygens17, oxygens18, sulfurs33, sulfurs34, sulfurs36
p = itertools.product(*elems)

totals = [
    ( sum(massdiffs), prod(chances) )
    for massdiffs, chances in
    ( zip(*i) for i in p )
]
```

itertools.product slower than nested for loops

Solution

Problem