With python csv module, why does creating a list of identical values speed up creation of a list of rows?
csv, python
Solution
Have you looked at `b` in your first example? It's empty because `r` was exhausted by the first list comprehension. All rows have already been iterated over, and - as @soulcheck has pointed out - it's much faster to create a list of 4 million `None`s than a list that contains 4 million sublists.
Problem
I'm reading a large csv file (more than 4 million rows) using the invaluable `csv` module in Python. In timing various approaches, I've come across an unintuitive result. If I run the following script it takes about 11-12 seconds. `b` is created almost instantly after `a`. ``` r = csv.reader(open("data.csv", "rb"), delimiter=";") a = [None for row in r] b = [row for row in r] ``` But if I run a similar script that doesn't create `a` at all, the code takes longer (21-22 seconds): ``` r = csv.reader(open("data.csv", "rb"), delimiter=";") b = [row for row in r] ``` I can comprehend why the creation of `b` takes almost no time after `a` has already been created. But I would have thought (naively) that the second code block, in which only `b` is created and not `a`, would be the faster script. At the risk of appearing un-Pythonic, I'm interested to know if anyone can explain why creating `a` and then `b` is almost twice as fast as creating `b` alone. Furthermore, if this speed boost is consistent across more complicated operations, are there good reasons (other than style/readibility concerns) not to take advantage of it? Are savvier Python programmers than I already achieving the same time savings with some conventional method I've never heard of? If I construct `a` using, say, an integer instead of `None`, I get the same result. If instead of iterating over a `csv.reader` object I iterate over `open("data.csv", "rb").readlines()`, the timing is as I expect it to be: creating `b` alone is faster than creating `a` then `b`. So the time disparity presumably has something to do with the properties of a `csv.reader` object, or of a more general class of objects like it. If I create `b` prior to `a`, the time is about the same as if I created `b` alone. Some notes: - Creating `b` prior to `a` takes the same time as creating `b` alone. - I'm not running these line-by-line in interactive mode. I'm running each as a separate script. - I'm not really trying to create a list full of ones with the same length as `r`, or a list of the rows in `r`. - In case it matters, I'm running Python 2.7.3, using the Enthought Python distribution 7.3-2, on 64-bit Windows 7.