Processing a large .txt file in python efficiently

Question

I am quite new to python and programming in general, but I am trying to run a "sliding window" calculation over a tab delimited .txt file that contains about 7 million lines with python. What I mean by sliding window is that it will run a calculation over say 50,000 lines, report the number and then move up say 10,000 lines and perform the same calculation over another 50,000 lines. I have the calculation and the "sliding window" working correctly and it runs well if I test it on a a small subset of my data. However, if i try to run the program over my entire data set it is incredibly slow (i've had it running now for about 40 hours). The math is quite simple so I don't think it should be taking this long. The way I am reading my .txt file right now is with the csv.DictReader module. My code is as follows: ``` file1='/Users/Shared/SmallSetbee.txt' newfile=open(file1, 'rb') reader=csv.DictReader((line.replace('\0','') for line in newfile), delimiter="	") ``` I believe that this is mak…

Accepted Answer

A `collections.deque` is an ordered collection of items which can take a maximum size. When you add an item to one end, one falls of the other end. This means that to iterate over a "window" on your csv, you just need to keep adding rows to the `deque` and it will handle throwing away complete ones already.

```
dq = collections.deque(maxlen=50000)
with open(...) as csv_file:
    reader = csv.DictReader((line.replace("\0", "") for line in csv_file), delimiter="	")

# initial fill
    for _ in range(50000):
        dq.append(reader.next())

# repeated compute
    try:
        while 1:
            compute(dq)
            for _ in range(10000):
                dq.append(reader.next())
    except StopIteration:
            compute(dq)
```

Processing a large .txt file in python efficiently

Solution

Problem

Related problems