Read Large Gzip Files in Python

Question

I am trying to read a gzip file (with size around 150 MB) and using this script (which I know is badly written): ``` import gzip f_name = 'file.gz' a = [] with gzip.open(f_name, 'r') as infile: for line in infile: a.append(line.split(' ')) new_array1 = [] for l in a: for i in l: if i.startswith('/bin/movie/tribune'): new_array1.append(l) filtered = [] for q in range(0, len(new_array1)): filtered.append(new_array1[q]) #at this point filtered array can be printed ``` The problem is that I am able to read files upto 50 MB using this technique into an array, but file sizes from 80 MB and above are not readable. Is there some problem with a technique that I am using or is there a memory constraint? If this is the second case, then what should be the best technique to read a large gz file (above 100 MB) in python array? Any help will be appreciated. Note: I am not using NumPy because I ran into some serious issues with C compilers on my server which are required for numpy and therefore I am…

Accepted Answer

My guess is that the problem is constructing `a` in your code, as that will undoubtedly contain a massive number of entries if your .gz is that large. This modification should solve that problem:

```
import gzip

f_name = 'file.gz'

filtered = []
with gzip.open(f_name, 'r') as infile:
    for line in infile:
        for i in line.split(' '):
            if i.startswith('/bin/movie/tribune'):
                filtered.append(line)
                break # to avoid duplicates
```

Read Large Gzip Files in Python

Solution

Problem