Using Python to parse a 12GB CSV

bigdata, csv, python, r

Solution

You could use the `csv` module to process the file line-by-line. Something like this might work:

import csv
infname = "csv.csv"
outfname = "csv_stripped.csv"
cols = ["col1", "col2", "col3"]
with open(infname) as inf, open(outfname, 'w+') as outf:
    reader = csv.DictReader(inf)
    writer = csv.DictWriter(outf, cols, extrasaction='ignore')
    writer.writeheader()
    for line in reader:
        writer.writerow(line)

For reference:

- https://docs.python.org/2/library/csv.html

Problem

I have a 12 GB CSV file. I'm hoping to extract only some columns from this data and then write a new CSV that hopefully I can load into R for analysis. The problem is that I'm getting a memory error when trying to load the entire list at once before writing the new CSV file. How can I parse the data row by row and then create a CSV output? Here is what I have so far: ``` import pandas colnames = ['contributor name', 'recipient name', 'recipient party', 'contributor cfscore', 'candidate cfscore', 'amount'] DATA = pandas.read_csv('pathname\filename.csv', names=colnames) DATA.to_csv(''pathname\filename.csv', cols = colnames) ```

Original source

Related problems