How to read a CSV file in reverse order in Python?

csv, python

Solution

Pretty much the same way as for a text file: read the whole thing into a list and then go backwards:

import csv
with open('test.csv', 'r') as textfile:
    for row in reversed(list(csv.reader(textfile))):
        print ', '.join(row)

If you want to get fancy, you could write a lot of code that reads blocks starting at the end of the file and working backwards, emitting a line at a time, and then feed that to `csv.reader`, but that will only work with a file that can be seeked, i.e. disk files but not standard input.

Some of us have files that do not fit into memory, could anyone come with a solution that does not require storing the entire file in memory?

That's a bit trickier. Luckily, all `csv.reader` expects is an iterator-like object that returns a string (line) per call to `next()`. So we grab the technique Darius Bacon presented in "Most efficient way to search the last x lines of a file in python" to read the lines of a file backwards, without having to pull in the whole file:

import os

def reversed_lines(file):
    "Generate the lines of file in reverse order."
    part = ''
    for block in reversed_blocks(file):
        for c in reversed(block):
            if c == '\n' and part:
                yield part[::-1]
                part = ''
            part += c
    if part: yield part[::-1]

def reversed_blocks(file, blocksize=4096):
    "Generate blocks of file's contents in reverse order."
    file.seek(0, os.SEEK_END)
    here = file.tell()
    while 0 < here:
        delta = min(blocksize, here)
        here -= delta
        file.seek(here, os.SEEK_SET)
        yield file.read(delta)

and feed `reversed_lines` into the code to reverse the lines before they get to `csv.reader`, removing the need for `reversed` and `list`:

import csv
with open('test.csv', 'r') as textfile:
    for row in csv.reader(reversed_lines(textfile)):
        print ', '.join(row)

There is a more Pythonic solution possible, which doesn't require a character-by-character reversal of the block in memory (hint: just get a list of indices where there are line ends in the block, reverse it, and use it to slice the block), and uses `chain` out of `itertools` to glue the line clusters from successive blocks together, but that's left as an exercise for the reader.

It's worth noting that the reversed_lines() idiom above only works if the columns in the CSV file don't contain newlines.

Aargh! There's always something. Luckily, it's not too bad to fix this:

def reversed_lines(file):
    "Generate the lines of file in reverse order."
    part = ''
    quoting = False
    for block in reversed_blocks(file):
        for c in reversed(block):
            if c == '"':
                quoting = not quoting
            elif c == '\n' and part and not quoting:
                yield part[::-1]
                part = ''
            part += c
    if part: yield part[::-1]

Of course, you'll need to change the quote character if your CSV dialect doesn't use `"`.

Problem

Original source

Related problems

• Most efficient way to search the last X lines of a file?