Can I stream a Python pickle list, tuple, or other iterable data type?

ipython, pickle, python, streaming

Solution

This would work.

What is does however is unpickle one object from the file, and then print the rest of the file's content to `stdout`

What you could do is something like:

import cPickle
with open( 'big_pickled_list.pkl' ) as p:
    try:
        while True:
            print cPickle.load(p)
    except EOFError:
        pass

That would unpickle all objects from the file until reaching EOF.

If you want something that works like `for line in f:`, you can wrap this up easily:

def unpickle_iter(file):
    try:
        while True:
             yield cPickle.load(file)
    except EOFError:
        raise StopIteration

Now you can just do this:

with open('big_pickled_list.pkl') as file:
    for item in unpickle_iter(file):
        # use item ...

Problem

I work with comma/tab-separated data files often that might look like this: ``` key1,1,2.02,hello,4 key2,3,4.01,goodbye,6 ... ``` I might read and pre-process this in Python into a list of lists, like this: ``` [ [ key1, 1, 2.02, 'hello', 4 ], [ key2, 3, 4.01, 'goodbye', 6 ] ] ``` Sometimes, I like saving this list of lists as a pickle, since it preserves the different types of my entries. If the pickled file is big, though, it would be great to read this list of lists back in a streaming fashion. In Python, to load a text file as a stream, I use the follwoing to print out each line: ``` with open( 'big_text_file.txt' ) as f: for line in f: print line ``` Can I do something similar for a Python list, i.e.: ``` import pickle with open( 'big_pickled_list.pkl' ) as p: for entry in pickle.load_streaming( p ): # note: pickle.load_streaming doesn't exist print entry ``` Is there a pickle function like "load_streaming"?

Original source