cPickle:SystemError: error return without exception set

gzip, pickle, system-error, ubuntu

Solution

`cPickle` can't be used for storing very large objects ( see http://bugs.python.org/issue11564 ).

You have several options:

- split the data into chunks and store it in multiple files

- numpy.save

- h5py <- my favorite because of its convenient numpy-like interface, plus the data are stored in `hdf5` which enables interopability with other software/languages.

- pytables

Problem

While dumping large matrix (170000*20000) as follows ``` cPickle.dump(train_set,gzip.open('train.pickle.gz','wb'), cPickle.HIGHEST_PROTOCOL) ``` I get following error: ``` SystemError: error return without exception set ``` How would I deal in this case?

Original source