Using compression with Pandas and HD5 / HDFStore
hdf5, pandas, python
Solution
see docs in regards to compression using `HDFStore`
`gzip` is not a valid compression option (and is ignored, that's a bug). try any of `zlib, bzip2, lzo, blosc` (bzip2/lzo might need extra libraries installed)
see for PyTables docs on the various compression
Heres a question semi-related.
Problem
For a few aspects of a project, using "h5" storage would be ideal. However, the files are becoming massive and frankly we're running out of space. This statement... ``` store.put(storekey, data, table=False, compression='gzip') ``` does not produce any difference in terms of file size than... ``` store.put(storekey, data, table=False) ``` Is using compression even possible when going through Pandas? ... if it isn't possible, I don't mind using h5py, however, I'm uncertain what to put for a "datatype" as the DataFrame contains all sorts of types (strings, float, int etc.) Any help/insight would be appreciated!