How to save a pandas dataframe in gzipped format directly?
gzip, pandas, python
Solution
Better serialization with compression has recently been added to Pandas. (Starting in pandas 0.20.0.) Here is an example of how it can be used:
df.to_csv("my_file.gz", compression="gzip")
For more information, such as different forms of compression available, check out the docs.
Problem
I have a pandas data frame, called `df`. I want to save this in a gzipped format. One way to do this is the following: ``` import gzip import pandas df.save('filename.pickle') f_in = open('filename.pickle', 'rb') f_out = gzip.open('filename.pickle.gz', 'wb') f_out.writelines(f_in) f_in.close() f_out.close() ``` However, this requires me to first create a file called `filename.pickle`. Is there a way to do this more directly, i.e., without creating the `filename.pickle`? When I want to load the dataframe that has been gzipped I have to go through the same step of creating filename.pickle. For example, to read a file `filename2.pickle.gzip`, which is a gzipped pandas dataframe, I know of the following method: ``` f_in = gzip.open('filename2.pickle.gz', 'rb') f_out = gzip.open('filename2.pickle', 'wb') f_out.writelines(f_in) f_in.close() f_out.close() df2 = pandas.load('filename2.pickle') ``` Can this be done without creating `filename2.pickle` first?