How to estimate how much memory a Pandas' DataFrame will need?
pandas, python
Solution
`df.memory_usage()` will return how many bytes each column occupies:
>>> df.memory_usage()
Row_ID 20906600
Household_ID 20906600
Vehicle 20906600
Calendar_Year 20906600
Model_Year 20906600
...
To include indexes, pass `index=True`.
So to get overall memory consumption:
>>> df.memory_usage(index=True).sum()
731731000
Also, passing `deep=True` will enable a more accurate memory usage report, that accounts for the full usage of the contained objects.
This is because memory usage does not include memory consumed by elements that are not components of the array if `deep=False` (default case).
Problem
I have been wondering... If I am reading, say, a 400MB csv file into a pandas dataframe (using read_csv or read_table), is there any way to guesstimate how much memory this will need? Just trying to get a better feel of data frames and memory...