Handling HUGE numbers in numpy or pandas

numpy, pandas, python

Solution

You can use Pandas converters to call `int` or some other custom converter function on the string as they are being imported:

import pandas as pd 
from StringIO import StringIO

txt='''\
line,Big_Num,text
1,1234567890123456789012345678901234567890,"That sure is a big number"
2,9999999999999999999999999999999999999999,"That is an even BIGGER number"
3,1,"Tiny"
4,-9999999999999999999999999999999999999999,"Really negative"
'''

df=pd.read_csv(StringIO(txt), converters={'Big_Num':int})

print df

Prints:

   line                                    Big_Num                           text
0     1   1234567890123456789012345678901234567890      That sure is a big number
1     2   9999999999999999999999999999999999999999  That is an even BIGGER number
2     3                                          1                           Tiny
3     4  -9999999999999999999999999999999999999999                Really negative

Now test arithmetic:

n=df["Big_Num"][1]
print n,n+1

Prints:

9999999999999999999999999999999999999999 10000000000000000000000000000000000000000

If you have any values in the column that might cause `int` to croak, you can do this:

txt='''\
line,Big_Num,text
1,1234567890123456789012345678901234567890,"That sure is a big number"
2,9999999999999999999999999999999999999999,"That is an even BIGGER number"
3,0.000000000000000001,"Tiny"
4,"a string","Use 0 for strings"
'''

def conv(s):
    try:
        return int(s)
    except ValueError:
        try:
            return float(s)
        except ValueError:
            return 0        

df=pd.read_csv(StringIO(txt), converters={'Big_Num':conv})
print df

Prints:

   line                                   Big_Num                           text
0     1  1234567890123456789012345678901234567890      That sure is a big number
1     2  9999999999999999999999999999999999999999  That is an even BIGGER number
2     3                                     1e-18                           Tiny
3     4                                         0              Use 0 for strings

Then every value in the column will be either a Python int or a float and will support arithmetic.

Problem

I am doing a competition where I am provided data that is anonymized. Quite a few of the columns have HUGE values. The largest was 40 digits long! I used `pd.read_csv` but those columns have been converted to objects as a result. My original plan was to scale the data down but since they are seen as objects I can't do arithmetic on these. Does anyone have a suggestion on how to handle huge numbers in Pandas or Numpy? Note that I've tried converting the value to a `uint64` with no luck. I get the error "long too big to convert"

Original source