Easiest way to create a NumPy record array from a list of dictionaries?

numpy, python

Solution

You could make an empty structured array of the right size and dtype, and then fill it from the list.

http://docs.scipy.org/doc/numpy/user/basics.rec.html

Structured arrays can be filled by field or row by row. ... If you fill it in row by row, it takes a take a tuple (but not a list or array!):

In [72]: dt=dtype([('weight',int),('animal','S10')])

In [73]: values = [tuple(each.values()) for each in d]

In [74]: values
Out[74]: [(5, 'cat'), (20, 'dog')]

fields in the `dt` occur in the same order as in `values`.

In [75]: a=np.zeros((2,),dtype=dt)

In [76]: a[:]=[tuple(each.values()) for each in d]

In [77]: a
Out[77]: 
array([(5, 'cat'), (20, 'dog')], 
      dtype=[('weight', '<i4'), ('animal', 'S10')])

With a bit more testing I found I can create the array directly from `values`.

In [83]: a = np.array(values, dtype=dt)

In [84]: a
Out[84]: 
array([(5, 'cat'), (20, 'dog')], 
      dtype=[('weight', '<i4'), ('animal', 'S10')])

The `dtype` could be deduced from one (or more) of the dictionary items:

def gettype(v):
    if isinstance(v,int): return 'int'
    elif isinstance(v,float): return 'float'
    else:
        assert isinstance(v,str)
        return '|S%s'%(len(v)+10)
d0 = d[0]
names = d0.keys()
formats = [gettype(v) for v in d0.values()]
dt = np.dtype({'names':names, 'formats':formats})

producing:

dtype=[('weight', '<i4'), ('animal', 'S13')]

Problem

Say I have data like `d = [dict(animal='cat', weight=5), dict(animal='dog', weight=20)]` (basically JSON, where all entries have consistent data types). In Pandas you can make this a table with `df = pandas.DataFrame(d)` -- is there something comparable for plain NumPy record arrays? `np.rec.fromrecords(d)` doesn't seem to given me what I want.

Original source