numpy.concatenate on record arrays fails when array has different length strings
numpy, python
Solution
To post a complete answer. As Pierre GM suggested the module:
import numpy.lib.recfunctions
gives a solution. The function that does what you want however is:
numpy.lib.recfunctions.stack_arrays((a,b), autoconvert=True, usemask=False)
(`usemask=False` is just to avoid creation of a masked array, which you are probably not using. The important thing is `autoconvert=True` to force the conversion from `a`'s `dtype` `"|S3"` to `"|S5"`).
Problem
When trying to concatenate record arrays which has a field of dtype string but has different length, concatenation fails. As you can see in the following example, concatenate works if 'f1' is of same length but fails, if not. ``` In [1]: import numpy as np In [2]: a = np.core.records.fromarrays( ([1,2], ["one","two"]) ) In [3]: b = np.core.records.fromarrays( ([3,4,5], ["three","four","three"]) ) In [4]: c = np.core.records.fromarrays( ([6], ["six"]) ) In [5]: np.concatenate( (a,c) ) Out[5]: array([(1, 'one'), (2, 'two'), (6, 'six')], dtype=[('f0', '<i8'), ('f1', '|S3')]) In [6]: np.concatenate( (a,b) ) --------------------------------------------------------------------------- TypeError Traceback (most recent call last) /u/jegannas/<ipython console> in <module>() TypeError: expected a readable buffer object ``` But, again if we just concatenate the arrays (not the records), it succeeds, though strings are of different size. ``` In [8]: np.concatenate( (a['f1'], b['f1']) ) Out[8]: array(['one', 'two', 'three', 'four', 'three'], dtype='|S5') ``` Is this a bug in concatenate when concatenating records or is this the expected behavior. I have figured only the following way to overcome this. ``` In [10]: np.concatenate( (a.astype(b.dtype), b) ) Out[10]: array([(1, 'one'), (2, 'two'), (3, 'three'), (4, 'four'), (5, 'three')], dtype=[('f0', '<i8'), ('f1', '|S5')] ``` But the trouble here is that I have to go through all the recarrays, I am concatenating and find the largest string length and I have to use that. If I have more than one string columns in the record array, I need to keep track of a few other things too. What do you think is the best way to overcome this, at least for now?