Understanding non-homogeneous numpy arrays

Question

I have recently started numpy and noticed a peculiar thing. ``` import numpy as np a = np.array([[1,2,3], [4,5,9, 8]]) print a.shape, "shape" print a[1, 0] ``` The shape, in this case, comes out to be `2L`. However if I make a homogenous numpy array as ` a = np.array([[1,2,3], [4,5,6]]`, then `a.shape` gives `(2L, 3L)`. I understand that the shape of a non-homogenous array is difficult to represent as a tuple. Additionally, `print a[1,0]` for non-homogenous array that I created earlier gives a traceback `IndexError: too many indices for array`. Doing the same on the homogenous array gives back the correct element `4`. Noticing these two peculiarities, I am curious to know how python looks at non-homogenous numpy arrays at a low level. Thank You in advance

Accepted Answer

When the sublists differ in length, `np.array` falls back to creating an `object dtype` array:

```
In [272]: a = np.array([[1,2,3], [4,5,9, 8]])
In [273]: a
Out[273]: array([[1, 2, 3], [4, 5, 9, 8]], dtype=object)
```

This array is similar to the list we started with. Both store the sublists as pointers. The sublists exist else where in memory.

With equal length sublsts, it can create a 2d array, with integer elements:

```
In [274]: a2 = np.array([[1,2,3], [4,5,9]])
In [275]: a2
Out[275]: 
array([[1, 2, 3],
       [4, 5, 9]])
```

In fact to confirm my claim that the sublists are stored elsewhere in memory, let's try to change one:

```
In [276]: alist = [[1,2,3], [4,5,9, 8]]
In [277]: a = np.array(alist)
In [278]: a
Out[278]: array([[1, 2, 3], [4, 5, 9, 8]], dtype=object)
In [279]: a[0].append(4)
In [280]: a
Out[280]: array([[1, 2, 3, 4], [4, 5, 9, 8]], dtype=object)
In [281]: alist
Out[281]: [[1, 2, 3, 4], [4, 5, 9, 8]]
```

That would not work in the case of `a2`. `a2` has its own data storage, independent of the source list.

The basic point is that `np.array` tries to create an n-d array where possible. If it can't it falls back on to creating an object dtype array. And, as has been discussed in other questions, it sometimes raises an error. It is also tricky to intentionally create an object array.

The shape of `a` is easy, (2,). A single element tuple. `a` is a 1d array. But that shape does not convey information about the elements of `a`. And the same goes for the elements of `alist`. `len(alist)` is 2. An object array can have a more complex shape, e.g. `a.reshape(1,2,1)`, but it is still just contains pointers

`a` contains 2 4byte pointers; `a2` contains 6 4byte integers.

```
n [282]: a.itemsize
Out[282]: 4
In [283]: a.nbytes
Out[283]: 8
In [284]: a2.nbytes
Out[284]: 24
In [285]: a2.itemsize
Out[285]: 4
```

Understanding non-homogeneous numpy arrays

Solution

Problem