Numpy __getitem__ delayed evaluation and a[-1:] not the same as a[slice(-1, None, none)]
magic-methods, numpy, python
Solution
You've been bit by a nasty bug indeed. It's kind of a relief to know I am not the only one! Fortunately it is easy to solve. Just add something like the following to your class. This is actually a copy-paste from some code I wrote a few months back, the docstring sort of tells what is going on, but you may want to read the python docs as well.
def __getslice__(self, start, stop) :
"""This solves a subtle bug, where __getitem__ is not called, and all
the dimensional checking not done, when a slice of only the first
dimension is taken, e.g. a[1:3]. From the Python docs:
Deprecated since version 2.0: Support slice objects as parameters
to the __getitem__() method. (However, built-in types in CPython
currently still implement __getslice__(). Therefore, you have to
override it in derived classes when implementing slicing.)
"""
return self.__getitem__(slice(start, stop))
Problem
So this is two questions about what I'm assuming is the same basic underlying confusion on my part. I hope that's ok. Here some code: ``` import numpy as np class new_array(np.ndarray): def __new__(cls, array, foo): obj = array.view(cls) obj.foo = foo return obj def __array_finalize__(self, obj): print "__array_finalize" if obj is None: return self.foo = getattr(obj, 'foo', None) def __getitem__(self, key): print "__getitem__" print "key is %s"%repr(key) print "self.foo is %d, self.view(np.ndarray) is %s"%( self.foo, repr(self.view(np.ndarray)) ) self.foo += 1 return super(new_array, self).__getitem__(key) print "Block 1" print "Object construction calls" base_array = np.arange(20).reshape(4,5) print "base_array is %s"%repr(base_array) p = new_array(base_array, 0) print "\n\n" print "Block 2" print "Call sequence for p[-1:] is:" p[-1:] print "p[-1].foo is %d\n\n"%p.foo print "Block 3" print "Call sequence for s = p[-1:] is:" s = p[-1:] print "p[-1].foo is now %d"%p.foo print "s.foo is now %d"%s.foo print "s.foo + p.foo = %d\n\n"%(s.foo + p.foo) print "Block 4" print "Doing q = s + s" q = s + s print "q.foo = %d\n\n"%q.foo print "Block 5" print "Printing s" print repr(s) print "p.foo is now %d"%p.foo print "s.foo is now %d\n\n"%s.foo print "Block 6" print "Printing q" print repr(q) print "p.foo is now %d"%p.foo print "s.foo is now %d"%s.foo print "q.foo is now %d\n\n"%q.foo print "Block 7" print "Call sequence for p[-1]" a = p[-1] print "p[-1].foo is %d\n\n"%a.foo print "Block 8" print "Call sequence for p[slice(-1, None, None)] is:" a = p[slice(-1, None, None)] print "p[slice(None, -1, None)].foo is %d"%a.foo print "p.foo is %d"%p.foo print "s.foo + p.foo = %d\n\n"%(s.foo + p.foo) ``` The output of this code is ``` Block 1 Object construction calls base_array is array([[ 0, 1, 2, 3, 4], [ 5, 6, 7, 8, 9], [10, 11, 12, 13, 14], [15, 16, 17, 18, 19]]) __array_finalize Block 2 Call sequence for p[-1:] is: __array_finalize p[-1].foo is 0 Block 3 Call sequence for s = p[-1:] is: __array_finalize p[-1].foo is now 0 s.foo is now 0 s.foo + p.foo = 0 Block 4 Doing q = s + s __array_finalize q.foo = 0 Block 5 Printing s __getitem__ key is -1 self.foo is 0, self.view(np.ndarray) is array([[15, 16, 17, 18, 19]]) __array_finalize __getitem__ key is -5 self.foo is 1, self.view(np.ndarray) is array([15, 16, 17, 18, 19]) __getitem__ key is -4 self.foo is 2, self.view(np.ndarray) is array([15, 16, 17, 18, 19]) __getitem__ key is -3 self.foo is 3, self.view(np.ndarray) is array([15, 16, 17, 18, 19]) __getitem__ key is -2 self.foo is 4, self.view(np.ndarray) is array([15, 16, 17, 18, 19]) __getitem__ key is -1 self.foo is 5, self.view(np.ndarray) is array([15, 16, 17, 18, 19]) new_array([[15, 16, 17, 18, 19]]) p.foo is now 0 s.foo is now 1 Block 6 Printing q __getitem__ key is -1 self.foo is 0, self.view(np.ndarray) is array([[30, 32, 34, 36, 38]]) __array_finalize __getitem__ key is -5 self.foo is 1, self.view(np.ndarray) is array([30, 32, 34, 36, 38]) __getitem__ key is -4 self.foo is 2, self.view(np.ndarray) is array([30, 32, 34, 36, 38]) __getitem__ key is -3 self.foo is 3, self.view(np.ndarray) is array([30, 32, 34, 36, 38]) __getitem__ key is -2 self.foo is 4, self.view(np.ndarray) is array([30, 32, 34, 36, 38]) __getitem__ key is -1 self.foo is 5, self.view(np.ndarray) is array([30, 32, 34, 36, 38]) new_array([[30, 32, 34, 36, 38]]) p.foo is now 0 s.foo is now 1 q.foo is now 1 Block 7 Call sequence for p[-1] __getitem__ key is -1 self.foo is 0, self.view(np.ndarray) is array([[ 0, 1, 2, 3, 4], [ 5, 6, 7, 8, 9], [10, 11, 12, 13, 14], [15, 16, 17, 18, 19]]) __array_finalize p[-1].foo is 1 Block 8 Call sequence for p[slice(-1, None, None)] is: __getitem__ key is slice(-1, None, None) self.foo is 1, self.view(np.ndarray) is array([[ 0, 1, 2, 3, 4], [ 5, 6, 7, 8, 9], [10, 11, 12, 13, 14], [15, 16, 17, 18, 19]]) __array_finalize p[slice(None, -1, None)].foo is 2 p.foo is 2 s.foo + p.foo = 3 ``` Please note two things: The call to `p[-1:]` does not result in a call to `new_array.__getitem__`. This is true if `p[-1:]` is replaced by things like `p[0:]`, `p[0:-1]`, etc... but statements like `p[-1]` and `p[slice(-1, None, None)]` do result in a call to `new_array.__getitem__`. It's also true for statements like `p[-1:] + p[-1:]` or `s = p[-1]` but isn't true for statements like `print s`. You can see this by looking at "blocks" given above. The variable `foo` is correctly updated during calls to `new_array.__getitem__` (see blocks 5 and 6) but is not correct once evaluation of `new_array.__getitem__` is complete (see, again, blocks 5 and 6). I should also add that replacing the line `return super(new_array, self).__getitem__(key)` with `return new_array(np.array(self.view(np.ndarray)[key]), self.foo)` does not work either. The following blocks are the only differences in output. ``` Block 5 Printing s __getitem__ key is -1 self.foo is 0, self.view(np.ndarray) is array([[15, 16, 17, 18, 19]]) __array_finalize__ __getitem__ key is -5 self.foo is 1, self.view(np.ndarray) is array([15, 16, 17, 18, 19]) __array_finalize__ __array_finalize__ __array_finalize__ __getitem__ key is -4 self.foo is 2, self.view(np.ndarray) is array([15, 16, 17, 18, 19]) __array_finalize__ __array_finalize__ __array_finalize__ __getitem__ key is -3 self.foo is 3, self.view(np.ndarray) is array([15, 16, 17, 18, 19]) __array_finalize__ __array_finalize__ __array_finalize__ __getitem__ key is -2 self.foo is 4, self.view(np.ndarray) is array([15, 16, 17, 18, 19]) __array_finalize__ __array_finalize__ __array_finalize__ __getitem__ key is -1 self.foo is 5, self.view(np.ndarray) is array([15, 16, 17, 18, 19]) __array_finalize__ __array_finalize__ __array_finalize__ new_array([[15, 16, 17, 18, 19]]) p.foo is now 0 s.foo is now 1 Block 6 Printing q __getitem__ key is -1 self.foo is 0, self.view(np.ndarray) is array([[30, 32, 34, 36, 38]]) __array_finalize__ __getitem__ key is -5 self.foo is 1, self.view(np.ndarray) is array([30, 32, 34, 36, 38]) __array_finalize__ __array_finalize__ __array_finalize__ __getitem__ key is -4 self.foo is 2, self.view(np.ndarray) is array([30, 32, 34, 36, 38]) __array_finalize__ __array_finalize__ __array_finalize__ __getitem__ key is -3 self.foo is 3, self.view(np.ndarray) is array([30, 32, 34, 36, 38]) __array_finalize__ __array_finalize__ __array_finalize__ __getitem__ key is -2 self.foo is 4, self.view(np.ndarray) is array([30, 32, 34, 36, 38]) __array_finalize__ __array_finalize__ __array_finalize__ __getitem__ key is -1 self.foo is 5, self.view(np.ndarray) is array([30, 32, 34, 36, 38]) __array_finalize__ __array_finalize__ __array_finalize__ new_array([[30, 32, 34, 36, 38]]) p.foo is now 0 s.foo is now 1 q.foo is now 1 ``` Which now contains excessive calls to `new_array.__array_finalize__`, but has no change in the "problem" with the variable `foo`. It was my expectation that a call like `p[-1:]` to a `new_array` object with `p.foo = 0` would result in this statement `p.foo == 1` returning `True`. Clearly that isn't the case, even if `foo` was being correctly updated during calls to `__getitem__`, since a statement like `p[-1:]` results in a large number of calls to `__getitem__` (once the delayed evaluation is taken into account). Moreover the calls `p[-1:]` and `p[slice(-1, None, None)]` would result in different values of `foo` (if the counting thing was working correctly). In the former case `foo` would have had `5` added to it, while in the later case `foo` would have had `1` added to it. The Question While the delayed evaluation of slices of numpy arrays isn't going to cause problems during evaluation of my code, it has been a huge pain for debugging some code of mine using pdb. Basically statements appear to evaluate differently at run time and in the pdb. I figure that this isn't good. That is how I stumbled across this behaviour. My code uses the input to `__getitem__` to evaluate what type of object should be returned. In some cases it returns a new instance of the same type, in other cases it returns a new instance of some other type and in still other cases it returns a numpy array, scalar or float (depending on whatever the underlying numpy array thinks is right). I use the key passed to `__getitem__` to determine what the correct object to return is. But I can't do this if the user has passed a slice, e.g. something like `p[-1:]`, since the method just gets individual indices, e.g. as if the user wrote `p[4]`. So how do I do this if the `key` in `__getitem__` of my numpy subclass doesn't reflect if the user is requesting a slice, given by `p[-1:]`, or just an entry, given by `p[4]`? As a side point the numpy indexing documentation implies that slice objects, e.g. `slice(start, stop, step)` will be treated the same as statements like, `start:stop:step`. This makes me think that I'm missing something very basic. The sentence that implies this occurs very early: Basic slicing occurs when obj is a slice object (constructed by start:stop:step notation inside of brackets), an integer, or a tuple of slice objects and integers. I can't help but feel that this same basic mistake is also the reason why I think the `self.foo += 1` line should be counting the number of times a user requests a slice, or an element of an instance of `new_array` (rather than the number of elements "in" a slice). Are these two issues actually related and if so how?