resize a 2D numpy array excluding NaN

arrays, image, numpy, python

Solution

You are operating on small windows of the array. Instead of looping through the array to make the windows, the array can be efficiently restructured by manipulating its strides. The numpy library provides the `as_strided()` function to help with that. An example is provided in the SciPy CookBook Stride tricks for the Game of Life.

The following will use a generalized sliding window function which I will include it at the end.

Determine the shape of the new array:

rows, cols = a.shape
new_shape = rows / 2, cols / 2

Restructure the array into the windows you need, and create an indexing array identifying NaNs:

# 2x2 windows of the original array
windows = sliding_window(a, (2,2))
# make a windowed boolean array for indexing
notNan = sliding_window(np.logical_not(np.isnan(a)), (2,2))

The new array can be made using a list comprehension or a generator expression.

# using a list comprehension
# make a list of the means of the windows, disregarding the Nan's
means = [window[index].mean() for window, index in zip(windows, notNan)]
new_array = np.array(means).reshape(new_shape)

# generator expression
# produces the means of the windows, disregarding the Nan's
means = (window[index].mean() for window, index in zip(windows, notNan))
new_array = np.fromiter(means, dtype = np.float32).reshape(new_shape)

The generator expression should conserve memory. Using `itertools.izip()` instead of

Your function:

def resize_2d_nonan(array,factor): """ Resize a 2D array by different factor on two axis skipping NaN values. If a new pixel contains only NaN, it will be set to NaN

Parameters ---------- array : 2D np array

factor : int or tuple. If int x and y factor wil be the same

Returns ------- array : 2D np array scaled by factor

Created on Mon Jan 27 15:21:25 2014

@author: damo_ma """ xsize, ysize = array.shape

if isinstance(factor,int): factor_x = factor factor_y = factor window_size = factor, factor elif isinstance(factor,tuple): factor_x , factor_y = factor window_size = factor else: raise NameError('Factor must be a tuple (x,y) or an integer')

if (xsize % factor_x or ysize % factor_y) : raise NameError('Factors must be integer multiple of array shape')

new_shape = xsize / factor_x, ysize / factor_y

# non-overlapping windows of the original array windows = sliding_window(a, window_size) # windowed boolean array for indexing notNan = sliding_window(np.logical_not(np.isnan(a)), window_size)

#list of the means of the windows, disregarding the Nan's means = [window[index].mean() for window, index in zip(windows, notNan)] # new array new_array = np.array(means).reshape(new_shape)

return new_array

I haven't done any time comparisons with your original function, but it should be faster.

Many solutions I've seen here on SO vectorize the operations to increase speed/efficiency - I don't quite have a handle on that and don't know if it can be applied to your problem. Searching SO for window, array, moving average, vectorize, and numpy should produce similar questions and answers for reference.

`sliding_window()` see attribution below:

import numpy as np from numpy.lib.stride_tricks import as_strided as ast from itertools import product

def norm_shape(shape): ''' Normalize numpy array shapes so they're always expressed as a tuple, even for one-dimensional shapes. Parameters shape - an int, or a tuple of ints Returns a shape tuple ''' try: i = int(shape) return (i,) except TypeError: # shape was not a number pass try: t = tuple(shape) return t except TypeError: # shape was not iterable pass raise TypeError('shape must be an int, or a tuple of ints')

def sliding_window(a,ws,ss = None,flatten = True): ''' Return a sliding window over a in any number of dimensions Parameters: a - an n-dimensional numpy array ws - an int (a is 1D) or tuple (a is 2D or greater) representing the size of each dimension of the window ss - an int (a is 1D) or tuple (a is 2D or greater) representing the amount to slide the window in each dimension. If not specified, it defaults to ws. flatten - if True, all slices are flattened, otherwise, there is an extra dimension for each dimension of the input. Returns an array containing each n-dimensional window from a ''' if None is ss: # ss was not provided. the windows will not overlap in any direction. ss = ws ws = norm_shape(ws) ss = norm_shape(ss) # convert ws, ss, and a.shape to numpy arrays so that we can do math in every # dimension at once. ws = np.array(ws) ss = np.array(ss) shape = np.array(a.shape) # ensure that ws, ss, and a.shape all have the same number of dimensions ls = [len(shape),len(ws),len(ss)] if 1 != len(set(ls)): raise ValueError(\ 'a.shape, ws and ss must all have the same length. They were %s' % str(ls)) # ensure that ws is smaller than a in every dimension if np.any(ws > shape): raise ValueError(\ 'ws cannot be larger than a in any dimension.\ a.shape was %s and ws was %s' % (str(a.shape),str(ws))) # how many slices will there be in each dimension? newshape = norm_shape(((shape - ws) // ss) + 1) # the shape of the strided array will be the number of slices in each dimension # plus the shape of the window (tuple addition) newshape += norm_shape(ws) # the strides tuple will be the array's strides multiplied by step size, plus # the array's strides (tuple addition) newstrides = norm_shape(np.array(a.strides) * ss) + a.strides strided = ast(a,shape = newshape,strides = newstrides) if not flatten: return strided # Collapse strided so that it has one more dimension than the window. I.e., # the new array is a flat list of slices. meat = len(ws) if ws.shape else 0 firstdim = (np.product(newshape[:-meat]),) if ws.shape else () dim = firstdim + (newshape[-meat:]) # remove any dimensions with size 1 dim = filter(lambda i : i != 1,dim) return strided.reshape(dim)

sliding_window() attribution I originally found this on a blog page that is now a broken link:

Efficient Overlapping Windows with Numpy - http://www.johnvinyard.com/blog/?p=268

With a little searching it looks like it now resides in the Zounds github repository. Thanks John Vinyard.

Note this post is pretty old and there are a lot of SO Q&A's regarding sliding windows, rolling windows, and for images- patch extraction. There are a lot of one-offs using numpy's as_strided but this function still seems the only one to handle n-d windowing. scikits sklearn.feature_extraction.image library seems to be often cited for extracting or viewing image patches.

Problem

Original source

Related problems

• ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all()