Density estimate with large array

Question

I'm using the `gaussian_kde` function in SciPy to generate a kernel density estimate: ``` from scipy.stats.kde import gaussian_kde from scipy.stats import norm from numpy import linspace,hstack from pylab import plot,show,hist # creating data with two peaks sampD1 = norm.rvs(loc=-1.0,scale=1,size=300) sampD2 = norm.rvs(loc=2.0,scale=0.5,size=300) samp = hstack([sampD1,sampD2]) # obtaining the pdf (my_pdf is a function!) my_pdf = gaussian_kde(samp) # plotting the result x = linspace(-5,5,100) plot(x,my_pdf(x),'r') # distribution function hist(samp,normed=1,alpha=.3) # histogram show() ``` The above code works, but can be prohibitively slow with a very large number of samples. Instead of storing my samples in arrays, I have a dictionary with key/value pairs of `value: counts`. For example the array `[1, 1, 1, 2, 2, 3]` would be encoded in this histogram dictionary as: `{1:3, 2:2, 3:1}`. My questions is, how can I generate a kernel density estimate using a dictionary data structure? As a…

Accepted Answer

You can do Gaussian KDE yourself: you just first need to create the simple histogram with small enough step size. Then convolve the result with the Gaussian using fftconvolve (scipy.signal.fftconvolve)

```
import numpy as np, numpy.random,scipy,scipy.stats,scipy.signal,matplotlib.pyplot as plt
N = 1e5
minx = -10
maxx = 10
bins = 10000
w = 0.1 # kernel sigma

xs1 = np.random.normal(0, 1, size=N)
xs2 = np.random.normal(1.9, 0.01, size=N)
xs = np.r_[xs1, xs2]
hh,loc = scipy.histogram(xs, range=(minx, maxx), bins=bins)
kernel = scipy.stats.norm.pdf((loc[1:]+loc[:-1]) * .5, 0, w)
kde = scipy.signal.fftconvolve(hh, kernel, 'same')
plt.plot((loc[1:] + loc[:-1])*.5, kde)
```

Density estimate with large array

Solution

Problem