matplotlib major display issue with dense data sets

Question

I've run into a fairly serious issue with matplotlib and Python. I have a dense periodogram data set and want to plot it. The issue is that when there are more data points than can be plotted on a pixel, the package does not pick the min and max to display. This means a casual look at the plot can lead you to incorrect conclusions. Here's an example of such a problem: The dataset was plotted with `plot()` and `scatter()` overlayed. You can see that in the dense data fields, the blue line that connects the data does not reach the actual peaks, leading a human viewer to conclude the peak at ~2.4 is the maximum, when it's really not. If you zoom-in or force a wide viewing window, it is displayed correctly. `rasterize` and `aa` keywords have no effect on the issue. Is there a way to ensure that the min/max points of a `plot()` call are always rendered? Otherwise, this needs to be addressed in an update to matplotlib. I've never had a plotting package behave like this, and this is a pretty…

Accepted Answer

This is due to the path-simplification algorithm in matplotlib. While it's certainly not desirable in some cases, it's deliberate behavior to speed up rendering.

The simplification algorithm was changed at some point to avoid skipping "outlier" points, so newer versions of mpl don't exhibit this exact behavior (the path is still simplified, though).

If you don't want to simplify paths, then you can disable it in the rc parameters (either in your `.matplotlibrc` file or at runtime).

E.g.

```
import matplotlib as mpl
mpl.rcParams['path.simplify'] = False
import matplotlib.pyplot as plt
```

However, it may make more sense to use an "envelope" style plot. As a quick example:

```
import matplotlib.pyplot as plt
import numpy as np

def main():
    num = 10000
    x = np.linspace(0, 10, num)
    y = np.cos(x) + 5 * np.random.random(num)

fig, (ax1, ax2) = plt.subplots(nrows=2)
    ax1.plot(x, y)
    envelope_plot(x, y, winsize=40, ax=ax2)
    plt.show()

def envelope_plot(x, y, winsize, ax=None, fill='gray', color='blue'):
    if ax is None:
        ax = plt.gca()
    # Coarsely chunk the data, discarding the last window if it's not evenly
    # divisible. (Fast and memory-efficient)
    numwin = x.size // winsize
    ywin = y[:winsize * numwin].reshape(-1, winsize)
    xwin = x[:winsize * numwin].reshape(-1, winsize)
    # Find the min, max, and mean within each window 
    ymin = ywin.min(axis=1)
    ymax = ywin.max(axis=1)
    ymean = ywin.mean(axis=1)
    xmean = xwin.mean(axis=1)

fill_artist = ax.fill_between(xmean, ymin, ymax, color=fill, 
                                  edgecolor='none', alpha=0.5)
    line, = ax.plot(xmean, ymean, color=color, linestyle='-')
    return fill_artist, line

if __name__ == '__main__':
    main()
```

matplotlib major display issue with dense data sets

Solution

Problem