calculating mean and standard deviation of the data which does not fit in memory using python

python, statistics

Solution

There is a simple online algorithm that computes both the mean and the variance by looking at each datapoint once and using `O(1)` memory.

Wikipedia offers the following code:

def online_variance(data):
    n = 0
    mean = 0
    M2 = 0

    for x in data:
        n = n + 1
        delta = x - mean
        mean = mean + delta/n
        M2 = M2 + delta*(x - mean)

    variance = M2/(n - 1)
    return variance

This algorithm is also known as Welford's method. Unlike the method suggested in the other answer, it can be shown to have nice numerical properties.

Take the square root of the variance to get the standard deviation.

Problem

I have a lot of data stored at disk in large arrays. I cant load everything in memory altogether. How one could calculate the mean and the standard deviation?

Original source

Related problems