Different result for std between pandas and numpy
mean, numpy, pandas, python, python-3.x
Solution
When calculating the standard deviation it matters whether you are estimating the standard deviation of an entire population with a smaller sample of that population or are you calculating the standard deviation of the entire population.
If it is a smaller sample of a larger population, you need what is called the sample standard deviation. As it turns out, when you divide the sum of squared differences from the mean by the number of observations, you end up with a biased estimator. We correct for that by dividing by one less than the number of observations. We control for this with the argument `ddof=1` for sample standard deviation or `ddof=0` for population standard deviation.
Truth is, it doesn't matter much if your sample size is large. But you will see small differences.
Use the degrees of freedom argument in your `pandas.DataFrame.std` call:
import pandas as pd
data = pd.read_csv("https://s3.amazonaws.com/demo-datasets/wine.csv")
numeric_data = data.drop("color", 1)
numeric_data1 = ((numeric_data - numeric_data.mean()) /
numeric_data.std(ddof=0)) # <<<
numeric_data2 = ((numeric_data - np.mean(numeric_data, axis=0)) /
np.std(numeric_data, axis=0))
np.isclose(numeric_data1, numeric_data2).all() # -> True
Or in the `np.std` call:
import pandas as pd
data = pd.read_csv("https://s3.amazonaws.com/demo-datasets/wine.csv")
numeric_data = data.drop("color", 1)
numeric_data1 = ((numeric_data - numeric_data.mean()) /
numeric_data.std())
numeric_data2 = ((numeric_data - np.mean(numeric_data, axis=0)) /
np.std(numeric_data, axis=0, ddof=1)) # <<<
np.isclose(numeric_data1, numeric_data2).all() # -> True
Problem
I am trying to subtract every element in the column from its mean and divide by the standard deviation. I did it in two different ways (`numeric_data1` and `numeric_data2`): ``` import pandas as pd data = pd.read_csv("https://s3.amazonaws.com/demo-datasets/wine.csv") numeric_data = data.drop("color", 1) numeric_data1 = ((numeric_data - numeric_data.mean()) / numeric_data.std()) numeric_data2 = ((numeric_data - np.mean(numeric_data, axis=0)) / np.std(numeric_data, axis=0)) type(numeric_data1) # -> pandas.core.frame.DataFrame type(numeric_data2) # -> pandas.core.frame.DataFrame ``` Both are pandas dataframes and they should have the same result. However, I get different results: ``` numeric_data2 == numeric_data1 # -> False ``` I think the problem stems from how numpy and pandas handle numeric precision: ``` numeric_data.mean() == np.mean(numeric_data, axis=0) # -> True numeric_data.std(axis=0) == np.std(numeric_data, axis=0) # -> False ``` For mean numpy and pandas gave me the same thing, but for standard deviation, I got little different results. Is my assessment correct or am I making some blunder?