I have 2 groups of people. I'm working with the data about their age. I know the means, the standard deviations and the number of people. I don't know the data of each person in the groups.
Group 1 :
Mean = 35 years old; SD = 14; n = 137 people
Group 2 :
Mean = 31 years old; SD = 11; n = 112 people
I want to combine those 2 groups to obtain a new mean and SD. It's easy for the mean, but is it possible for the SD? I do not know the distribution of those samples, and I can't assume those are normal distributions. Is there a formula for distributions that aren't necessarily normal?
Best Answer
Continuing on from BruceET's explanation, note that if we are computing the unbiased estimator of the standard deviation of each sample, namely $$s = \sqrt{\frac{1}{n-1} \sum_{i=1}^n (x_i - \bar x)^2},$$ and this is what is provided, then note that for samples $\boldsymbol x = (x_1, \ldots, x_n)$, $\boldsymbol y = (y_1, \ldots, y_m)$, let $\boldsymbol z = (x_1, \ldots, x_n, y_1, \ldots, y_m)$ be the combined sample, hence the combined sample mean is $$\bar z = \frac{1}{n+m} \left( \sum_{i=1}^n x_i + \sum_{j=1}^m y_i \right) = \frac{n \bar x + m \bar y}{n+m}.$$ Consequently, the combined sample variance is $$s_z^2 = \frac{1}{n+m-1} \left( \sum_{i=1}^n (x_i - \bar z)^2 + \sum_{j=1}^m (y_i - \bar z)^2 \right),$$ where it is important to note that the combined mean is used. In order to have any hope of expressing this in terms of $s_x^2$ and $s_y^2$, we clearly need to decompose the sums of squares; for instance, $$(x_i - \bar z)^2 = (x_i - \bar x + \bar x - \bar z)^2 = (x_i - \bar x)^2 + 2(x_i - \bar x)(\bar x - \bar z) + (\bar x - \bar z)^2,$$ thus $$\sum_{i=1}^n (x_i - \bar z)^2 = (n-1)s_x^2 + 2(\bar x - \bar z)\sum_{i=1}^n (x_i - \bar x) + n(\bar x - \bar z)^2.$$ But the middle term vanishes, so this gives $$s_z^2 = \frac{(n-1)s_x^2 + n(\bar x - \bar z)^2 + (m-1)s_y^2 + m(\bar y - \bar z)^2}{n+m-1}.$$ Upon simplification, we find $$n(\bar x - \bar z)^2 + m(\bar y - \bar z)^2 = \frac{mn(\bar x - \bar y)^2}{m + n},$$ so the formula becomes $$s_z^2 = \frac{(n-1) s_x^2 + (m-1) s_y^2}{n+m-1} + \frac{nm(\bar x - \bar y)^2}{(n+m)(n+m-1)}.$$ This second term is the required correction factor.