There are several statistical texts that examine the properties of the sample mean and sample variance for IID normal data. It is well-known that these statistics are unbiased estimators for the true mean and variance parameters. Moreover, for the case of IID normal data, it is well-known that the statistics are independent, and the second-order moments of the statistics are:
$$\mathbb{V}(\bar{X}_n) = \frac{\sigma^2}{n}
\quad \quad \quad
\mathbb{C}(\bar{X}_n, S_n^2) = 0
\quad \quad \quad
\mathbb{V}(S_n^2) = \frac{2 \sigma^4}{n-1}.$$
Question: What is the corresponding generalised result for the second-order moments when the data is IID but may come from another distribution?
.svg){kind=link}
Best Answer
Detailed analysis of the moments of the sample moments can be found in O'Neill (2014). This includes all the first and second-order moments of the sample mean and sample variance, their unsampled counterparts, and various components related to these statistics. The general form of the second-order moments depends on the first four moments of the distribution. To facilitate our analysis, we suppose that $X_1,...,X_n$ are IID with mean $\mu$, variance $\sigma^2$, skewness $\gamma$ and kurtosis $\kappa$. The variance of the sample variance exists if $\kappa < \infty$ and does not exist otherwise.
Under these preliminary assumptions, the relevant second-order moments are:
$$\mathbb{V}(\bar{X}_n) = \frac{\sigma^2}{n} \quad \quad \quad \mathbb{C}(\bar{X}_n, S_n^2) = \frac{\gamma \sigma^3}{n} \quad \quad \quad \mathbb{V}(S_n^2) = \bigg( \kappa - \frac{n-3}{n-1} \bigg) \frac{\sigma^4}{n}.$$
It is also worth noting that the correlation between the sample mean and sample variance is:
$$\mathbb{Corr}(\bar{X}_n, S_n^2) = \frac{\gamma}{\sqrt{\kappa - (n-3)/(n-1)}}.$$
The normal distribution is an unskewed mesokurtic distribution, so we have $\gamma = 0$ and $\kappa = 3$, which gives the more specific results in the question. As we can see, in the more general case the sample mean and sample variance will be correlated if the underlying distribution is skewed. If the underlying distribution is positively skewed then they are positively correlated, and if the underlying distribution is negatively skewed then they are negatively correlated.