I am trying to understand ANOVA. In one-way ANOVA to test equality of mean of several population groups, each group has a normal distribution with the same variance. I wonder if the common variance of each group is known and used somewhere or unknown and estimated from samples in ANOVA? Thanks and regards!
Solved – Is the variance of each group unknown in one-way ANOVA
anova
Related Solutions
The difference between your animal and your vitamin example is that one involved random assignment of participants to groups and the other did not. This has implications for causal inference, but it is not especially related to how you might analyse the data.
A standard approach when you have three or more groups and a numeric dependent variable is to first test the null hypothesis that all group means are equal. ANOVA provides a significance test for this. Then, if the ANOVA is significant then this is followed up by some procedure to understand the pattern of group means (e.g., post hoc tests, contrasts and so on).
The rationale for this approach is that the overall ANOVA provides some initial overall check of group mean differences before you commence examination of the group mean differences.
Furthermore, there is a difference between performing all possible pairwise comparisons using a t-test versus using common post-hoc tests (e.g., Tukey's). Post-hoc tests generally have some built-in component to protect against type-1 errors.
That said, you don't have to perform an ANOVA before performing follow-up tests. In fact, if there are particular comparisons that interest you, then you might want to perform those first, or you might argue that you have no interest in the overall ANOVA test.
The main thing is that you have a rationale for how you are doing your inference.
Consider the following setup. We have a $p$-dimensional parameter vector $\theta$ that specifies the model completely and a maximum-likelihood estimator $\hat{\theta}$. The Fisher information in $\theta$ is denoted $I(\theta)$. What is usually referred to as the Wald statistic is
$$(\hat{\theta} - \theta)^T I(\hat{\theta}) (\hat{\theta} - \theta)$$
where $I(\hat{\theta})$ is the Fisher information evaluated in the maximum-likelihood estimator. Under regularity conditions the Wald statistic follows asymptotically a $\chi^2$-distribution with $p$-degrees of freedom when $\theta$ is the true parameter. The Wald statistic can be used to test a simple hypothesis $H_0 : \theta = \theta_0$ on the entire parameter vector.
With $\Sigma(\theta) = I(\theta)^{-1}$ the inverse Fisher information the Wald test statistic of the hypothesis $H_0 : \theta_1 = \theta_{0,1}$ is $$\frac{(\hat{\theta}_1 - \theta_{0,1})^2}{\Sigma(\hat{\theta})_{ii}}.$$ Its asymptotic distribution is a $\chi^2$-distribution with 1 degrees of freedom.
For the normal model where $\theta = (\mu, \sigma^2)$ is the vector of the mean and the variance parameters, the Wald test statistic of testing if $\mu = \mu_0$ is $$\frac{n(\hat{\mu} - \mu_0)^2}{\hat{\sigma}^2}$$ with $n$ the sample size. Here $\hat{\sigma}^2$ is the maximum-likelihood estimator of $\sigma^2$ (where you divide by $n$). The $t$-test statistic is $$\frac{\sqrt{n}(\hat{\mu} - \mu_0)}{s}$$ where $s^2$ is the unbiased estimator of the variance (where you divide by the $n-1$). The Wald test statistic is almost but not exactly equal to the square of the $t$-test statistic, but they are asymptotically equivalent when $n \to \infty$. The squared $t$-test statistic has an exact $F(1, n-1)$-distribution, which converges to the $\chi^2$-distribution with 1 degrees of freedom for $n \to \infty$.
The same story holds regarding the $F$-test in one-way ANOVA.
Best Answer
Standard ANOVA estimates the common variance from the data. It uses a "pooled" estimate by taking the differences between each data point and the group mean for that data point, squaring those differences, summing, then dividing by the degrees of freedom (which is the number of data points minus the number of groups, for simple one-way ANOVA).
It would be possible to derive an equivalent of ANOVA where the common variance is known (probably based on the $\chi^2$ distribution ranther than the F), but the liklihood of finding a real world situation where the variance is known but the means are not is low enough that most people don't worry about that situation.
In truth it is probably never true that the populations are exactly normal or that the variances are exactly equal. The Central Limit Theorem covers the normality assumption for normal enough data and big enough sample sizes. The ANOVA tests have been shown to be fairly robust to the assumption that the variances are equal as long as they are at least similar (a common rule of thumb is that ANOVA is ok as long as the ratio of the largest variance and smallest variance is less than 4).