I understand that Anova is preferred over multiple t-tests, but what exact is multiple t-tests?
Examples involving taking groups from the population (with the same population mean):
- We have groups taken from a population, then tested with different treatment per group, then comparing the effect of the treatment between each pair of groups. E.g. students from a school are randomly split into 3 groups, each takes a different vitamin pill and we compare the fitness level between group 1 & 2, 2 & 3, 3 & 1.
- You have two or more groups, and you are comparing multiple parameters. E.g. the you have two groups of students, one taken a vitamin pill and the other a placebo but we are comparing the fitness level, height, weight, fat content etc between the groups. This also require multiple t-tests and would be best to use Anova with some post-test correct method.
A different example involving comparing existing groups:
- We take 5 different species of animals, subject them with the same treatment, and compare their response with each other. E.g. Sheep, cows, horses, chicken and mice were given a chemical injection and we recorded their change in fitness level (in the same manner).
If I want to compare the result in pairs (e.g. sheep and cows, cows and horses, horse and chicken, chicken and mice, etc) using t-test, is this still "multiple t-tests" (they don't have a common population mean…). Do I still have to use Anova?
Best Answer
The difference between your animal and your vitamin example is that one involved random assignment of participants to groups and the other did not. This has implications for causal inference, but it is not especially related to how you might analyse the data.
A standard approach when you have three or more groups and a numeric dependent variable is to first test the null hypothesis that all group means are equal. ANOVA provides a significance test for this. Then, if the ANOVA is significant then this is followed up by some procedure to understand the pattern of group means (e.g., post hoc tests, contrasts and so on).
The rationale for this approach is that the overall ANOVA provides some initial overall check of group mean differences before you commence examination of the group mean differences.
Furthermore, there is a difference between performing all possible pairwise comparisons using a t-test versus using common post-hoc tests (e.g., Tukey's). Post-hoc tests generally have some built-in component to protect against type-1 errors.
That said, you don't have to perform an ANOVA before performing follow-up tests. In fact, if there are particular comparisons that interest you, then you might want to perform those first, or you might argue that you have no interest in the overall ANOVA test.
The main thing is that you have a rationale for how you are doing your inference.