ANOVA Comparisons – Can All Possible Pairs of Comparison Be Planned Comparisons in ANOVA?

anovacontrastsmultiple-comparisonstype-i-and-ii-errors

Let's say we have three intervention groups (condition A, condition B, and condition C/control) to be analyzed in ANOVA, and we are theoretically interested in the difference between each pair of all the possible comparisons (i.e., A and B, B and C, and A and C). Since the three comparisons are according to the hypotheses, I thought that it would be ok to describe this as planned comparisons. However, the planned comparisons do look like post-hoc tests (that test all possible comparisons for exploratory purposes). Is it still acceptable to consider it as planned comparisons without the necessity to adjust the p values? Or should I regard it as post-hoc tests and adjust the p values for multiple testing?

Summary of the suggestions:

  • I am starting to understand that what I described is not really how planned comparisons should be done. When I run planned contrasts, I may start from comparing intervention groups (A & B) vs. control group (C), followed by the comparison of the two intervention groups (A vs. B). Alternatively, I can focus on a specific pair, although the maximum number of contrasts should be k-1, as BruceET has suggested. (15/Sep/2021)
  • Thank you so much for your brilliant insights, Tanner! It is important to adjust significance level to avoid false findings when running multiple tests. Together with COOLSerdash's critical suggestions, I may not adjust the significance level when I have a specific hypothesis for each comparison (17/Sep/2021).

Best Answer

It's not only possible, it's explicitly recommended (see Ruxton & Beauchamp 2008).

First, if the omnibus-test of homogeneity across all groups is not of interest (e.g. the overall ANOVA $F$-test), consider not paying attention to it at all or not doing it in the first place. Second, if each planned comparison tests a different specific hypothesis, it is actually controversial if a formal control of the experimentwise type 1 error rate (EER) is required or not. Some text do not consider it necessary (Kirk 1995, Quinn & Keough 2002, Rothman 1990, Rubin 2021, Sokal & Rohlf 1995). Ruxton & Beauchamp (2008) recommend not controlling EER if the set of pre-planned contrasts is orthogonal. If all possible pairwise comparisons between groups are planned, the set of contrasts is not orthogonal and so, Ruxton & Beauchamp (2008) recommend controlling EER.

Rubin (2021) on the other hand argues that in the case of individual testing, no alpha adjustment should be done. He defines individual testing as tests, where each individual result must be significant in order to reject each associated individual null hypothesis. This seems to be the case here if you want to test each pairwise group difference individually. Personally, I find his arguments convincing and have subsequently changed my own opinion on the matter.

References

Kirk RE. 1995. Experimental design. Pacific Grove (CA): Brooks/Cole.

Quinn GP, Keough MJ. 2002. Experimental design and data analysis for biologists. Cambridge (UK): Cambridge University Press.

Rothman, K. J. (1990). No adjustments are needed for multiple comparisons. Epidemiology, 43-46. (link)

Rubin, M. (2021). When to adjust alpha during multiple testing: A consideration of disjunction, conjunction, and individual testing. Synthese, 1-32. (link)

Ruxton, G. D., & Beauchamp, G. (2008). Time for some a priori thinking about post hoc testing. Behavioral ecology, 19(3), 690-693. (link)

Sokal RR, Rohlf FJ. 1995. Biometry. 3rd ed. New York: WH Freeman.