Hypothesis Testing – Meaning of Single-Step and Multi-Step in Post-Hoc Testing of ANOVAs

anovahypothesis testingmultiple-comparisons

From Wikipedia

Post-hoc testing of ANOVAs

Multiple comparison procedures are commonly used in an analysis of
variance after obtaining a significant omnibus test result, like the
ANOVA F-test. The significant ANOVA result suggests rejecting the
global null hypothesis H0 that the means are the same across the
groups being compared. Multiple comparison procedures are then used to
determine which means differ. In a one-way ANOVA involving K group
means, there are K(K − 1)/2 pairwise comparisons.

A number of methods have been proposed for this problem, some of which
are:

Single-step procedures

  • Tukey–Kramer method (Tukey's HSD) (1951)
  • Scheffe method (1953)

Multi-step procedures based on Studentized range statistic

  • Duncan's new multiple range test (1955)
  • The Nemenyi test is similar to Tukey's range test in ANOVA.

  • The Bonferroni–Dunn test allows comparisons, controlling the familywise error rate.[vague]

  • Student Newman-Keuls post-hoc analysis
  • Dunnett's test (1955) for comparison of number of treatments to a single control group.

I was wondering what "single-step" and "multi-step" mean here?

Are they both for pairwise comparisons for every two groups, right?

Thanks and regards!

Best Answer

Single-step and step-wise relate to the view of the procedures as dynamic.

A single-step procedure implies there is no dynamics: without looking at the data, the procedure offers some rejection threshold.

A step-wise procedure implies there is dynamics: rejection boundaries are data driven and are updated along the sequence of p-values/test statistics in the data.

In reality, there is no real dynamics, as even the step-wise procedures take all test statistics, and returns a rejection boundary. The name stems mainly from the motivation to the procedure.