Hypothesis Testing – What Does It Mean for a Statistical Test to be Robust?

anovadefinitionhypothesis testingrobustterminology

Is there an intuitive way of understanding what these two sentences mean and why they're true?:

"ANOVA is 'robust' to deviations from normality with large samples", and…
"ANOVA is 'robust' to heteroscedasticity if the groups have similar sample sizes".

Best Answer

Roughly speaking, a test or estimator is called 'robust' if it still works reasonably well, even if some assumptions required for its theoretical development are not met in practice. Comments:

  • If you need to do one-factor ("one-way") ANOVA for data with different variances at each level of the factor, then it is best to use some variant of one-way ANOVA such as oneway.test in R that does not require equal variances. As you say, a 'pooled' t test or simple one-way ANOVA where the numbers of replications per factor differ greatly, may be problematic if variances also differ among levels of the factor.

  • Some texts seem to say 2-sample t test and one-way ANOVA are OK for non-normal data whenever there are more than 30 replications per group. But this may not be true if data within groups are highly skewed.

  • If levels of 2-sample t or one-factor ANOVA are far from normal, but differences between groups are mainly a 'shift' of location (with little change in shape or variance) then it may be best to use Welch t test or Kruskal-Wallis nonparametric test instead of t or ANOVA, respectively.

Note: I could show an example to illustrate, if you could say what test is of particular interest and what assumption you feel unsure of.

Related Question