Solved – Anova and post-hoc’s vs CI’s

anovaconfidence intervalpost-hoc

I'm wondering about the use of anova and post-hoc tests versus CI's.
If I have a three-way anova design (age group, sex, strain) with n replicates within each cell, the common way to analyze it would be anova, and then post-hoc tests to look for differences within each grouping factor. What I'm wondering is why can't I look at the confidence intervals instead. If, say, I want to compare the value of young females of one strain to that of young females in the other strain – I can generate the CI for each of the two groups (1.96*SEM for 95%, or using non-parametric bootstrap resampling the individuals with all their info intact) and then look at how much they overlap for a measure of how different they are and how significant are the differences. Seems a lot more straightforward to me then doing all these tests. Would that be valid or am I missing something?
Thanks

Best Answer

Comparing means using side-by-side confidence intervals may seem straightforward, but it is in fact inappropriate, so please don’t do it.

Sample means are examples of statistics; and differences between sample means are other statistics. These two kinds of statistics have different sampling distributions and different standard errors — sometimes dramatically different. There are many easy-to-construct examples where CIs overlap substantially but the differences are highly significant. It’s harder to construct an example of the reverse case, but it’s possible.

Repeat: One should not use CIs for means to test differences between them.

Related Question