Solved – Confidence intervals and bonferroni correction

bonferroniconfidence intervaleffect-size

Currently I'm thinking about using confidence intervals to compare the difference in means between a few groups. However, if I understand correctly I need to use the Bonferroni correction if I test multiple hypothesis with the same data set. Obviously, this means that my confidence interval becomes more strict. I wonder if the Bonferroni correction is really neccessary. I read papers that discourage using the Bonferroni correction and one should rather take a look on effect sizes to interpret the results.

Are they correct? Is the Bonferroni correction really overrated and one should rather use effect sizes to interpret the results?

Best Answer

If you wish to control the familywise type I error rate, then you need to adjust for multiplicity. In particular, if you wish to emphasize p-value, statistical significance or whether CIs exclude some value, to claim that some of multiple comparisons you are looking at are "statistically significant", then that is often a situation where that might be something you wish to do.

The Bonferroni correction is one of the most simple (and most conservative) adjustments for multiplicity. You can easily adjust your CIs to match the adjustment (e.g. with 2 hypotheses you calculate 97.5% confidence intervals instead of 95% CIs). Other adjustments are uniformly more powerful (e.g. Bonferroni-Holm), but make it hard to find matching CIs.

There are of course approaches for dealing with multiplicity other than controlling the familywise type I error rate, e.g. using shrinkage in Bayesian hierarchical models instead.