Solved – Do we need a Bonferonni correction when running numerous univariate regression analyses

multiple-comparisonsregression

For my data, I need to run numerous linear regression analyses, separately for each of my two groups of subjects as well as different conditions. Together, I would have to run approximately 40 regression analyses across the 3 conditions for each group. I'm aware that we should correct for multiple comparisons for the regression coefficients within a model. But in my case, should I also do a similar correction for the p-value associated with the regression model itself (ie: the F test)?

For example, if in one condition, I have to run 25 regression analyses for each group, would it make sense to divide alpha=.05 by 25?

I've been going through different websites and textbooks, but can't seem to find an answer. Any suggestions/help would be appreciated!

Best Answer

As Glen says, it depends on your hypotheses, and especially the relative importance of Type I vs Type II errors. For example, if you're testing a new drug, you don't want to risk missing side-effects because they were rendered nonsignificant by the multiple comparison correction. A similar principle applies when testing for effects of contaminants in the environment. If you're data mining without strong a priori expectations, you might be more inclined to control the Type I error rate to ensure that any 'interesting' results you find are genuine.

Bonferroni in particular has fallen out of favour because it is very conservative (i.e. by controlling false positives you're drastically increasing the number of false negatives). Other options are the Sidak correction (which is less conservative when the family of comparisons is large), or my favourite, a false discovery rate (FDR) correction using some variant of the Benjamini-Hochberg procedure.

A family of comparisons (hypotheses) is difficult to define, but a loose definition is any group of comparisons in which you expect the difference to be in the same direction. For example, several alternative correlated measures of plant growth would certainly constitute a family of comparisons that should be corrected.

With 25 comparisons, you probably should do some correction. If you're using R, check out the fdrtool or qvalue packages. qvalue is especially easy, because it allows you to input a string of p-values from your 25 comparisons, and returns a list of 25 q-values (significance controlled for FDR).

In any case, make sure you include effect sizes (r, R^2, d). Effect sizes with confidence intervals are much more informative than arbitrary significance cutoffs.