I am doing final revisions to my dissertation, which includes a large number of regressions. I have six outcome variables (DVs) and I am testing the effect of four primary (correlated) IVs, but I am testing them separately to determine the unique contribution to the model of each IV. Therefore, I have 24 regressions.
One of my committee members would like me to use Bonferroni correction to control for Type I error; however, I am struggling with determining the number to use to determine the significance threshold. I guess the main thing I'm struggling with is that while I include these 24 regressions in my dissertation, any publication that is derived from my dissertation will not include all of these analyses, so setting the Bonferroni correction to .05/24=.002 seems (1) extremely conservative and (2) would be less conservative in a subsequent journal publication, as I would be correcting for fewer tests.
Are there other ways to approach determining the number of "tests" that alleviate Type I error concerns and make theoretical sense without being so conservative? I've been struggling to find some useful resources/articles for the last few days that account for a situation like mine, but most discussions of Bonferroni (or Holm-Bonferroni or other derivatives) focus on treatment of a single regression or analysis rather than a range of analyses across a dataset. I also realize that this is one of the biggest criticisms of Bonferroni–that the idea of constantly adjusting the alpha with each test would (in theory) require us to put off publishing anything until we had exhausted a given dataset.
Any links to useful articles or advice on how to approach my current problem would be greatly appreciated!
Best Answer
To adjust for multiple testing, instead of controlling the family-wise error rate as Bonferroni adjustment does, you could control the false discovery rate, for example by means of the Benjamini-Hochberg procedure. While this is qualitatively different - you no longer control the proportion of false positives of all tests you did, but the proportion of false positives of the tests you did that turned out positive - it is much less conservative.
Judgement of whether this is appropriate in your situation would require more detailed knowledge of your application.
The wikipedia article might be a good place to get an overview.