This feels like a really basic question, but I haven't found a definitive answer.
When correcting p-values for multiple comparisons, are you supposed to only run the analysis/correction on the p-values that are significant before correction, or do you run it on all of them?
Or does it depend on whether you are running a FWER (Bonferroni, Holm, etc.) or a FDR (Benjamini-Hochberg, etc.) correction?
Best Answer
A general bonferroni correction requires that you correct based on the total number of tests that have been run. Given that corrections for multiple comparisons should be planned for in advance of the actual analysis, the results of the tests in question have no bearing on the validity of the multiple comparisons correction method.
Consider the case where you run 100 random tests, and you find that only one of them gives you significant results. If you only 'corrected' for that one test, then that one test remains significant even though you ran 100 tests, clearly this wouldn't make sense.
There are of course other ways to correct for random comparisons, especially if the tests are related in some form. However, all of the methods that I know of are independent of the actual result of the tests.