This may be a really basic question, but I haven't found any helpful answers. Or I simply don't understand.
When correcting p-values of linear regression models, which p-values are corrected?
My problem:
I ran two multiple linear regression models with each 3 predictors and one dependent variable. Now I would want to run a FDR (Benjamini-Hochberg) correction. But I am not sure which p-values to correct and how.
1) I thought about correcting the p-values of all 6 coefficients of both models (t-statistics). I would sort them in ascending order, following this approach.
Would this be the correct way for adjusting the p-values? And what about the p- values of the F-Statistic of both models?
Thank you in advance.
Best Answer
It is good that you are concerned with the type I error rate of running multiple models. However, keep in mind that any kind of correction will come at the cost of an increased type II error rate: By being more strict you will lose power.
Hence, only perform and correct for tests that are directly relevant to your research question. A summary of a linear model may automatically generate a $p$-value for the intercept, but if this test is meaningless (as it usually is), correcting for it would be wasteful in terms of power, if it weren't significant. If it is significant and you use e.g. the Benjamini Hochberg procedure, you could even inflate your false positive rate by including a nonsense result.
Another example is including confounders in your model. If you include a variable because you have reason to assume there might otherwise be omitted-variable bias, then its significance is not relevant. It need neither be checked, nor corrected for.
Likewise, suppose your research question revolves around the presence (or absence) of an interaction. In this case, significance of the marginals is irrelevant.
To conclude, first determine which $p$-values are actually of interest, then correct for those.