In this suspiciously brief comment on another article, the author suggests that analysts should only correct for multiple comparisons in univariate regression when the predictor has more than two levels, and that in non-hierarchical multivariate regression, with several or many predictors
"…statistical correction of the P value is not conventionally applied as this is a procedure that controls for multiple variables simultaneously"
They assert
"It is rare to see textbooks recommending adjustments for P values while testing partial regression coefficients in an M(ultiple) L(inear) R(egression) analysis."
Is the author right? Is imposing type-1 error control on p-values using procedures like the Hochberg procedure unnecessary in multivariate regression?
Best Answer
If you want to control the type I error rate* across multiple comparisons from a single regression model with multiple independent variables (this is what the author of the linked comment talks about, while multivariate could also/usually does refer to having multiple dependent variables/outcomes), then you need to do an adjustment.
While the author says that a correction is not required in this setting, the only argument the author offers is that it's not usually done ("not conventionally applied"). Fair enough, that is true, but it does not change that by not doing it you would inflate the type I error rate across these multiple comparisons.
It gets even worse: The authors proposes pre-selecting terms for the model based on first looking at models with each independent variable on its own (aka "univariate pre-screening"), which has long been known in the statistical literature to be a bad idea (it will certainly invalidate subsequent p-values even if they were done with some multiplicity correction).
In summary, the main value of the reference you linked is a description of the research practices being used by (part of the) people in one particular research area. These research practices do not seem to follow statistical best practices.
* We can of course debate whether type I error rate control is desirable, useful etc., but let's take it as a given that that's what we aim to do. Alternatively, we could of course have other aims, which might be better served by other approaches. E.g. we could of course do false discovery rate control, or some kind of Bayesian hierarchical models that shrink all predictors (for standardized covariates) e.g. with the regularized horseshoe prior, or do some kind of prediction modeling e.g. with elastic net regression with regularization strength chosen based on some suitable cross-validation.