This question comes from a discussion on the recent post by @rvl It's all in the family; but do we include the in-laws too?
Here's a common scenario that I've seen many times. A researcher runs a simple linear regression with, say, 5 covariates.
lm(Y ~ X1 + X2 + X3 + X4 + X5, data = df)
Ignoring interaction effects for the moment. They run the summary on the regression and observe the following:
Estimate Pr(>|t|)
X1 a 0.10
X2 b 0.47
X3 c 0.04
X4 d 0.38
X5 e 0.12
From this, they conclude that covariate X3 is a significant predictor of outcome Y. I've seen this done many times.
My question is, why do we not have to adjust these $P$-values for multiple comparisons? Are we not doing 5 tests simultaneously, even though they are covariates, thus increasing the chances of seeing a false positive? Assuming 5 completely independent tests, there would be a $1-(1-\alpha)^M = 1-0.95^5 \sim 0.23$ or 23% chance of seeing a false positive, rather than than usual 5%, however this is in no way indicated in the reporting of the "significant association".
This article from the question Is adjusting p-values in a multiple regression for multiple comparisons a good idea? seems to indicate that if you are doing some kind of stepwise model selection, then it is advantageous to correct the $P$-values of your covariates to account for the increased type-1 error rate. This seems to indicate that tests covariates do not act differently than usual tests.
Has anyone had any experience with this? I would love to hear any flaws in my logic, or reasons why this should not be done.
Best Answer
For the multiple testing problem it might be good to take a look at Family-wise error boundary: Does re-using data sets on different studies of independent questions lead to multiple testing problems?.
In your example above, if you estimate a regression on one sample, then you can, with a t-test only decide on the significance of an individual coefficient, so, yes, there is a multiple testing problem if you draw conclusions for multiple coefficients, based on multiple t-tests.
Let us call the coefficients $\beta_i, i = 1, 2, \dots 5$, then you can test $H_0^{(1)}: \beta_1 = 0$ versus $H_1^{(1)}: \beta_1 \ne 0$ with a t-test and conclude that $\beta_1$ is significant. Note that, if you can not reject $H_0^{(1)}$ that you can not conclude that $\beta_1$ is zero (see What follows if we fail to reject the null hypothesis?).
So if you want to find 'statistical evidence' for $\beta_1$ not being zero, then your $H_1^{(1)}$ must be the expression that you want to 'prove', i.e. $H_1^{(1)}: \beta_1 \ne 0$ and then $H_0^{(1)}$ is the opposite, i.e. $\beta_1=0$. As you assume $H_0^{(1)}$ to be true (to derive a statistical contradiction) you have a fixed value for the parameter $\beta_1=0$ and therefrom it follows that you know the distribution of the estimator $\hat{\beta}_1$ (see theory on linear regression) and you can compute p-values.
Let us now take the case where you want to show that $(\beta_1 \ne 0 \text{ and } \beta_2 \ne 0)$, then this must be your $H_1^{(1,2)}$ and the opposite $H_0^{(1,2)}$ is that either $(\beta_1 = 0 \text{ or } \beta_2 = 0)$, as there is an 'or' in there you can not fix all the parameters of the combined distribution of $(\hat{\beta}_1, \hat{\beta}_2)$ !
Can you apply multiple testing procedures ? Most of them assume that the individual p-values are independent, in this example $\hat{\beta}_1$ and $\hat{\beta}_2$ can not be shown to be independent !
But, in an advanced book on econometrics (e.g. W.H. Greene, "Econometric Analysis") you will find applicable test for J (simultaneous) linear restrictions ($\beta_i=0, i=1,2,3,4,5$ is a special type of 5 linear restrictions) that avoid the multiple testing problem.