See the related, but old question: Correcting p values for multiple tests where tests are correlated (genetics).
Multiple comparison methods based on bootstrap have the advantage of taking account of dependence structure of pvalues. Regression models are just a little more difficult to bootstrap, though. There have been several methods proposed.
I've come across at least three ways of adjusting p-values in such problems.
As far as I see it, there is no single "best" solution. The question is: what methods are available and what are the advantages/disadvantages of each?
I propose the following definitions:
$\vec{\theta_0}$ is vector of the hypothesized values of the parameters, which is assumed under the complete null hypothesis.
$\hat{\vec{\theta}}$ is an estimator of the parameters, calculated on the original sample.
$\hat{\vec{\theta}^*}$ is an estimator of the parameters, calculated on one of the bootstrap samples.
$T$, $T^*$ are pivot statistics calculated on original data and bootstrap sample. All pivot statistic are assumed to be Wald, i.e. $T = \frac{\theta}{\operatorname{SE}\theta}$.
$m$ is number of terms of interest in all regressions.
Best Answer
Method 1: Naive bootstrap
This method violates both guidelines stated in the article by Hall P. and Wilson S.R. "Two Guidelines for Bootstrap Hypothesis Testing" (1992), so it lacks power (in our case should be too conservative).
Method 2: Free step-down resampling (MaxT) using Wald statistics
The name and decription comes from the book by "Applied Statistical Genetics with R (2009)" by Foulkes Andrea.
The problem is that this method is not working, when one cannot simply calculate the regression residual, when e.g. there are many independent regression models, and some of them share the same dependent variable.
Method 3: Null unrestricted bootstrap
This method is very similar to Free step-down resampling, with the difference that instead of calculating the residuals, one adjusts the $T$ statistic: