I am interested in working out the correct way to correct for multiple testing in multilevel models for longitudinal data, where I am investing a potential interaction between two predictors.
My data consists of 7 outcome variables, 2 predictor variables with a interaction, and 5 covariates. For the model building strategy I am using a set up approach.
Model 1: Null model
Model 2: Time is introduced
Model 3: Covariates are introduced
Model 4: Predictors are introduced
Model 5: Interaction between Predictors are introduced.
This strategy is then repeated for each of the outcome variables.
Currently I am using bonferroni correction to account for multiple testing, .05/35 = .001, were 35 is 7 outcome variables multiplied by 5 models. Is this the correct way to perform the calculation for the bonferroni correction? Alternatively would you divide .05 by the number of final models in the analysis (7) or the number of parameters in the models been analyzed?
EDIT
The following paper may be of interest to anyone coming across this post.
Evaluation of multi-outcome longitudinal studies
Evaluation of intervention effects on multiple outcomes is a common scenario in clinical studies. In longitudinal studies, such evaluation is a challenge if one wishes to adequately capture simultaneous data behavior. In this situation, a common approach is to analyze each outcome separately. As a result, multiple statistical statements describing the intervention effect need to be reported and an adjustment for multiple testing is necessary. This is typically done by means of the Bonferroni procedure, which does not take into account the correlation between outcomes, thus resulting in overly conservative conclusions. We propose an alternative approach for multiplicity adjustment that incorporates dependence between outcomes, resulting in an appreciably less conservative evaluation. The ability of the proposed method to control the familywise error rate is evaluated in a simulation study, and the applicability of the method is demonstrated in two examples from the literature.
http://onlinelibrary.wiley.com/doi/10.1002/sim.6461/full
Best Answer
What are you doing with first 4 models?
From your description, I assume what you are interested in is the predictor variables and their interaction. You don't have hypotheses about the covariates or the time series. They are only there for validity purposes. Further, from your first sentence, it seems what you are really interested in is the interaction. Therefore, you are really making 1 test per dependent variable. Models 1-4 seem superfluous to me for the purposes of type 1 error correction. You are not using them to test hypotheses (or at least hypotheses of the same family), and therefore shouldn't be lumped into your familywise error correction.
Where things get tricky is the multiple dependent variable issue. Do you have predictions for all of the variables or is this data dredging (i.e. are you looking for the 1 variable where things work out, and you'll throw away the rest)? If you have specific, pre-planned (i.e. before data collection) prediction about the effect of your predictors on all the dependent variables, then you are really only testing one hypothesis, and you don't need to correct for anything. That means, though, that if any of your hypothesized patterns is wrong, then you cannot reject the null that all of your hypothesis is wrong. If you have 7 measures of, ostensibly, the same things, and you want at least one of them to come back significant, first, probably not the best research practice, but second, you are essentially doing 7 tests, so your adjusted alpha criteria should be $0.05/7=.0071$. However, again, if you measure 7 similar things in the hopes that one comes back significant, and only 1 does, you should think hard about what that means for your hypothesis (i.e. your hypothesis is probably way to broad).
To go one step further, however, you are really in the territory of multivariate multilevel modeling (aside: is there any way to work another multi- in there... multivariate multilevel multimodeling maybe?). If you are using R, here is another post that provides some help with the how-to. If you aren't familiar with multivariate modeling, the gist is you are simultaneously testing if the set of predictors predicts the set of dependent variables. However, if you aren't familiar it is probably better to stick to analysis you are familiar with and do some error correction and be very judicious with your analysis and conclusions.