When deconstructing my mixed effects model, I found a three-way significant interaction. I calculated my p-value by using maximum likelihood ratio tests allowing for a comparison of the fit of the two models (the model with all predictors minus the model with all predictors but the predictor of interest – in this case, the three-way interaction). When I conduct follow-up comparisons of the three-way interaction, do I need to correct the alpha level of significance with Bonferroni correction?
Thanks for all input!
EDIT: (merged from answer –mbq)
I want to look at the significance of the three-way interaction and then…I wanted to look at any other significant interactions within that first three-way interaction.
I use the same dataset…the model is a crossed random effects of participants and items (the data is comprised of repeated observations (response times) with Valence (positive and negative) and age as a between subjects factor as well as two continuous predictor variables (attachment dimensions). Thus my model is Valence x Age x Attachment anxiety x Attachment avoidance. I found that Valence x Age x Attachment avoidance is significant. However, I want to examine this interaction further. I did this by examining the same model but just for young adults vs old adults separately. Thus, I found with older adults a significant interaction of Valence and Attachment avoidance. However, when I calculated the p-value (as described above) of this two-way interaction, can I take the p-value as is or do I need to correct with Bonferroni? And if so, how? I hope this is clearer?
Thank you!
Basically I want to examine the 'direction' of my three-way interaction and test whether or not the differences within that interaction is significant.
Best Answer
It sounds like you basically have a problem of model choice. I think this is best treated as a decision problem. You want to act as if the final model you select is the true model, so that you can make conclusions about your data.
So in decision theory, you need to specify a loss function, which says how you are going to rank each model, and a set of alternative models which you are going to decide between. See here and here for a decision theoretical approach to hypothesis testing in inference. And here is one which uses a decision theory approach to choose a model.
It sounds like you want to use the p-value as your loss function (because that's how you want to compare the models). So if this is your criterion, then you pick the model with the smallest p-value.
But the criterion needs to apply to something which the models have in common, an "obvious" choice based on a statistic which measures how well the model fits the data.
One example is the sum of squared errors for predicting a new set of observations which were not included in the model fitting (based on the idea that a "good" model should reproduce the data it is supposed to be describing). So, what you can do is, for each model:
1) randomly split your data into two parts, a "model part" big enough for your model, and a "test" part to check predictions (which particular partition should not matter if the model is a good model). The "model" set is usually larger than the "test" set (at least 10 times larger, depending on how much data you have)
2) Fit the model to the "model data", and then use it to predict the "test" data.
3) Calculate the sum of squared error for prediction in the "test" data.
4) repeat 1-3 as many times as you feel necessary for your data (just in case you did a "bad" or "unlucky" partition), and take the average of the sum of squared error value in step 3).
It does seem as though you have already defined a class of alternative models that you are willing to consider.
Just a side note: Any procedure that you use to select the model, should go into step 1, including "automatic" model selection procedures. This way you properly account for the "multiple comparisons" that the automatic procedure does. Unfortunately, you need to have an alternative (maybe one is "foward selection" one is "forward stepwise" one is "backward selection", etc.). To "keep things fair" you could keep the same set of partitions for all models.