Solved – Mixed effects model hypothesis testing

hypothesis testingmixed model

I ran a 2 x 2 x 2 full factorial repeated measures experiment where 20 participants were exposed 30 times to all combinations of the factors A, B and C in random order. This is a standard procedure in response time experiments. 

A1 - B1 - C1 
A2 - B1 - C1 
A1 - B2 - C1 
A2 - B2 - C1 
A1 - B1 - C2
A2 - B1 - C2
A1 - B2 - C2
A2 - B2 - C2

I'm having a hard time figuring out how to optimally analyse the data using a mixed effects model. I've specified the factors as fixed and the participants with random intercepts, but I am not sure whether this is the correct thing to do.

My main hypothesis was that there will be a significant interaction between factors A and B, but only at level 1 of factor C. I'm interested specifically in the interaction of A and B at the levels of C since I predicted that the combination of (A1B1) – (A1B2) will be significantly higher than (A2B1) – (A2B2), but only for C1.

enter image description here

When I ran the model to see whether there was a three-way interaction between A, B, and C, I found that the interaction was not statistically significant.

(interaction_full<-lmer(dv ~ A * B * C + (1|participant), data_full))

When I ran separate models for the levels of C it turns out that I was right. There is a major A X B interaction for C1, but not for C2. Would this be problematic?

(interaction_C1<-lmer(dv ~ A * B + (1|participant), data_C1))

(interaction_C2<-lmer(dv ~ A * B + (1|participant), data_C2))

enter image description here

Where do I go from here to address my hypothesis? Would it be sufficient to run separate models for C1 and C2 and then indicate that because there was a significant A X B interaction when running the models on C1, but not on C2 that I can reject the null?

p.s Load is A, compatibility is B and salience is C.

Best Answer

As I understand it, this issue is not at all specific to mixed models, but is a variant of "the difference between significant and non-significant is not significant" (see also other blog posts by Gelman); see also Pockock et al.2002. Consider the following statements:

  • "within C1, we can reject the null hypothesis of no $A \times B$ interaction; the effect of A on responses differs significantly across levels of B (and vice versa)"
  • "within C2, we cannot reject the null hypothesis; the effect of A on responses does not differ significantly ..."
  • "we cannot reject the hypothesis that the $A \times B$ interaction is the same in both levels of C"

All three are true. As @Wolfgang points out below, the test of the three-way interaction is the formal test that there is a difference between the strength of the interaction across different levels.

The corresponding picture:

enter image description here

Related Question