I'm analyzing reaction time data from a repeated measures ANOVA with the following design:
Factor 1 (between-group): GROUP (controls, clinical)
Factor 2 (within-group): TASK TYPE (social, non-social)
Factor 3 (within-group): DECISION DIFFICULTY (easy, hard)
ANOVA results indicate a significant 2-way interaction (GROUP×DIFFICULTY). No other main effects, 2-way interactions, nor the 3-way interaction are significant (all $p > .4)$.
Since there was no effect of task, I pooled data across the 2 task types. I'm not sure what is the right way to go from here in terms of reporting and interpreting the results. My specific questions are:
-
Should I always perform follow-up tests with a significant 2-way interaction, or should I just stop there and interpret based on the figures (and without further stats)?
-
Between- or within-group post-hoc: Should post-hoc tests be restricted to an independent t-test between groups (i.e. comparing "pooled easy" data between groups, and "pooled hard" data between groups)? Can I also perform and report a within-group paired t-test comparing "pooled easy" to "pooled hard" within each group?
-
Interpretation: If neither independent t-test from (2) is significant, can I still interpret the 2-way interaction based on a significant paired t-test from (2)? (The clinical group's reaction times appear to increase with task difficulty, whereas reaction times don't change with task difficulty in controls.)
In the absence of significant between-group t-tests, can I still report these results based on the 2-way interaction and a within-group t-test as "task difficulty affects reaction times in the clinical group and not the control group"?
Thank you. I appreciate any help you can give!
Best Answer
Pooling across the remaining group was correct. You don't need anymore tests. Take choice 1. Your interaction means the effect of difficulty (hard - easy) in controls is different from the effect of difficulty in clinical. Since what it means is exactly what you want to know a post hoc is completely unnecessary.
You might want to see Gelman and Stern (2006) for a lesson in why the path you were thinking of taking with post hocs was faulty logic.
Gelman, A., & Stern, H. (2006). The difference between “significant” and “not significant” is not itself statistically significant. The American Statistician, 60(4), 328-331. doi: 10.1198/000313006X152649
Some general information might help others in the same situation. There seems to be a broadly accepted mistaken belief that there must be a significant simple effect when you have a significant interaction. Not only is this false but there is no guarantee of any significant simple effect in an ANOVA with more than 2 levels (in which case it's just an alternative to a t-test). Interactions are about differences among differences. None of the individual differences needs to be significant, or even close to significant, for a significant interaction. Consider a situation where one effect is +5 and another is -5. The difference between those effects is 10, which is much larger than either simple effect. Therefore, it would be very easy to not have a significant simple effect of 5 but a significant interaction of 10.
With something as simple as a 2x2 designs one should never perform a post hoc on the interaction because the interaction told you all of the necessary information. Even with slightly more complex designs post hoc tests are usually unnecessary, especially when one variable only has 2 levels. People generally do far too many post hoc tests following ANOVAs. In school they tell you to do an ANOVA to avoid multiple comparison problems. Of what benefit is that when, after you perform the ANOVA, you perform a bunch of comparisons? Interpret the ANOVA first.