I conducted an experiment to investigate the effects of musical complexity on visual attention and am now analyzing my data.
I ran a two-way repeated measure ANOVA, with my two factors being Musical condition (control, simple, complex), and the second factor being flanker type (neutral, congruent, incongruent). Now the results indicate a significant difference for both factors but also their interaction (my DV is reaction time). Pairwise comparisons tell me which musical conditions are significantly different from each other, and the same goes for flanker type.
But for the interaction between them, Musical Condition*Flanker Type (such that I want to know whether there were significant difference between Simple Music+neutral flankers vs. Complex Music+neutral flankers for example), am I correct in saying that I have to run a paired t-tests? If I want to compare each condition combination to each other, I would have to run 9 paired t-tests, correct?
If I do that, would I have to divide my significance level of 0.05/9 to account for family wise error?
Best Answer
I would first run simple omnibus tests of the effect of one variable conditional on the levels of the other. That would be analogous to running three one-way ANOVAs where you test the effect of Musical condition on response time for the neutral level of flanker, for the congruent level of flanker and for the incongruent level of flanker. The only difference is that if homogeneity of variance is assumed, you would use MS_error (aka MS_within) from the two-way design and its degrees of freedom as the denominator for testing each one-way ANOVA.
In general, if we give the conventional 0.05 to each main effect and the interaction, then each simple effect should get 2*0.05 = 0.10 because, for example, $SS_{B|A} = SS_{B} + SS_{A \times B}$. That is, the sum of squares for the simple effect is equal to the sum of the sum of squares for the main effect plus the interaction.
Thus, each of your three simple omnibus tests would use an alpha significance level of 0.10/3 = 0.033. Any significant simple one-way ANOVAs should be followed up with pairwise comparisons. The non-significant simple one-way ANOVAs should not be followed up on.
For the follow-up pairwise comparison, I would use J.P. Shaffer's logical procedure for planned post-omnibus pairwise comparisons. In the three group case, this is equivalent to Fisher's LSD (i.e., no adjustment; see Levin, Serlin, & Seaman, 1994, Psych. Bulletin for a citation).