Solved – A mixed ANOVA showed no significant interaction, but a dep. t-test showed a significant effect for only one of the groups

anovainteraction

I have come across something that has got me kind of confused, I hope someone can help me out.

I did a mixed ANOVA with one within subjects factor with two levels 'time' and one between subjects factor 'group'. It was a comparison between a positive group and a negative group on scores pretraining and posttraining.

The results showed a significant main effect but no significant interaction effect. I should have stopped there maybe, but I did separate dependent t-tests for each group. These indicated that the scores of one group had changed significantly but the scores of the other group had not.

So now I don't understand anymore. Wouldn't a significant interaction effect indicate that there was a change in scores and that this change was affected by group (that the decrease in scores was significantly different between the groups)? Since there was no significant interaction shouldn't I find similar changes in both groups (either both significant or both not)?

Well, I hope this turns out to be easier than it seems to me right now, merci d'avance.

S.

Best Answer

It is easier than it seems. Significance is not a cutoff - it's not that things are significant, so there's a difference, or there isn't. Significantly different does not mean "different" and "not significantly different" does not mean "not different".

Imagine a succession of values: B is higher, but not significantly higher than A, C is higher, but not significantly higher than B, etc up to K is higher, but not significantly higher than J. But K is significantly higher than A. That makes sense, as each difference is small - you have the same issue.

This is (part of) the message of the paper by Gelman and Stern, called The Difference Between “Significant” and “Not Significant” is not Itself Statistically Significant, which you can find here: http://www.stat.columbia.edu/~gelman/research/published/signif4.pdf .

Related Question