Solved – report simple main effects without an interaction effect

anovageneralized linear modelinteraction

I have a quick question about reporting simple main effects. I am running a two-way mixed ANOVA and the interaction effect was not significant. I understand that if you have no significant interaction effect, you should not carry out a test of simple main effects and instead just interpret the main effects. I really don't want to do this, the main effects give me nothing. So I carried out the test of simple effects anyway (just to see) and they are exactly the results I was expecting/looking for, and now I'm doubting my source which claims you need a significant interaction to run test of simple effects? Any help would be greatly appreciated.

Best Answer

Generally, when somebody says that data must satisfy some condition C (e.g., they must produce a significant interaction term in an ANOVA) before you can use a follow-up procedure (e.g., tests of simple main effects), what they mean is that any guarantees that the follow-up procedure will be accurate or useful require C. So, if C does not hold, you have lost your reason to believe anything the follow-up procedure tells you. If doing the follow-up test gets you results you wanted anyway, that's no reason to believe them. Thinking that your analysis is correct because it got the results you wanted is just wishful thinking.

The caveat to the above is that the mechanistic procedures suggested by certain quantitatively weak textbooks in quantitatively weak fields (as a psychologist, I ought to know) where you do significance tests and then make further analytic decisions on that basis (e.g., a lack of significant departure from normality means that ANOVA is appropriate) have, to my knowledge, no good basis in mathematics or in science. They are cargo-cult statistics.