I have animals which took either drug A or drug B (drug factor), and these same animals were either dehydrated or euhydrated (treatment factor).
The F-tests for my ANOVA model tell me that only treatment is significant (drug isn't, and drug*treatment interaction isn't either); but Student–Newman–Keuls (SNK) post-hoc tests show that drug-A-dehydrated is different from drug-B-dehydrated.
Why are the post-hoc tests for this difference significant if only treatment factor is responsible for the differences? Should I change post-hoc? What (different) questions are F test and post-hoc answering?
Best Answer
If I were you, I wouldn't bother with any multiple comparisons testing.
Your two-way ANOVA answers all the relevant questions. You've learned that dehydration makes a difference (by P value; you really should quantify how large that difference is and assess whether that difference is large enough to care about). You've also established that there is no evidence that drug A has a different effect than drug B, or that the effect of dehydration differs between A and B (no statistically significant interaction). I don't see any thing else to test, so would ignore the SNK results.
Why are the SNK results different than the ANOVA interaction results? Without seeing the data, of course it is impossible to know. But the SNK test is not highly regarded, and doesn't really control the familywise significance level the way it is supposed to (MA Seaman, JR LEvin and RC Serlin, Psychological Bulletin 110:577-586, 1991).