There has been some discussion about why differences exist between factor significance after an ANOVA and no significance in the pairwise comparisons, and it has been very helpful. However, there is no directions regarding how to proceed. For instance, in my case the overall P value given by a one-way ANOVA for the treatment factor was P=0.0127 but all of the treatments came out as not significantly different from each other following Tukey's LSD. Should I run a different post-hoc test in this case? Which one would you recommend?
Thank you so much!
Best Answer
This is not an area where there is universal agreement.
My view is that 1) the two tests answer different questions, so it's not surprising that they get different answers. 2) This discrepancy is more a demonstration of the problems of p values and, especially, using cutoff values like p < .05. 3) It also gets at the problems of looking for "cookie cutter" methods of doing statistics.
Elaborating a bit: The overall ANOVA asks about the relationships among all the levels; tests like Tukey's LSD compare individual levels. Those are two different questions. Which one are you interested in? Perhaps both?
You say (in the title) that the results are inconsistent. But you only give evidence that the p-values are on opposite sides of .05. So? Why do you care about that? It would be much better to examine effect sizes. And these effect sizes don't have the problem of being inconsistent, because they don't vary.
Finally, it seems like (but perhaps I am wrong) you are asking for a general solution to a general problem; but really, the questions are particular.