Statistical Significance – Non-significant Kruskal-Wallis With Significant Post Hoc Tests Explained

kruskal-wallis test”p-valuepost-hocstatistical significance

I have done this analysis where in Figure A-D. the Kruskal-Wallis global p-value is non-significant, but the p-value for pair CS and SCS in Figure A however is significant (at P<0.05). I read somewhere that if the global p-value is non-significant we don't report post-hoc pairwise tests. So In figure A, B, C, D and H, should I just delete the p-values for paired tests as the global p-values in these plots are not significant? Could someone please clarify this?

enter image description here

Best Answer

In general, if you use an omnibus test, such as an ANOVA F-test or a Kruskal-Wallis H-test, it is illogical and poor practice to conduct pairwise comparisons when you fail to reject the null hypothesis on the omnibus test. Conducting the comparisons flies in the face of the omnibus: insufficient evidence to conclude differences does not warrant further investigation, as a general rule.

Usually, I would say report analyses you run, but in this case (which is different from selective reporting), the post-hoc p-values are inappropriate to interpret and should be omitted. The omnibus p-value is appropriate since this is the “gatekeeper” test.