Kruskal-Wallis and Mann-Whitney U – Multiple Comparisons Correction

false-discovery-ratekruskal-wallis test”multiple-comparisonswilcoxon-mann-whitney-test

I have 5 independent samples which I want to compare, to define if there are significant differences in the distribution of values across samples.

The samples' sizes are:
4562, 1116, 314, 151, 77

I don't assume that the samples are normally distributed, so I've decided to use:

  • Kruskal-Wallis test to have an overall suggestion that some samples are differently distributed
  • If Kruskal-Wallis p is significant ($<0.01$), I run post-hoc Mann-Whitney U test to compare each group to another
  • I am running several tests, so I want to correct for multiple comparisons. I was thinking of using Benjamini-Hochberg FDR

I have several questions:

  • Does the choice of these tests together make sense? Did I miss something?
  • Do I have to test if the tests are sufficiently powered?
  • How can I test for effect size between the samples? Just taking the difference between the medians?
  • If the choice of tests makes sense, since I want to run Mann-Whitney U test on the same data used for Kruskal-Wallis, do I have to account also for that in the correction for multiple comparisons, and if yes how can I do it?

Best Answer

I think your approach makes sense, but I would recommend the Dunn test as a post-hoc test. (No citation on the opinion.) This test is implemented in R, and in SPSS. I suspect it is found in other common software packages.

For effect size statistics for Kruskal-Wallis, I think the most common appropriate ones are epsilon-squared and Freeman's theta. It's important to understand that these effect size statistics are related to the probability that an observation in one group is higher than an observation in another group. That is, they don't measure the absolute difference in observations. For that, you might use the difference in medians, or some kind of standardized version of the difference in medians.

Related Question