I want to allow users of GraphPad Prism to use the False Discovery Rate (FDR) approach to multiple comparisons after nonparametric one-way ANOVA (Kruskal-Wallis). The idea is first to compute an exact (not corrected for multiple comparisons) P value for each comparison, and then to use a standard FDR algorithm to decide which of those P values are small enough for the associated comparison to be designated a "discovery".
How to compute the P value for each comparison?
- Use the Mann-Whitney test.
- Use the Dunn's test, usually used for multiple comparisons, adapted to give an uncorrected P value for each comparison.
Dunn's method uses the rankings from all the groups (and the sample size from all the groups) even when just comparing two groups. Does this add power over doing the Mann-Whitney test for each pair?
In another CV case, Alexis posted "Dunn's test preserves a pooled variance for the tests implied by the Kruskal-Wallis null hypothesis.". But I don't understand how pooled variance is a concern with nonparametric tests.
Best Answer
From what I understood of the OP question:
1) He ran a omnibus Kruskal-Wallis with significant results
2) He want to run a pairwise test on all groups and he is in doubt whether to use Mann-Whitney or Dunn's test
3) He want to run his own multiple comparison adjustment procedure, so he needs the uncorrected p-values of each pairwise comparions.
The source of confusion is that Dunn test implemented in GraphPad seems to already include a multiple comparison adjustment (which looks like a Bonferroni adjustment - see http://www.graphpad.com/guides/prism/6/statistics/index.htm?stat_nonparametric_multiple_compari.htm).
Answering:
2) You should use Dunn test. Both the CV answer by @Alexis for Post-hoc tests after Kruskal-Wallis: Dunn's test or Bonferroni corrected Mann-Whitney tests? and this site from XLSAT http://www.xlstat.com/en/products-solutions/feature/kruskal-wallis-test.html agree that Dunn (or Conover-Iman or Steel-Dwass-Critchlow-Fligner ) are the appropriate post-hoc tests after a KW (disclosure - I did not know that until today - have been using Mann-Whitney as post-hoc to KW until today).
3) I did not understand the GaphPad page, but let me point you to the dunn.test package in R does what the OP want. In particular it distinguishes the Dunn test and multiple comparison adjustments, and one can set the adjustment method to "none", which will return the unadjusted p-values. Also notice that among the adjustment procedures there are the Benjamini-Hochberg (95) and the Benjamini-Yekutieli (2001) adjustments that are FDR (maybe one of them is the one the OP is thinking in using).
Let me stress of many of the commentators have been saying - there is no good reason to use the unadjusted p-values EXCEPT to implement your own adjustment procedure - no decision should be made based on the unadjusted p-values.