I have a data set with three groups, one of which is a control and the other two groups have been administered either a moderate or a high dose of a drug. The data are not normally or log-normally distributed, hence I am using the Kurskal-Wallis test to find if there is a difference among the three. Next, I would like to perform a post-hoc (the dosages were not planned ahead and the reason behind this is kind of complicated; however, I'm not sure if this warrants the use of the term "post-hoc" or not) analysis of the two dosages to the control (essentially, a non-parametric analog of Dunnett's test [1]).
I understand that I can run a multiple comparison test (such as MATLAB's "multcomp") to answer my next questions. However, I'm not sure which correction procedure is most suitable to use on my data. The two procedures that I suspect are the most relevant are Tukey's HSD and Dunn-Sidak.
My understanding is that there are no good multiple comparison tests that do not assume normality, though the ones that do are generally robust to the assumption of normality ([2] Ch. 4).
On the other hand, since I only need 2 of the 3 possible comparisons, I thought I could use Dunn's procedure and again according to [2] this could have a higher power than HSD if fewer comparisons that there are possible are desired.
How would one choose between Tukey's HSD and Dunn-Sidak in general? More specifically, how about in my design?
Finally, in [3], there is a section (§ 11.5.a) that introduces what's called "Dunn's test" therein (and the reference cited is [4]). Is this the same as the Dunn-Sidak correction procedure? If not, should I use that instead? If so, will this not run into the problems mentioned in Post-hoc tests after Kruskal-Wallis: Dunn's test or Bonferroni corrected Mann-Whitney tests? ?
Thank you.
[1] C. W. Dunnett (1955). A Multiple Comparison Procedure for Comparing Several Treatments with a Control, J Amer Statist Assoc, 50: 1096-1121.
[2] L. E. Toothaker, "Multiple Comparison Procedures", 1993.
[3] J. H. Zar, "Biostatistical Analysis", 2010.
[4] O. J. Dunn (1964). Multiple comparisons using rank sums, Technometrics 6: 241-252.
Best Answer
Tukey HSD, Dunn-Sidak correction, and Dunn test are all different things, and aren't really alternatives to one another.
Tukey HSD is a method to compare among means of multiple groups, as would be common as a post-hoc test after a one-way anova. It is limited by requirements for normality and homoscedasticity. Modern statistical software has techniques with fewer restrictions, e.g. this, and this.
Dunn-Sidak, usually just called Šidák, is a method of correcting either the alpha value or the p-value of a series of hypotheses to control the familywise error rate. It can be used for any collection of (independent) hypotheses. That is, not just to compare multiple means. It's not clear in the question what the "Dunn's procedure" would be in regard to post-hoc testing. Perhaps pairwise t-tests with alpha or p corrected by Sidak.
Dunn test is a post-hoc procedure appropriate to follow a Kruskal-Wallis test. In this sense, it is analogous in use to Tukey HSD, though very different.