My data consists of 3 groups, and 6 variables. The data are not normal so I want to use non-parametric tests to uncover significance. The first thing I wanted to get your opinion on is, is it acceptable for me to run 6 Kruskal-Wallis tests but use a Bonferroni correction ($p'=p/6$). If this is acceptable, is it then reasonable to use 6 subsequent and corrected Dunn's tests to see which variables are significant?
It seems to me that this way of doing it is clunky, but I am not aware of a better way to compare my data.
P.S. I am doing this analysis in MATLAB with the statistics tool box.
Best Answer
You would use the Bonferroni for a one-way test. But let's be clear:
You would not use the Bonferroni adjustment on the Kruskal-Wallis test itself. The Kruskal-Wallis test is an omnibus test, controlling for an overall false-positive rate.
You would use the Bonferroni for post hoc Dunn's pairwise tests. Indeed, Dunn introduced the "Bonferroni" adjustment.
You could also consider using more powerful family-wise error rate adjustment methods, such as the Holm-Sidak method, or still more-powerful false discovery rate adjustment methods, such as the Benjamini-Yekutieli adjustment. These and other multiple comparisons adjustments are implemented specifically for Dunn's test in Stata (within Stata type
net describe dunntest, from(https://alexisdinno.com/stata)), and in R (see http://cran.r-project.org/web/packages/dunn.test/).You might also consider using the more powerful Conover-Iman post hoc (only) test statistic. This test is implemented for Stata (within Stata type
net describe conovertest, from(https://alexisdinno.com/stata)), and for R in the conover.test package, and includes the same selection of multiple comparisons adjustment procedures.EDIT: I missed that you seem to be wanting something like a multivariate (multiple dependent variables) nonparametric one-way ANOVA. Katz and McSweeney offer such a generalization of the Kruskal-Wallis test, although I am not aware of an implementation in software. They also provide post hoc Scheffé-like univariate and multivariate procedures, which ought to be amenable to multiple comparisons adjustments. Their test is likewise omnibus for all variables.
References
Conover, W. J. (1999). Practical Nonparametric Statistics. Wiley, Hoboken, NJ, 3rd edition.
Conover, W. J. and Iman, R. L. (1979). On multiple-comparisons procedures. Technical Report. LA-7677-MS, Los Alamos Scientific Laboratory.
Dunn, O. J. (1964). Multiple comparisons using rank sums. Technometrics, 6(3):241–252.
Katz, B. M. and Mcsweeney, M. (1980). A multivariate Kruskal-Wallis test with post hoc procedures. Multivariate Behavioral Research, 15:281–297.