I want to conduct a comparative analysis across a large amount of studies by looking at the effect sizes of their results with another variable. I am not too familiar with effect sizes so I'd appreciate some help.
All the data is non-parametric and some studies use different tests (Wilcoxon rank sum for the most part but sometimes ANOVAs or Mann-Whitney U tests). Is there a meaningful way to compare the effect sizes across all these studies? About half of the studies present the data with means and standard errors/deviations from which I can get Cohen's d, but is this applicable to non-normal data? Or can I just use the r=Z/SQRT(N) formula on all the test values? Additionally, can I compare effect sizes obtained from means + standard deviations with effect sizes obtained from test statistics?
Any help would be appreciated, clearly I'm a bit lost.
Best Answer
Cohen's D is not a reliable effect size for non-parametric methods.
I have seen the formula r=Z/SQRT(N) suggested here on this forum but am unsure about its assumptions. In my reading about non-parametric effect sizes I have learnt that factors such as skewed distributions, random sampling or statistical independence can influence the reliability of a non-parametric effect size measure. source 1, source 2, source 3, source 4.
I don't see any problem if you use means and standard deviations to compare effect sizes based on parametric tests. But I would be careful applying these measures to non-parametric test results.
Comparing Cohen's D formula with the non-parametric one you select (there are many different ones) gives you an idea how comparable the effect sizes are.
I guess the crux is that you need more information about the samples of the studies you are comparing, which you probably don't have. How have other meta reviews in your field dealt with it?