Solved – Comparison of medians in samples with unequal variance, size and shape

meanmediannonparametrict-testwilcoxon-mann-whitney-test

What am I trying to do and how does my data look?

I am trying to perform some comparisons of medians of groups of data that, most often than not, are not-normally distributed. In some cases I am comparing some groups with 100-200 data points, but in others I'd like to compare groups with as little as 4 vs 10 data points if possible. Practically all of the compared groups have unequal sample size.

How am I doing this?

To this regard, I am doing the following:

  1. Using the Kolmogorov–Smirnov test to compare the shape of the two samples (i.e. testing whether data1 exhibits a distribution that follows that of data2; In Mathematica: KolmogorovSmirnovTest[data1,data2]);
  2. If shape is statistically similar, I use the Mann-Whitney U test to compare medians;
  3. In most cases, groups do not have statistically similar shape. I therefore would like to perform a t-test with Welch correction (as suggested in this answer and in this wikipedia page). However, this has not been implemented in Mathematica and I currently do not have the time to do it myself.

I understand that by shifting to the Welch t-test I am no longer comparing medians but means, but I would do this if no other option is available.

Questions:

  • Is there other test for comparing medians between samples with unequal variance, shape and size?
  • What is the minimum sample size that I could take? I am aware that no minimal sample size exists for t-tests, because it depends on the power of the analysis. But maybe comparing shapes of distributions of 4 vs 10 points is not very good… And in such cases, the Central Limit Theorem should not have a very strong effect
  • Anyone aware of such implementation in mathematica? Or maybe how to perform a Welch t-test there?

P.S. – I have searched a lot, including in this forum, but nothing shows up in this regard. I am truly sorry if this question was already answered before.

Additional Information:

In the comments it was asked some additional information. I hope this all makes sense, and I'm sorry if I cannot provide higher context into our real question.

  • The information for each sample was collected from the genomic composition of many organisms. So I have two lists of independent samples in the form (0.01,0.7,0.15,...) whose distribution I do not expect to follow any particular trend (i.e. Normal, Log-normal, etc).
  • For each organism we have the whole frequencies, and not only a sample, but for the groups to compare: we only have a small sample. Ideally, we would be able to collect information for some thousands of organisms, but this is not feasible or currently available. So, in our largest group we have a few hundreds of organism, in the smallest groups we have some tens.
  • What we want to test is whether one population of organisms exhibits higher frequencies in one group than the other population. This because we want to get a location comparison, where the measure of comparison is representative of that population Quantile comparison, skewedness, standard deviation, etc, though informative, would possibly be not as good to get insight into how different the populations are (bearing in mind our particular biological question).So we want some general measure of comparison of both populations that permits simply saying stuff like "the two populations tend to not differ greatly in [this] overall measure" or "this population tends to exhibit higher median/mean frequency than the other".
  • I am aware that means/medians per se are not informative enough about the distributions of both populations. For instance, if one is heavily tailed, they may both exhibit similar medians but one of them may exhibit higher means. However, for our purposes, we believe that medians (or means if necessary) provide enough insight.

Best Answer

I would suggest that you consider relative summary effects/relative treatment effects methods of Akritas, Arnold, Brunner, etc. The best book on the subject, generally, is Nonparametric Analysis of Longitudinal Data in Factorial Experiments by Brunner, Domhof, and Langer which is very well-written and clear.

These methods are based on modified notions of Mann-Whitney and can accommodate many of the features you mention using standard statistical software.

Related Question