Solved – What are the differences of the Student’s T-Test, KS-Test and the MWW-Test

hypothesis testingkolmogorov-smirnov testt-testwilcoxon-mann-whitney-test

I have got a question that

What are the differences of the Student's T-Test, Kolmogorov-Smirnov Test (KS-Test), and the Mann-Whitney-Wilcoxon Test (MWW-Test)?

In the case of the T-test, the null hypothesis is μ1 = μ2, indicating that the mean of feature values for class 1 is the same as the mean of the feature values for class 2. In the case of the KS-test, the null hypothesis is cdf(1) = cdf(2), meaning that feature values from both classes have an identical cumulative distribution. Both tests determine if the observed differences are statistically significant and return a score representing the probability that the null hypothesis is true (From Levner 2005).

How about the MWW-Test? Are there any assumptions of the distribution of the data for these tests?

When we want to test if there is significant difference between paired variables, which one or ones we should use?

Please also provide reference papers/links when you answer the questions.

Best Answer

Partially answered in comments:

The t-test assumes normality and (generally) equal variance so that under the null distributions are identical, and tests against a location shift. The WMW is usually used to test a location shift with equality of distributions under the null (but without the normality assumption). However its sensitive to more general alternatives of the form $P(X>Y)\not = 1/2$ (e.g. see Conover's Practical Nonparametric Statistics which discusses that kind of alternative). K-S also has a null of identical distributions, but its alternatives are completely general. – Glen_b

Related Solutions

Mann-Whitney U Test – Conducting Mann-Whitney U Test and K-S Test with Unequal Sample Sizes

With such large sample sizes both tests will have high power to detect minor differences. The 2 distributions could be almost identical with a small difference in shape location that is not of practical importance and the tests would reject (because they are different).

If all you really care about is a statistically significant difference then you can be happy with the results of the KS test (and others, even a t-test will be meaningful with non-normal data of those sample sizes due to the Central Limit Theorem).

If you care about practical or meaningful differences then things become subjective, but you can compare using various plots to help you decide if you think there are differences that are enough to care about.

Another possibility is doing a visual test as documented in

 Buja, A., Cook, D. Hofmann, H., Lawrence, M. Lee, E.-K., Swayne,
 D.F and Wickham, H. (2009) Statistical Inference for exploratory
 data analysis and model diagnostics Phil. Trans. R. Soc. A 2009
 367, 4361-4383 doi: 10.1098/rsta.2009.0120

The vis.test function in the TeachingDemos package for R helps implement the test, but it can be done by hand as well.

Basically you create a bunch of graphs and then see if you can tell which is which. For your question one possibility would be to create a histogram of the 122,000 observations from the one month, then take several samples of 122,000 from the 300,000 observations of the other month and create histograms of each of those samples. Then present someone (or several someones) with all the histograms in random order and see if they can pick out the one that represents the second month. If they consistently pick out the correct graph then that says there is something visually different and you can further explore how they differ. If they don't pick out the correct graph then that suggests that while there may be a statistally significant difference, it is not important enough to distinguish them visually.

Solved – Difference Estimate & Confidence Interval for Paired Mann-Whitney Test

One can construct a confidence interval for the median difference using Walsh averages. It does assume that the distribution of the differences is symmetric, but it seems to be OK here. See this document for an explanation of the procedure. It appears that somebody even wrote a Matlab function for this calculation.

EDIT: implementation in Matlab The psignrank function in R calculates the cumulative distribution of the Sign-rank test statistic. If Matlab does not have that function, then you probably can't easily calculate an exact confidence interval. However you can calculate an approximate interval as described here. I think that is the calculation attempted at the end of the Matlab code.

Best Answer

Related Solutions

Mann-Whitney U Test – Conducting Mann-Whitney U Test and K-S Test with Unequal Sample Sizes

Solved – Difference Estimate & Confidence Interval for Paired Mann-Whitney Test

Related Question