Solved – Why does the Kolmogorov-Smirnov test work

distributionskolmogorov-smirnov testnonparametricstatistical significance

In reading about the 2-sample KS test, I understand exactly what it is doing but I don't understand why it works.

In other words, I can follow all the steps to compute the empirical distribution functions, find the maximum difference between the two to find the D-statistic, calculate the critical values, convert the D-statistic to a p-value etc.

But, I have no idea why any of this actually tells me anything about the two distributions.

Someone could have just as easily told me that I need jump over a donkey and count how fast it runs away and the if the velocity is less than 2 km/hr then I reject the null-hypothesis. Sure I can do what you told me to do, but what does any of that have to do with the null-hypothesis?

Why does the 2-sample KS test work? What does computing the maximum difference between the ECDFs have to do with how different the two distributions are?

Any help is appreciated. I am not a statistician, so assume that I'm an idiot if possible.

Best Answer

Basically, the test is consistent as a direct result of the Glivenko Cantelli theorem, one of the most important results of empirical processes and maybe statistics.

GC tells us that the Kolmogorov Smirnov test statistic goes to 0 as $n \rightarrow \infty$ under the null hypothesis. It may seem intuitive until you grapple with real analysis and limit theorems. This is a revelation because the process can be thought of as an uncountably infinite number of random processes, so the laws or probability would lead one to believe that there is always one point which could exceed any epsilon boundary but no, the supremum will converge in the long run.

How long? Mmyyeeaa I don't know. The power of the test is kind of dubious. I'd never use it in reality.

http://www.math.utah.edu/~davar/ps-pdf-files/Kolmogorov-Smirnov.pdf

Related Solutions

Kolmogorov-Smirnov Test – How to Perform a Kolmogorov-Smirnov Two-Sample Test

I am assuming you are asking because the Suanshu help page reports in reference to the K-S distribution, "This is not done yet." Luckily, it is very easy to do in R. If x and y are your two samples, ks.test(x,y) returns the test statistic and pvalue. For example,

> x <- rnorm(50)
> y <- runif(30)
> ks.test(x, y)    
        Two-sample Kolmogorov-Smirnov test    
data:  x and y 
D = 0.5, p-value = 9.065e-05
alternative hypothesis: two-sided

By default, it will compute exact or asymptotic p-values based on the product of the sample sizes (exact p-values for n.x*n.y < 10000 in the two-sample case), or you can specify this option with a third argument, exact=F or exact=T. Exact p-values are calculated using the methods of Marsaglia, et al. (2003), which the Suanshu documentation also cites. Some large sample approximations are given here, although I don't have a proper citation. Lastly, if you don't want to install R, there are web calculators for the two-sample K-S test, although I don't know if they use the same algorithm as R because the one I found only reported three decimal points for the p-value.

Solved – Understanding Kolmogorov-Smirnov test in R

The KS test is premised on testing the "sameness" of two independent samples from a continuous distribution (as the help page states). If that is the case then the probability of ties should be astonishingly small (also stated). The test statistic is the maximum distance between the ECDF's of the two samples. The p-value is the probability of seeing a test statistic as high or higher than the one observed if the two samples were drawn from the same distribution. (It is not the "probability that var1 = var2". And furthermore, 1-p_value is NOT the that probability either.) High p-values say you cannot claim statistical support for a difference, but low p-values are not evidence of sameness. Low p-values can occur with low sample sizes (as your example provides) or the presence of interesting but small differences, e.g. superimposed oscillatory disturbances. If you are working with situations with large numbers of ties it suggests you may need to use a test that more closely fits your data situation.

My explanation of why ties were a violation of assumptions was not a claim that ties invalidated the results. The statistical properties of the KS test in practice are relatively resistant or robust to failure of that assumption. The main problem with the KS test as I see is that it is excessively general and as a consequence is under-powered to identify meaningful differences of an interesting nature. The KS test is a very general test and has rather low power for more specific hypotheses.

On the other hand, I also see the KS-test (or the "even more powerful" Anderson Darling or Lillefors(sp?) test) used to test "normality" in situations where such a test is completely unwarranted, such as test for the normality of variables being used as predictors in a regression model before the fit. One might legitimately want to be testing the normality of the residuals since that is what is assumed in the modeling theory. Even then modest departures from normality of the residuals do not generally challenge the validity of the results. Persons would be better of using robust methods to check for important impact of "non-normality" on conclusions about statistical significance.

Perhaps you should consult with a local statistician? It might assist you in defining the statistical question a bit more precisely and therefore have a better chance of identifying a difference if one actually exists. That would be avoidance of a "type II error": failing to support a conclusion of difference when such a difference is present.

Best Answer

Related Solutions

Kolmogorov-Smirnov Test – How to Perform a Kolmogorov-Smirnov Two-Sample Test

Solved – Understanding Kolmogorov-Smirnov test in R

Related Question