Solved – Null hypothesis rejection with Kolmogorov-Smirnov

distributionshypothesis testingkolmogorov-smirnov test

I want to compare two samples with a Kolmogorov-Smirnov test. Wikipedia states the null hypothesis is rejected at $\alpha$ if:

$\sqrt( \frac{nn'}{n+n'}) D_{nn'} > K_\alpha$

where $n$ and $n'$ are the sizes of samples, D the KS-statistic and $K_\alpha$ the critical value (probably everyone here already knows). I wonder about the sample sizes: according to this formula every null hypothesis is rejected, if the samples are just large enough.

Could anybody enlighten me, what I am misunderstanding?

Best Answer

It is correct that every result will result in rejecting the null if the sample is large enough. This is true, not just for test using KS, but for any significance test. On one level, this is very reasonable: As sample size increases, our estimates get more precise. At some point, they will get precise enough that any difference $d$ will be large enough to be different from $0$ at a level $\alpha$ .

However, it is also one of the fundamental problems with significance testing: The p-value does not (in the vast majority of cases) tell you what you want to know. That is, the question you are answering is this one:

If, in the population from which this sample was drawn there really was no effect at all, how likely is it that, in a sample of the size we have, the test statistic would be as large as the one we got, or larger?

The simple alternative is to pay more attention to effect sizes and confidence intervals. The complex alternative is to become a Bayesian. :-).

Related Solutions

Solved – Kolmogorov-Smirnov two-sample test

I am assuming you are asking because the Suanshu help page reports in reference to the K-S distribution, "This is not done yet." Luckily, it is very easy to do in R. If x and y are your two samples, ks.test(x,y) returns the test statistic and pvalue. For example,

> x <- rnorm(50)
> y <- runif(30)
> ks.test(x, y)    
        Two-sample Kolmogorov-Smirnov test    
data:  x and y 
D = 0.5, p-value = 9.065e-05
alternative hypothesis: two-sided

By default, it will compute exact or asymptotic p-values based on the product of the sample sizes (exact p-values for n.x*n.y < 10000 in the two-sample case), or you can specify this option with a third argument, exact=F or exact=T. Exact p-values are calculated using the methods of Marsaglia, et al. (2003), which the Suanshu documentation also cites. Some large sample approximations are given here, although I don't have a proper citation. Lastly, if you don't want to install R, there are web calculators for the two-sample K-S test, although I don't know if they use the same algorithm as R because the one I found only reported three decimal points for the p-value.

Solved – Using Kolmogorov–Smirnov test

1) The null hypothesis is that the data is distributed according to the theoretical distribution.

2) Let $N$ be your sample size, $D$ be the observed value of the Kolmogorov-Smirnov test statistic, and define $\lambda = D(0.12 + \sqrt{N} + 0.11 / \sqrt{N})$. Then the p-value for the test statistic is approximately:

$Q = 2 \sum_{j=1}^{\infty}(-1)^{j-1}\exp\{-2j^2\lambda^2\}$

Obviously you can't calculate the infinite sum, but if you sum over 100 values or so this will get you very, very, very close. This approximation is quite good even for small values of $N$, as low as 5 if I recall correctly, and gets better as $N$ increases. Note, however, that @whuber in comments proposes a better approach.

This is a perfectly reasonable alternative to the Shapiro-Wilk test I suggested in answer to your other question, by the way. Shapiro-Wilk is more powerful, but if your sample size is in the high hundreds, the Kolmogorov-Smirnov test will have quite a bit of power too.

Best Answer

Related Solutions

Solved – Kolmogorov-Smirnov two-sample test

Solved – Using Kolmogorov–Smirnov test

Related Question