Solved – the test statistic in Kolmogorov–Smirnov test

kolmogorov-smirnov teststatistical significance

I am reading the Wikipedia page on the Kolmogorov-Smirnov test, specifically the section titled Kolmogorov_distribution.

$x$ is $D_\text{max}$ in the CDF $Pr(k\leq x)$?

My question is the number of points matters too,right?

When $n$ is large, $x$ approximates $\sqrt n D_\text{max}$ ?

Here is a snapshot of the section from Wikipedia:

enter image description here

Q1
How do we obtain the distribution of $D_n$ when $n$ is fixed?

Q2
If I get the value of $D_\text{max}$ and the sample size is $n$, I have to calculate $Pr(K<=x)$, right? Is $D_\text{max}$ the $x$ in the formula on the Wikipedia page?

By formula 14.3.9 of Numerical Recipes, we should calculate a value got from the expression in the brackets – should that be the $x$? The value approximates $\sqrt n D_\text{max}$ when $n$ is large.

I am totally confused here.

Q3
We make tests and get a distribution,right? Could you please explain your figure in a "test" way? Maybe that is easier to follow.

I calculate several values(significance)and compare them with the tableks table

The $x$ in $Pr(K<=x)$ is not $D_{max}$ .If sample size is large,x is $(\sqrt{n}+0.12+0.11/\sqrt{n})*D_{max}$ .That means there is no error in 14.4.9 of NR . The significance we want to get is determined by sample size and $D_{max}$ .

Please let me explain my questions in a "test" way.

Suppose sample size is 30, we obtain a dataset of 30 points and we can get a $D_{max}$ from the comparison between the empirical distribution of the sample and the reference probability distribution. We do it 1000 times and we get 1000 $D_{max}$ . There is a distribution for $D_{max}$ ,right?

From your figure , there should be 1000*0.01 points with a $D_{max}$ larger than 0.29 and 1000*0.05 points with a $D_{max}$ larger than 0.24 ,more or less .

Like you said, I am very confused. Please let me make sure the several statements below are right or wrong first.

When sample size is large,my calculation method is right?
we can take $(\sqrt{n}+0.12+0.11/\sqrt{n})*D_{max}$ as x to input to $Pr(K<=x)$ and in this way we can get the significance when sample size is large,right?

From your figure , there should be 1000*0.01 points with a $D_{max}$ larger than 0.29 and 1000*0.05 points with a $D_{max}$ larger than 0.24 . Is that your figure tells us ?

Best Answer

Note that the Kolmogorov-Smirnov test statistic is very clearly defined in the immediately previous section:

$$D_n=\sup_x|F_n(x)−F(x)|\,.$$

The reason they discuss $\sqrt{n}D_n$ in the next section is that the standard deviation of the distribution of $D_n$ goes down as $1/\sqrt n$, while $\sqrt{n}D_n$ converges in distribution as $n\to\infty$.

Yes, the number of points, $n$, matters to the distribution; for small $n$, tables are given for each sample size, and for large $n$ the asymptotic distribution is given for $\sqrt{n}D_n$ $-$ the very same distribution discussed in the section you quote.

Without some result on asymptotic convergence in distribution, you'd have the problem that you'd have to keep producing tables at larger and larger $n$, but since the distribution of $\sqrt{n}D_n$ pretty rapidly 'stabilizes', only a table with small values of $n$ is required, up to a point where approximating $\sqrt{n}D_n$ by the limiting Kolmogorov distribution is sufficiently good.

Below is a plot of exact 5% and 1% critical values for $D_n$, and the corresponding asymptotic critical values, $K_\alpha/\sqrt n$.

Kolmogorov-Smirnov critical values

Most tables finish giving the exact critical values for $D_n$ and swap to giving the asymptotic values for $\sqrt n D_n$, $K_\alpha$ (as a single table row) somewhere between $n=20$ and $n=40$, from which the critical values of $D_n$ for any $n$ can readily be obtained.

$\text{Responses to followup questions:}$

How do we obtain the distribution of $D_n$ when $n$ is fixed?

There are a variety of methods for obtaining the distribution of the test statistic for small $n$; for example, recursive methods build the distribution at some given sample size in terms of the distribution for smaller sample sizes.

There's discussion of various methods given here, for example.

If I get the value of $D_\text{max}$ and the sample size is $n$, I have to calculate $Pr(K<=x)$, right?

Your test statistic is your observed sample value of the $D_n$ random variable, which will be some value, $d_n$ (what you're calling $D_\text{max}$, but note the usual convention of upper case for random variables and lower case for observed values). You compare it with the null distribution of $D_n$. Since the rejection rule would be "reject if the distance is 'too big'.", if it is to have level $\alpha$, that means rejecting when $d_n$ is bigger than the $1-\alpha$ quantile of the null distribution.

That is, you either take the p-value approach and compute $P(D_n> d_n)=1-P(D_n\leq d_n)$ and reject when that's $\leq\alpha$ or you take the critical value approach and compute a critical value, $d_\alpha$, which cuts off an upper tail area of $\alpha$ on the null distribution of $D_n$, and reject when $d_n \geq d_\alpha$.

By formula 14.3.9 of Numerical Recipes, we should calculate a value got from the expression in the brackets - should that be the x?

14.3.9 looks like it has a typo (one of many in NR). It is trying to give an approximate formula for the p-value of "observed" (that is, my "$d_n$", your $D_\text{max}$), by adjusting the observed value so you can use the asymptotic distribution for even very small $n$ (in my diagram, that corresponds to changing the $y$-value of the observed test statistic via a function of $n$, equivalent to pushing the circles 'up' to lie very close the dotted lines) but then it (apparently by mistake) puts the random variable (rather than the observed value, as it should) into the RHS of the formula. The actual p-value must be a function of the observed statistic.

We make tests and get a distribution, right?

I don't know what you mean to say there.

Could you please explain your figure in a "test" way?

My figure plots the 5% and 1% critical values of the null distribution of $D_n$ for sample sizes 1 to 40 (the circles) and also the value from the asymptotic approximation $K_\alpha/\sqrt n$ (the lines).

It looks to me like you have some basic issues with understanding hypothesis tests that's getting in the way of understanding what is happening here. I suggest you work on understanding the mechanics of hypothesis tests first.

That means there is no error in 14.4.9 of NR .

(Presumably you mean 14.3.9, since that's what I was discussing.)

Yes there is an error. I think you may have misunderstood where the problem is.

The problem isn't with "$(\sqrt{n}+0.12+0.11/\sqrt{n})$". It's with the meaning of the term they multiply it by. They appear to have used the wrong variable from the LHS in the RHS formula, putting the random variable where its observed value should be.

[When the thing you're reading is confused about that, it's not surprising you have a similar confusion.]

Related Solutions

Solved – the definition of one-sample Kolmogorov–Smirnov test

First, on the programing side, passing 'uniform' is essentially passing scipy.stats.uniform.cdf() to kstest. So whatever you have in args= will be passed scipy.stats.uniform.cdf() as parameters, which only takes two parameters, location and scale (see the document for detail). If you have more than two values in args=, the extra will simply ignored:

>>> a=np.random.random(10)
>>> stats.kstest(a, 'uniform', args=(0.5,1,3,4))
(0.303993262358352, 0.25725219759419549)
>>> stats.kstest(a, 'uniform', args=(0.5,1,300, 400)) #see how these two give same result
(0.303993262358352, 0.25725219759419549)

Second, since you already normalized CDF of the photon arrival times, it will make sense to do one-sample KS test against the standard uniform distribution. http://journals.ametsoc.org/doi/abs/10.1175/1520-0450%281975%29014%3C1600%3AANOTPM%3E2.0.CO%3B2 Basically what that paper says is that if If one or more parameters must be estimated from the sample, then $D$ no longer follows a Kolmogrov-Smirnov distribution and if you still the CDF of KS to get $P$ value from $D$, you will get wrong $P$. Also, I don't think it is correct apporach to generate a uniform distributed random variable and apply 2-sample KS test.

Third, the CDF of Kolmogrov-Smirnov distribution is given by: $\operatorname{Pr}(K\leq x)=1-2\sum_{k=1}^\infty (-1)^{k-1} e^{-2k^2 x^2}=\frac{\sqrt{2\pi}}{x}\sum_{k=1}^\infty e^{-(2k-1)^2\pi^2/(8x^2)}$ and this is how you can calculate $P$ from $D$. In scipy it is not provided by pure python code, but by a C extension.

Solved – Bootstrapping and Kolmogorov-Smirnov

I think I understand your problem now. You alluded to your assumption that somehow KS test should show that all bootstrapped samples should be shown to be from the original sample. However, consider this: what does it mean to show that they're from the same distribution?

It usually means that p-value is over some $\alpha$ confidence. If bootstrapping is done properly you'll get p-value sometimes over, sometimes under the $\alpha$. Build the distribution of test statistics you get from running KS test on bootstrapped samples. Observe p-values for various critical values, they should match the theoretical values for which KS test was designed.

Best Answer

Related Solutions

Solved – the definition of one-sample Kolmogorov–Smirnov test

Solved – Bootstrapping and Kolmogorov-Smirnov

Related Question