Solved – Two sample one-sided Kuiper Test and KS-statistic

cumulative distribution functionhypothesis testingkolmogorov-smirnov testp-value

With the KS-Test it is possible to conduct a two sample one-sided test between two different random samples $A$ and $B$ to test whether one CDF is larger or smaller than the other, i.e. is $CDF_A$ larger than $CDF_B$. Standard implementations for this exist, for example MATLAB's kstest2.

So with respect to the above test (is $CDF_A$ larger than $CDF_B$) the KS-statistic will be calculated as $max(CDF_A – CDF_B)$, which can then be used to calculate the p-value.

Is it possible to define a two sample one-sided Kuiper Test? Essentially the question that I want to answer is that whether the two CDFs, $CDF_{A}$ and $CDF_{B}$ differ at the tails and that $CDF_{A}$ is overall larger than $CDF_{B}$.

For example in the image below the two CDFs differ at the tails but absolute value of $D^{+}$ is greater than absolute value of $D^{-}$. So in this case can we use the Kuiper Test to determine that one is greater than the other and how would we calculate the KS-statistic for this case?

enter image description here

Best Answer

This is a good question. My answer is it cannot be done properly, and I add

Use the one-sided KS test at your own risks.

For instance, R offers the choice between two.sided, less or greater. I consider this is a mistake because it does not really explain how to interpret the one-sided case. Here is an example that will explain the main issue.

Say that $A$ and $B$ differ only by their variance. For instance assume $A \sim N(0, 1)$ and $B \sim N(0,5)$. The two distributions are different, the null hypothesis is false, so the p-value of the two-sided KS test decreases as the sample size increases. But what happens with the one-sided KS tests? Let us find out with an example.

set.seed(123)
A <- rnorm(100, sd=1)
B <- rnorm(100, sd=5)
ks.test(A, B, alternative="less")
# D^- = 0.44, p-value = 3.909e-09
# Null is rejected. Accept alternative A < B
ks.test(x, y, alternative="greater")
# D^+ = 0.26, p-value = 0.001159
# Null is rejected. Accept alternative A > B

That's right. We have accepted both A < B and A > B. And what takes the cake is that A and B are centered on the same value. Contrary to a popular belief, the one-sided versions of the KS test do not tell whether one distribution is shifted respective to the other. What do they tell exactly? I am not sure.

The best explanation to understand what happens is your picture. You can see that in this case, both $D_+$ and $D_-$ are large, so both alternatives are accepted. It is also important to understand that by symmetry, $D = D_+$ or $D = D_-$ with probability $1/2$ each. Which statistic “won” does not tell you anything interesting.

To be honest, I do not fully understand your proposal for a one-side Kuiper test, but I foresee that it would have the same difficulties. I wrote a more detailed description of this issue on my blog (see (mis)using the KS test for p-hacking) and in the scientific literature (see The signed Kolmogorov-Smirnov test: why it should not be used). The suggestions from the reviewers Garrett Jenkinson and Desmond Campbell were to use the Wilcoxon test and the $t$ test instead.

Related Question