KS Test – Significance Level to Accept Null Hypothesis

distributionshypothesis testingkolmogorov-smirnov test

Apologies in advance if I am thinking about this in the totally wrong way. Essentially, I have a theoretical distribution that I want to test, using the KS test, if my numerical data follows the same distribution. I know that it should follow the same distribution, and get an accordingly very low D statistic. At p=0.05 for example, I clearly cannot reject the null hypothesis, however, I am confused whether this strongly supports the fact that my null hypothesis is correct. Would I need to use a high p-value instead to support the null hypothesis that they are the same distribution?

Again, I am quite confused about this so I apologise if it does not make sense.

Thanks

Best Answer

Numerous posts on site address this issue, in various guises.

however, I am confused whether this strongly supports the fact that my null hypothesis is correct.

It can't! Let us assume the null is true for a moment. Nevertheless there's essentially always going to be population distributions that are closer to the data than the hypothesized (i.e. actual) distribution is.

This is no different from the problem of demonstrating that the population mean is some hypothesised value.

Indeed even if the p value was exactly 1 you still could not assert the null; there's an infinite number of adjacent alternatives with p value as close as you like to 1.

Data cannot demonstrate an equality null; you can sometimes discover a discrepancy large enough to place doubt on it.

You might consider whether an equivalence test might make more sense for your circumstances, but if not there's generally going to be little you can do.