I have two samples where I want to tell if their distributions are significantly different. I conducted a Kolmogorov-Smirnov test which rejected the null hypothesis (D=0.0983, p=2.14e-11) leading me to believe that the samples do indeed come from different distributions. The problem is, how can I determine what is significantly different about these samples? Are the distributions different in mean? skew? something else? Examining the histograms with each other do not give any immediate indication of difference. Sure, the red sample has a bit lower values than the blue sample, but is that what caused the KS test to be significant? How can I know what caused the KS test null hypothesis to fail?
Sample size of red = 1306
Sample size of blue = 646,513
Best Answer
The K-S is an omnibus test. It doesn't identify the form of difference.
You can identify the place or places where the difference in ECDF is largest but that doesn't always tell you much.
You can spot some kinds of change in the ECDF. Here's some examples:
If you look at the ECDFs in your post, you can see the blue cdf is lower than the red across pretty much the entire range (which implies the blue values are typically larger*), though the shape is broadly similar.
* You can also see that in the histogram where the pink is higher in the leftmost bar, but past 300 the blue bars are typically a little taller.
So anyway, that's how I'd describe the main difference in distributions (somewhat similar in general shape, but the values in the blue category tend to be a bit larger on average). I wouldn't characterize it as either a location shift or a scale-shift.
One thing that worries me is that the distribution looks quite discrete. If so, the p-value from the Kolmogorov-Smirnov test isn't particularly meaningful.
What's your sample size?