I am looking at fitting distributions to data (with a particular focus on the tail) and am leaning towards Anderson-Darling tests rather than Kolmogorov-Smirnov. What do you think are the relative merits of these or other tests for fit (e.g. Cramer-von Mises)?
Solved – What do you think is the best goodness of fit test
fittinghypothesis testing
Related Solutions
I think the question asks for a precise statistical test, not for an histogram comparison. When using the Kolmogorov-Smirnov test with estimated parameters, the distribution of the test statistics under the null depends on the tested distribution, as opposed to the case with no estimated parameter. For instance, using (in R)
x <- rnorm(100)
ks.test(x, "pnorm", mean=mean(x), sd=sd(x))
leads to
One-sample Kolmogorov-Smirnov test
data: x
D = 0.0701, p-value = 0.7096
alternative hypothesis: two-sided
while we get
> ks.test(x, "pnorm")
One-sample Kolmogorov-Smirnov test
data: x
D = 0.1294, p-value = 0.07022
alternative hypothesis: two-sided
for the same sample x. The significance level or the p-value thus have to be determined by Monte Carlo simulation under the null, producing the distribution of the Kolmogorov-Smirnov statistics from samples simulated under the estimated distribution (with a slight approximation in the result given that the observed sample comes from another distribution, even under the null).
What is the correct (or optimal) approach to distribution fitting in terms of using parametric versus non-parametric methods?
There won't be one correct approach, and what might be suitable depends on what you want to "optimize" and what you're trying to achieve with your analysis.
When there is little data, you don't have much ability to estimate distributions.
There is one interesting possibility that sort of sits between the two. It's effectively parametric (at least when you fix the dimension of the parameter vector), but in a sense the approach spans the space between a simple parametric model and a model with arbitrarily many parameters.
That is to take some base distributional model and build an extended family of distributions based on orthogonal polynomials with respect to the base distribution as weight function. This approach has been investigated by Rayner and Best - and a number of other authors - in a number of contexts and for a variety of base distributions. This includes "smooth" goodness of fit tests, but also similar approaches for analysis of count data (which allow decomposing into "linear", "quadratic" etc components that deviate from some null model), and a number of other such ideas.
So for example, one would take a family distributions based around the normal distribution and Hermite polynomials, or uniforms and Legendre polynomials, and so on.
This is especially useful where a particular model is expected to be close to suitable, but that the actual distribution will tend to deviation "smoothly" from the base model.
In the normal and uniform cases the methods are very simple, often more easily interpretable than other flexible methods, and often quite powerful.
Does it makes sense to combine both approaches for validation?
It would often make sense to use a nonparametric approach to check a parametric one.
The other way around may make sense in some particular circumstances.
Best Answer
I have been told many times that the Anderson Darling (AD) test is much better than the Kolmogorov-Smirnov (KS) one because AD does a better job at fitting the tails of the distribution. KS is only good at fitting the mid-range of the distribution; but, is not better than AD even in this regard. I think the main advantage of the KS test is its very intuitive visual interpretation (fitting of the respective cumulative distributions). Because of the KS easy visual and intuitive interpretation it has become dominant in certain specialties such as credit scoring models within the financial service industry. But, more visually intuitive does not mean better.
When using Monte Carlo simulation models that automatically fit a statistical distribution to a data set; their respective software manuals typically recommend leaning more on the AD than the KS test for the reason mentioned above (fits the tails better).