Solved – Fitting data sample to a distribution

data visualizationdatasetdistributions

I'm trying to fit a data sample to a distribution. So far I have created a histogram and fitted the data with a lognormal distribution in R and made a Q-Q plot in excel (of log(benefits paid) against theoretical normal quantiles).

Here are my histogram and Q-Q plot:
enter image description here

However, I don't think this distribution is fitting the data closely enough.

I was hoping for an opinion on whether there are any distributions which should fit the data better. Or is this as closer fit as I'm going to be able to get?

Best Answer

The bins in your histogram are a bit too wide (the default has too few bins), making it hard to discern the shape clearly.

The QQ-plot suggests that a finite mixture of perhaps two or three components, possibly of something right skew, perhaps like gamma distributions.

Related Solutions

Distributions – Goodness of Fit for Long-Tailed Distributed Data Using Python

Kolmogorov_Smirnov test is an option but it is more sensitive to deviations in the center than in the tails. Anderson Darling test statistic puts more weight in the tails than the KS-test. There are also goodness-of-fit tests in the Von-Mises group with different weighting schemes.

RMSE will be an approximation to the integrated means squared error, IMSE, which is also used in kernel density estimation as a distance measure.

The test statistics of the goodness of fit tests can be used as a distance measure to rank distributions, however as hypothesis tests they have a non-standard distribution that depends in many cases on the parameters and cannot be tabulated. P-values for standard KS or AD tests will not be correct if some parameters are estimated.

Similar to gung's comment: The choice of weights for different parts of the distribution will depend on the intended use.

An alternative is to estimate the tail separately from the main part of the distribution which I have seen in related cases when the emphasis is on modeling the tail. For example, the tails will asymptotically follow a few possible patterns and a common approach is to estimate Pareto tails for values beyond a threshold. (E.g. Matlab has a distribution that fits a kernel density to the center part and a Pareto distribution to the tails.)

Solved – Formal definition of the qqline used in a Q-Q plot

Sort of "both" - the line depends both on the observed quantiles (which define the y-axis of the QQ plot) and the expected/theoretical/reference quantiles (which the define the x-axis). The documentation (which you quote) should always be taken as the canonical reference:

‘qqline’ adds a line to a “theoretical”, by default normal, quantile-quantile plot which passes through the ‘probs’ quantiles, by default the first and third quartiles.

If in doubt, USTL ("Use the Source, Luke") , which can be found here: here's a slightly abridged and commented version

 ## quantiles (.25 and 0.75 by default) of data
 y <- quantile(y, probs, names=FALSE, type=qtype, na.rm = TRUE)
 ## quantiles of reference/theoretical distribution
 x <- distribution(probs)
 ## ...
 slope <- diff(y)/diff(x)  ## observed slope between quantiles
 int <- y[1L]-slope*x[1L]  ## intercept
 abline(int, slope, ...)   ## draw the line

For what it's worth, I believe that this approach (line connecting central quantiles) is used because it fulfills the following criteria for exploratory/diagnostic approaches:

quick (e.g. no need to run a linear regression, just find the quantiles and draw a straight line)
robust (it only depends on the behavior of the central part of the distribution, won't be thrown off by weird tails)

Best Answer

Related Solutions

Distributions – Goodness of Fit for Long-Tailed Distributed Data Using Python

Solved – Formal definition of the qqline used in a Q-Q plot

Related Question