For some data (where I have the mean and standard deviation) I currently estimate the probability of getting samples greater than some x by using the Q function; i.e., I'm calculating the tail probabilities. But this assumes a normal (Gaussian) distribution of my data, and I may be better off assuming a heavy tailed distribution, like log-normal or Cauchy. How can I calculate the tail probabilities for heavy tailed distributions?
Solved – Tail probability for heavy tailed distributions
cauchy distributiondistributionsfat-tailslognormal distributionprobability
Related Solutions
Wikipedia is often a reasonable start point for basic definitions. In this case there is an entry for heavy-tailed distributions.
The distribution of a random variable $X$ with distribution function $F$ is said to have a heavy right tail if $$\lim_{x\rightarrow\infty} e^{\lambda x}P[X>x]=\infty.$$
for all $\lambda>0$. This can be interpreted as: the tails decay slower than the exponential and this has implications on the existence of moments (see the same wikipedia entry).
Then,
(1). To the exponential or compared to an exponential-type behaviour.
(2). This can be empirically checked using a normal QQ-plot, for example,
x <- rt(1000,3)
qqnorm(x)
qqline(x,col="red")
As you can see, the lack of linear fit is observed in both tails.
(3). An immediate implication of the use of heavy-tailed distribution is that you observe more extreme observations or that the model can capture this sort of behaviour. This is, values far from the shoulders of the distribution. This is reflected in the summary statistics, for example compare the statistics of a normal sample and those of a $t$ sample with $3$ degrees of freedom
summary(rnorm(1000))summary(rt(1000,3))
I applied Tukey-Lambda PPCC to your data to get the following plot:
You see the local max at $\lambda=-0.47$, which implies heavier tails than normal distribution. For normal it's 0.14 and for Cauchy it's -1. So, if you go with this view, then your data's tail is not as "fat" as Cauchy's, but much fatter than normal.
However the global max is at $\lambda=33.17$
I drew the Tukey lambda for the above two maxima here:
So, you have two very different views of the distribution. One is fat tailed, the other one's very convex.
I think that the convex distribution is the better fit. Cauchy and other fat tailed ones are not a good fit here, go for these weird distributions like in my last plot.
Here's how Tukey Lambda distribution looks like for some $\lambda$.
I use this tool to gauge the shape of my data.
Best Answer
The Wikipedia article you linked shows that the Q-function is just 1 minus the cumulative distribution function (CDF) of the normal distribution. Every distribution has its own CDF. So just use the CDF for the distribution in question. In R, for example,
1 - pcauchy(5)tells you the probability of getting 5 or more from a standard Cauchy distribution.