normal distributionqq-plot
I have ran a QQ plot on R on my data using
par(mfrow=c(1,2))
par(pty="s")
qqnorm(TEDS$LST1); qqline(TEDS$LST1)
which gave me this:
The histogram of the data showed a positive skew to left, but I do not know how to interpret the QQ plot. Why are the data points clustering along the line? and does the straight tail at the bottom signify the left skew that I see in the histogram?
This seems to be a qqplot of the data compared with a standard normal distribution, so I would have thought the $x$ values should the typical values of the population quantiles of a standard normal distribution
So with $105$ observations I would have thought the extreme left $x$ value should be not far away from $\Phi^{-1}\left(\dfrac{0.5}{105}\right) \approx -2.59$ and the one next to it near $\Phi^{-1}\left(\dfrac{1.5}{105}\right) \approx -2.19$, with the extreme right values being the corresponding $\Phi^{-1}\left(\dfrac{104.5}{105}\right) \approx +2.59$ and $\Phi^{-1}\left(\dfrac{103.5}{105}\right) \approx +2.19$. Visually, this seems to be close to what you have in the charts
Here is a relevant section from Hoaglin, Mosteller and Tukey (2000): Understanding Robust and Exploratory Data Analysis. Wiley. Chapter 3, "Boxplots and Batch Comparison", written by John D. Emerson and Judith Strenio (from page 62):
[...] Our definition of outliers as data values that are smaller than $F_{L}-\frac{3}{2}d_{F}$ or larger than $F_{U}+\frac{3}{2}d_{F}$ is somewhat arbitrary, but experience with many data sets indicates that this definition serves well in identifying values that may require special attention.[...]
$F_{L}$ and $F_{U}$ denote the first and third quartile, whereas $d_{F}$ is the interquartile range (i.e. $F_{U}-F_{L}$).
They go on and show the application to a Gaussian population (page 63):
Consider the standard Gaussian distribution, with mean $0$ and variance $1$. We look for population values of this distribution that are analogous to the sample values used in the boxplot. For a symmetric distribution, the median equals the mean, so the population median of the standard Gaussian distribution is $0$. The population fourths are $-0.6745$ and $0.6745$, so the population fourth-spread is $1.349$, or about $\frac{4}{3}$. Thus $\frac{3}{2}$ times the fourth-spread is $2.0235$ (about $2$). The population outlier cutoffs are $\pm 2.698$ (about $2\frac{2}{3}$), and they contain $99.3\%$ of the distribution. [...]
So
[they] show that if the cutoffs are applied to a Gaussian distribution, then $0.7\%$ of the population is outside the outlier cutoffs; this figure provides a standard of comparison for judging the placement of the outlier cutoffs [...].
Further, they write
[...] Thus we can judge whether our data seem heavier-tailed than Gaussian by how many points fall beyond the outlier cutoffs. [...]
They provide a table with the expected proportion of values that fall outside the outlier cutoffs (labelled "Total % Out"):
So these cutoffs where never intended to be a strict rule about what data points are outliers or not. As you noted, even a perfect Normal distribution is expected to exhibit "outliers" in a boxplot.
As far as I know, there is no universally accepted definition of outlier. I like the definition by Hawkins (1980):
An outlier is an observation which deviates so much from the other observations as to arouse suspicions that it was generated by a different mechanism.
Ideally, you should only treat data points as outliers once you understand why they don't belong to the rest of the data. A simple rule is not sufficient. A good treatment of outliers can be found in Aggarwal (2013).
Aggarwal CC (2013): Outlier Analysis. Springer.
Hawkins D (1980): Identification of Outliers. Chapman and Hall.
Hoaglin, Mosteller and Tukey (2000): Understanding Robust and Exploratory Data Analysis. Wiley.
Best Answer
This QQ plot has the following salient features:
The stairstep pattern, in which only specific, separated heights ("sample quantiles") are attained, shows the data values are discrete. Almost all are whole numbers from $3$ through $21$. A few half-integers appear. Evidently some form of rounding has occurred.
Because the extreme "theoretical quantiles" are at $\pm 3.2$ (roughly), there must be around $1400$ data shown. This is because the extremes for this much Normally distributed data would have Z-scores about $\pm 3.2$. (This estimate of $1400$ is rough, but it's in the right ballpark.)
There is a large number of values at the minimum of $3$, far more than any other value. This is characteristic of left censoring, whereby any value less than a threshold ($3$) is replaced by an indicator that it is less than that threshold--and, for plotting purposes, all such values are plotted at the threshold. (For more on what censoring does to probability plots, see the analysis at https://stats.stackexchange.com/a/30749.)
Apart from this "spike" at $3$, the rest of the points come fairly close to following the diagonal reference line. This suggests the remaining data are not too far from Normally distributed.
A closer look, though, shows the remaining points are initially slightly lower than the reference line (for values between $5$ and $10$) and then slightly greater (for values between $13$ and $20$) before returning to the line at the end (value $21$). This "curvature" indicates a certain form of non-normality.
This particular kind of curvature is consistent with data that are starting to follow an extreme-value distribution. Specifically, consider the following data-generation mechanism:
Collect $k\ge 1$ independent, identically distributed Normal variates and retain just the largest of them.
Do that $n = 1400$ times.
Left-censor the data at a threshold of $3$.
Record their values to two or three decimal places.
Round the values to the nearest integer--but don't round any value that is exactly a half-integer (that is, ends in $.500$).
If we set $k=50$ or thereabouts and adjust the mean and standard deviation of those underlying Normal variates to be $\mu = -10$ and $\sigma = 7.5$, we can produce random versions of this QQ plot and most of them are practically indistinguishable from it. (This is an extremely rough estimate; $k$ could be anywhere between $8$ and $200$ or so, and different values of $k$ would have to be matched with different values of $\mu$ and $\sigma$.) Here are the first six such versions I produced:
What you do with this interpretation depends on your understanding of the data and what you want to learn from them. I make no claim that the data actually were created in such a way, but only that their distribution is remarkably like this one.
This is
Rcode to reproduce the figure (and generate many more like it if you wish).