This question has come up in another thread that I started so I thought I would get more people's opinions on it. My question is
Is the residual, e, an estimator of the error, $\epsilon$?
The reason that I ask is the following. In OLS, the variance of the residuals, $\frac{\text{RSS}}{(n – K )}$, is known as the variance of the regression (where RSS is the residual sum of squares). Similarly the square root of this variance, $\sqrt\frac{\text{RSS}}{(n – K )}$, is the standard error of the regression. Given the fact that the square root of the variance, $\frac{\text{RSS}}{(n – K )}$,is a standard error, it must mean that this variance is the variance of an estimator. We already know that it is the variance of the residuals, therefore, the residual is an estimator?? (I assume of $\epsilon$)
Thoughts??
Best Answer
Certainly the residuals are some sort of estimators of $\epsilon$ (to be clear, the definition of the residual is the estimator, the observed residual is an estimate). If the model is correct, then they may sometimes be a fairly good estimate.
Indeed
$e = y - \hat y = X\beta + \epsilon - X(X'X)^{-1} X'(X\beta + \epsilon) = (I - H)\epsilon$,
where $H = X(X'X)^{-1} X'$ is the hat-matrix (because it 'puts the hat on' $y$) -- also sometimes called the projection matrix.
http://en.wikipedia.org/wiki/Hat_matrix
That is, the $e$'s are each a linear combination of the $\epsilon$'s; if $1-h_{ii}$ is reasonably big relative to $\sum_{j\neq i}h_{ij}$ (if $H$ is 'small' relative to I), then most of the weight is on the $i^\textrm{th}$ error (this is frequently not the case, though).
Note that $e_i/\sqrt{1-h_{ii}}$ will have the same expectation and variance as $\epsilon_i$ and if the elements of $H$ are small, in the manner just described, will be highly correlated with it -- in fact, if I have done my algebra right, the correlation between $e_i$ and $\epsilon_i$ is actually: $\text{corr}(e_i,\epsilon_i) = \sqrt{1-h_{ii}}$.