I'm trying to understand the expression $Cov(\hat y,\hat \epsilon)$ in regards to the usual linear regression model/assumptions
$y = \beta_0 + \beta_1 x_1 + \beta_2 x_2 + … \beta_n x_n +\epsilon$. Also, $x_i$ are not considered to be random variables. Rather, they are considered to be fixed measurements with negligible error.
Each $\hat y_i$ is the sum of the $x_i$s weighted by the $\hat \beta_i$s (with $x_0=1)$. And each $\hat \epsilon_i$ is the difference ($y_i – \hat y_i$).
Are $\hat y,\hat \epsilon$, and for that matter, $\hat \beta,$ random vectors?
If u,v are random vectors, then $Cov(u,v)$ is the matrix of elements $Cov(u_i,v_j)$
If $u,v$ are not random vectors, then $Cov(u,v)$ is the scalar $\Sigma u_i v_i$.
Best Answer
Your last assertion is not true. If two vectors are not random, then their covariance is zero.
You are right nevertheless that the fitted values, the residuals and the betas are random vectors. The reason for this is that they are all linear combinations of the random $\mathbf{y}$. To see this we are going to need to define the projection matrix and its orthogonal complement. The projection matrix is defined as
$$\mathbf{H}= \mathbf{X} \left(\mathbf{X}^{\prime} \mathbf{X} \right)^{-1} \mathbf{X} ^{\prime}$$
and its orthogonal complement, also referred to as the annihilator, is
$$\mathbf{M}= \mathbf{I}_n- \mathbf{H}$$
Using these matrices, we may represent the fitted values and the residuals as
$$\widehat{\mathbf{y}}= \mathbf{H} \mathbf{y}$$
$$\mathbf{e} = \mathbf{M} \mathbf{y}$$
respectively, while the beta vector is given by
$$\mathbf{b} =\left(\mathbf{X}^{\prime} \mathbf{X} \right)^{-1} \mathbf{X} ^{\prime} \mathbf{y}$$
and all are linear combinations of the random response vector, as claimed. It can also be shown that the covariance between the residuals and the betas and consequently, the fitted values, is zero.
Is that what you were looking for?