Consider the linear regression equation $y_i = \beta_0 + \beta_1x_i + \beta_2x_i^2 + \epsilon_i$ in the quadratic polynomial model with the independent normal noise $\epsilon_i \sim N(0, \sigma^2)$. Assume that the observations of the response values $y_i$ are available at the design points $x_1 = -10, x_2 = -9, … , x_{20} = 9, x_{21} = 10$. Prove that the Least Squares Estimator (LSE) of $\beta_1$ is independent of the LSE of $\beta_2$.
Independence of Least Squares Estimators (LSE)
covarianceleast squareslinear algebrastatisticsvariance
Related Solutions
Your approach is correct.
By differentiating with respect to $\beta_2$, we can see that at the optimal value, we must have
$$\hat{\beta}_2 = y_n -\hat{\beta_1}x_n-\hat{\beta_0}$$
That is the last term of the objective function must vanish.
Hence the problem to solve for $\hat{\beta_0}$ and $\hat{\beta_1}$ is the same as minimizing
$$\sum_{i=1}^{n-1} (y_i-\beta_0-\beta_1 x_i)^2$$
Hence, we know that $\hat{\beta_1}=\hat{\alpha_1}$ and furthermore, $\hat{\beta_0}=\hat{\alpha_0}$.
The answer is $$ \operatorname{Var}(e_i) = \sigma^2\left(1-\frac1n-\frac{(x_i-\bar x)^2}{\text{SSX}}\right), $$ where SSX is shorthand for $\sum(x_i-\bar x)^2$.
The derivation is quite involved. Here is one approach. We require the formula for the variance of the difference of two random variables: $$ \operatorname{Var}(A-B)=\operatorname{Var}(A) + \operatorname{Var}(B) - 2\operatorname{Cov}(A,B).\tag{*} $$
Write the $i$th residual in the form $$ e_i:= y_i-\hat y_i = (\epsilon_i-\bar\epsilon)-(\hat\beta_1-\beta_1)(x_i-\bar x)\tag1$$ by plugging in the definitions for $y_i$ and $\hat y_i$ into $y_i-\hat y_i$, and then substituting $\hat \beta_0:=\bar y - \hat\beta_1 \bar x$.
Applying (*) to (1), the desired variance is $$\operatorname{Var}(e_i) = \operatorname{Var}(\epsilon_i-\bar \epsilon) + (x_i-\bar x)^2\operatorname{Var}(\hat\beta_1-\beta_1)-2(x_i-\bar x)\operatorname{Cov}(\epsilon_i-\bar\epsilon, \hat\beta_1-\beta_1).\tag2$$
Using (*), calculate $$\operatorname{Var}(\epsilon_i-\bar\epsilon)=\operatorname{Var}(\epsilon_i) + \operatorname{Var}(\bar\epsilon) - 2\operatorname{Cov}(\epsilon_i,\bar\epsilon)=\sigma^2\left(1-\frac1n\right).\tag3$$ The tricky calculation is $\operatorname{Cov}(\epsilon_i,\bar\epsilon)$, which requires you to observe that $\epsilon_i$ is independent of $\epsilon_k$ when $k\ne i$.
The variance of $\hat\beta_1-\beta_1$ is well known to be $$\operatorname{Var}(\hat\beta_1-\beta_1)=\frac{\sigma^2}{\text{SSX}}.\tag4$$
The covariance in (2) reduces to $E(\epsilon_i-\bar\epsilon)(\hat\beta_1-\beta_1)$, since $E(\hat\beta_1)=\beta_1$. Substitute the formula $$\hat\beta_1-\beta_1=\frac{\sum_k(x_k-\bar x)(\epsilon_k-\bar\epsilon)}{\text{SSX}}\tag 5$$ to obtain $$(\epsilon_i-\bar\epsilon)(\hat\beta_1-\beta_1)=\frac{\sum_k(x_k-\bar x)(\epsilon_i-\bar\epsilon)(\epsilon_k-\bar\epsilon)}{\text{SSX}}.\tag6 $$ Break up the sum in (6) into $\sum_{k=i} + \sum_{k\ne i}$ and take expectations. The answer will be $$\operatorname{Cov}(\epsilon_i-\bar\epsilon, \hat\beta_1-\beta_1)=E(\epsilon_i-\bar\epsilon)(\hat\beta_1-\beta_1)=\frac{(x_i-\bar x)\sigma^2}{\text{SSX}}.\tag 7$$
Best Answer
We know $$\widehat{\beta}=\beta+\left(X'X\right)^{-1}X'\epsilon$$
which clearly mans that $\widehat{\beta}$ is normally distributed with mean zero and covariance matrix $$\Sigma=\sigma^2\left(X'X\right)^{-1}.$$ Now note that the (2,3) element of $\Sigma$ is zero, therefore the covariance between $\widehat{\beta}_1$ and $\widehat{\beta}_2$ is zero and since these two quantities are jointly normally distributed, they are by definition independent.