Let us assume that data is generated according to a true model $$y_i = \beta_{true}x_i + \epsilon_i$$
for $i = 1, …, n$
Assume that $x_i$ are fixed, and $\epsilon_i$~ N(0, $\sigma^2$) independently.
Let $$\hat\beta =\frac{\sum^{n}_{i=1}y_ix_i}{\sum^{n}_{i=1}x^2_i + \lambda}$$ be the shrinkage estimator from the ridge regression.
How to calculate expectation and variance of $\hat\beta$, and mean squared error E$[(\hat\beta – \beta_{true})^2]$ ?
I'm stuck on this part for expectation. What to do next?
$$E(\hat\beta)= E(\frac{\beta_{true}\sum^{n}_{i=1}x_i^2 + \sum^{n}_{i=1}x_i\epsilon_i}{\sum^{n}_{i=1}x^2_i + \lambda}) = (E(\frac{\beta_{true}\sum^{n}_{i=1}x_i^2 }{\sum^{n}_{i=1}x^2_i + \lambda})$$
Best Answer
\begin{align*} E(\hat\beta) &= E\left(\frac{\beta_{true}\sum^{n}_{i=1}x_i^2 + \sum^{n}_{i=1}x_i\epsilon_i}{\sum^{n}_{i=1}x^2_i + \lambda}\right) = E \left(\frac{\beta_{true}\sum^{n}_{i=1}x_i^2 }{\sum^{n}_{i=1}x^2_i + \lambda} \right)=\frac{\beta_{true}\sum^{n}_{i=1}x_i^2 }{\sum^{n}_{i=1}x^2_i + \lambda} \\ var(\hat\beta) &= var\left(\frac{\beta_{true}\sum^{n}_{i=1}x_i^2 + \sum^{n}_{i=1}x_i\epsilon_i}{\sum^{n}_{i=1}x^2_i + \lambda}\right) = var \left(\frac{\sum^{n}_{i=1}x_i\varepsilon_i }{\sum^{n}_{i=1}x^2_i + \lambda} \right)=\frac{\sigma^2 \sum^{n}_{i=1}x_i^2 }{(\sum^{n}_{i=1}x^2_i + \lambda)^2} \end{align*} where the second equality follows because everything is a constant except $\varepsilon_i$ and the second equality follows because with $var(\varepsilon_i)=\sigma^2$, $\varepsilon_i$ are independent so we can take out the sum and taking out constant results in squaring them.