Solved – Prove consistency

consistencyleast squaresridge regressionself-study

Consider the following estimator
$$\hat\beta = \left(\sum_{i=1}^{N}x_ix_i' + \lambda I_k\right)^{-1}\left(\sum_{i=1}^Nx_iy_i\right)$$
where $x_i$ is a column vector $k\times1$ from $X$ and $\lambda > 0$ is a scalar and $\mathbb{E}(x_ie_i) = 0$ .

  1. Define bias and show that $\hat\beta$ is biased
  2. Define consistency and show that $\hat \beta$ is consistent
  3. Define conditional variance of $\hat\beta$.

For number 1 I have
\begin{equation}
\begin{aligned}
\hat\beta &= \left(\sum_{i=1}^{N}x_ix_i' + \lambda I_k\right)^{-1}\left(\sum_{i=1}^Nx_iy_i\right) \\
& = (X'X + \lambda I_k)^{-1}X'y \\
& = (X'X + \lambda I_k)^{-1}X'(X\beta + e)\\
& = (X'X + \lambda I_k)^{-1}X'X\beta
\end{aligned}
\end{equation}

and hence it is unbiased only if $\lambda = 0$.

I am stuck on number 2. For $\hat\beta$ to be consistent I need
$$(X'X + \lambda I_k)^{-1}X'X \xrightarrow{p} I$$
but how it could be the case for $\lambda > 0$?

Best Answer

Note that $$ \text{plim} \Big[(X'X + \lambda I_k)^{-1} X'X\Big] =\text{plim}(n^{-1}X'X + n^{-1}\lambda I_k)^{-1}\text{plim}(n^{-1}X'X)$$

The second plim converges by asumption. For the first we have $$\text{plim}(n^{-1}X'X + n^{-1}\lambda I_k)^{-1}=\left(\text{plim}n^{-1}X'X + \text{plim}n^{-1}\lambda I_k\right)^{-1} $$

and that

$$\text{plim}n^{-1}\lambda I_k = \text{lim}n^{-1}\lambda I_k = 0$$

leading to the desired consistency result. Intuitively the purpose of adding a term like $\lambda I_k$ is to handle a "bad sample", i.e. it is a finite-sample "tactic" to get results, but whose effect is eliminated asymptotically.