Suppose we have a model
$$y_i = g(x_i) + \epsilon_i, \quad i = 1, \ldots, n$$
where $y_i \in \mathbb{R}$ is a response variable, $x_i \in \mathcal{X} \subseteq \mathbb{R}^d$, and $\epsilon_i$ is the noise where $$E[\epsilon_i|x_i] = 0.$$
We approximate the function $g$ as follows:
$$g(x_i) = x_i^T\beta + r_i$$
where $\beta \in \mathbb{R}^d$ is a vector of regression coefficients, and $r_i$ is the approximation error. We let $$A := E[x_ix_i^T]$$and $$\hat{A} := \frac{1}{n}\sum_{i=1}^n x_ix_i^T.$$
The least squares parameter $\beta$ is defined by
$$\beta := argmin_{b \in \mathbb{R}^d} E[(y_i – x_i^Tb)^2]$$
and the least squares estimator of $\beta$ is
$$\hat{\beta}:= argmin_{b \in \mathbb{R}^d} \frac{1}{n}\sum_{i=1}^n (y_i – x_i^Tb)^2 = \hat{A}^{-1}\frac{1}{n}\sum_{i=1}^n x_iy_i$$
Let $\alpha \in \mathbb{R}^d$ denote a vector such that $||\alpha|| = 1$, where $||\cdot||$ denotes the Euclidean norm.
I want to show that $$\sqrt{n}\alpha^T(\hat{\beta} – \beta) = \alpha^T \hat{A}^{-1}G_n[x_i(\epsilon_i + r_i)]$$
where $$G_n(f(w_i)) = \frac{1}{\sqrt{n}}\sum_{i=1}^n (f(w_i)) – E[f(w_i)]).$$
My attempt:
\begin{align*}
\sqrt{n}\alpha^T(\hat{\beta} – \beta)&= \sqrt{n}\alpha^T\left(\hat{A}^{-1}\frac{1}{n}\sum_{i=1}^n x_iy_i – \beta\right)\\
&= \sqrt{n}\alpha^T\left(\hat{A}^{-1}\frac{1}{n}\sum_{i=1}^n x_i(x_i^T\beta + r_i + \epsilon_i) – \beta\right)\\
&= \sqrt{n}\alpha^T\left(\hat{A}^{-1}A\beta + \hat{A}^{-1}\frac{1}{n}\sum_{i=1}^n x_i(r_i + \epsilon) – \beta\right)\\
&= \sqrt{n}\alpha^T\left(\hat{A}^{-1}\frac{1}{n}\sum_{i=1}^n x_i(r_i + \epsilon_i)\right)\\
&= \frac{1}{\sqrt{n}}\alpha^T\left(\hat{A}^{-1}\sum_{i=1}^n x_i(r_i + \epsilon_i)\right)
\end{align*}
I'm stuck here and not sure how to show the rest because I'm stuck on the algebra. Any help would be appreciated.
Best Answer
Define $u_i:=\epsilon_i+r_i=y_i-x_i^{\top}\beta$. Then $$ \sqrt{n}(\hat{\beta}-\beta)=\left(\frac{1}{n}\sum_{i=1}^n x_ix_i^{\top}\right)^{-1}\frac{1}{\sqrt{n}}\sum_{i=1}^n x_iu_i, $$ where $$ \mathsf{E}x_iu_i=\mathsf{E}[x_ig(x_i)+x_i\epsilon_i-x_ix_i^{\top}\beta]=0 $$ because $\beta=(\mathsf{E}x_ix^{\top})^{-1}\mathsf{E}x_ig(X_i)$.