[Math] Linear Regression: Expectation Proof

regressionstatistics

I found the following proof in my notes:

$E(Y_i) = E[\beta_0 + \beta X_i + \varepsilon_i] =\cdots= \beta_0 + \beta X_i$. This does not seem right to me, however. Why would $E(\beta_1 X_i) = \beta_1 X_i$? I wonder if i might have written it down incorrectly, with the actual proof meaning to be for the estimated value Yi hat (I don't know how to code this unfortunately). Does anyone recall this property of linear regression?

Best Answer

If $Y_i=\beta_0+\beta X_i+\epsilon_i$, where $\beta_0$ and $\beta$ are constants and $\epsilon_i$ is an "error" random variable with mean $0$, then $E(Y_i)=\beta_0+\beta E(X_i)$.

Related Solutions

[Math] Variance of Coefficients in a Simple Linear Regression

From the least squares estimation method, we know that $$\hat{\beta}=(X'X)^{-1}X'Y$$ and that $\hat{\beta}$ is an unbiased estimator of $\beta$, i.e $E[\hat{\beta}]=\beta$. Moreover, the linear model $$\begin{equation} Y=X\beta +u \end{equation}$$ has the assumption that $$Y\sim N(\mu=\beta_0+\beta_1x,\sigma)$$ or equivalently that $u \sim N(\mu=0,\sigma)$. Based on the above we can prove all three results (simultaneously) by calculating the variance-covariance matrix of $b$ which is equal to: $$Var(\hat{\beta)}:=\sigma^2(\hat{\beta})=\left( \begin{array}{ccc} Var(\hat{\beta_0}) & Cov(\hat{\beta_0},\hat{\beta_1}) \\ Cov(\hat{\beta_0},\hat{\beta_1}) & Var(\hat{\beta_1}) \end{array} \right)$$ By the properties of variance we have that

$$\begin{align*}Var(\hat{\beta})&=E[\hat{\beta}\phantom{}^2]-E[\hat{\beta}]^2=E[((X'X)^{-1}X'Y)^2]-\beta^2=E[((X'X)^{-1}X'(X\beta +u))^2]-\beta^2=\\&=E[((X'X)^{-1}X'X\beta +(X'X)^{-1}X'u))^2]-\beta^2=E[(\beta+(X'X)^{-1}X'u))^2]-\beta^2=\\&=E[\beta^2]+2(X'X)^{-1}X'E[u]+E[((X'X)^{-1}X'u))^2]-\beta^2=\\&=\beta^2+0+E[((X'X)^{-1}X'u))^2]-\beta^2=E[((X'X)^{-1}X'u))^2]=\\&=\left((X'X)^{-1}X'\right)^2\cdot E[u^2]\end{align*}$$But, since $E[u]=0$ we have that $E[u^2]=Var(u)=\sigma^2$ and by substituting in the above equation we find that $$\begin{align*}Var(\hat{\beta})&=\left((X'X)^{-1}X'\right)^2\cdot E[u^2]=(X'X)^{-1}X'\cdot(X'X)^{-1}X'\cdot\sigma^2=\sigma^2(X'X)^{-1}\cdot I=\\&=\sigma^2(X'X)^{-1}.\end{align*}$$ Now, since $$(X'X)^{-1}=\left( \begin{array}{ccc} \frac{\sum x_i^2}{n\sum (x_1-\bar{x})^2} & \frac{-\sum x_i}{n\sum (x_1-\bar{x})^2} \\ \frac{-\sum x_i}{n\sum (x_1-\bar{x})^2} & \frac{1}{\sum (x_1-\bar{x})^2} \end{array} \right)$$ (which is also known or can be easily derived algebraically) you have the result that: $$\begin{align*} Var(\hat{\beta})&=\left( \begin{array}{ccc} Var(\hat{\beta_0}) & Cov(\hat{\beta_0},\hat{\beta_1}) \\ Cov(\hat{\beta_0},\hat{\beta_1} & Var(\hat{\beta_1}) \end{array} \right)=\sigma^2\left(X'X\right)^{-1}=\\&\phantom{kl}\\&=\left( \begin{array}{ccc} \frac{\sigma^2 \sum x_i^2}{n\sum (x_1-\bar{x})^2} & \frac{-\sigma^2 \sum x_i}{n\sum (x_1-\bar{x})^2} \\ \frac{-\sigma^2 \sum x_i}{n\sum (x_1-\bar{x})^2} & \frac{\sigma^2}{\sum (x_1-\bar{x})^2} \end{array} \right) \end{align*}$$

[Math] How to find the closed form formula for $\hat{\beta}$ while using ordinary least squares estimation

I'm going to show this using partial differentiation.

Consider the assumed linear model $$y_i = \mathbf{x}_i^{T}\boldsymbol\beta + \epsilon_i$$ where $y_i, \epsilon_i \in \mathbb{R}$ and $\mathbf{x}_i=\begin{bmatrix} x_{i0} \\ x_{i1} \\ \vdots \\ x_{ip} \end{bmatrix}, \boldsymbol\beta = \begin{bmatrix} \beta_0 \\ \beta_1 \\ \vdots \\ \beta_p \end{bmatrix} \in \mathbb{R}^{p+1}$ for $i = 1, \dots, n$, with $x_{i0} = 1$.

Our aim is to solve for $\hat{\boldsymbol\beta}$ by minimizing the residual sum of squares, or minimizing $$\text{RSS}(\boldsymbol\beta) = \sum_{i=1}^{n}(y_i-\mathbf{x}_i^{T}\boldsymbol\beta)^2\text{.}$$ To compute this sum, consider the vector of residuals $$\mathbf{e}=\begin{bmatrix} y_1 - \mathbf{x}_1^{T}\boldsymbol\beta \\ y_2 - \mathbf{x}_2^{T}\boldsymbol\beta \\ \vdots \\ y_n - \mathbf{x}_n^{T}\boldsymbol\beta \end{bmatrix}$$ Then $\text{RSS}(\boldsymbol\beta) = \mathbf{e}^{T}\mathbf{e}$. Our next step is to find the "partial derivatives" of $\text{RSS}(\boldsymbol\beta)$.

To do this, note that for $k = 1, \dots, p$, $$\dfrac{\partial \text{RSS}}{\partial \beta_k}=\dfrac{\partial}{\partial\beta_k}\left\{\sum_{i=1}^{n}\left[y_i- \sum_{j=0}^{p}\beta_jx_{ij}\right]^2 \right\}=-2\sum_{i=1}^{n}x_{ik}\left(y_i - \sum_{j=0}^{p}\beta_jx_{ij}\right)\text{.}$$ "Stacking" these, we obtain $$\begin{align} \dfrac{\partial \text{RSS}}{\partial \boldsymbol\beta}&=\begin{bmatrix} \dfrac{\partial \text{RSS}}{\partial \beta_0} \\ \dfrac{\partial \text{RSS}}{\partial \beta_1} \\ \vdots \\ \dfrac{\partial \text{RSS}}{\partial \beta_p} \end{bmatrix} \\ &= \begin{bmatrix} -2\sum_{i=1}^{n}x_{i0}\left(y_i - \sum_{j=0}^{p}\beta_jx_{ij}\right) \\ -2\sum_{i=1}^{n}x_{i1}\left(y_i - \sum_{j=0}^{p}\beta_jx_{ij}\right) \\ \vdots \\ -2\sum_{i=1}^{n}x_{ip}\left(y_i - \sum_{j=0}^{p}\beta_jx_{ij}\right) \end{bmatrix} \\ &= -2\begin{bmatrix} \sum_{i=0}^{n}x_{i0}(\mathbf{y}-\mathbf{x}_1^{T}\boldsymbol\beta)\\ \sum_{i=0}^{n}x_{i1}(\mathbf{y}-\mathbf{x}_1^{T}\boldsymbol\beta) \\ \vdots \\ \sum_{i=0}^{n}x_{ip}(\mathbf{y}-\mathbf{x}_1^{T}\boldsymbol\beta) \end{bmatrix} \\ &= -2\left(\begin{bmatrix} \sum_{i=0}^{n}x_{i0}\mathbf{y}\\ \sum_{i=0}^{n}x_{i1}\mathbf{y} \\ \vdots \\ \sum_{i=0}^{n}x_{ip}\mathbf{y} \end{bmatrix} - \begin{bmatrix} \sum_{i=0}^{n}x_{i0}\mathbf{x}_1^{T}\boldsymbol\beta)\\ \sum_{i=0}^{n}x_{i1}\mathbf{x}_1^{T}\boldsymbol\beta) \\ \vdots \\ \sum_{i=0}^{n}x_{ip}\mathbf{x}_1^{T}\boldsymbol\beta) \end{bmatrix}\right)\\ &= -2(\mathbf{X}^{T}\mathbf{y}-\mathbf{X}^{T}\mathbf{X}\boldsymbol\beta)\text{.} \end{align}$$ where $$\mathbf{X} = \begin{bmatrix} \mathbf{x}_1^{T} \\ \mathbf{x}_2^{T} \\ \vdots \\ \mathbf{x}_n^{T} \end{bmatrix}\text{.}$$ Setting $\dfrac{\partial \text{RSS}}{\partial \boldsymbol\beta} = \mathbf{0}$, we obtain $$\mathbf{X}^{T}\mathbf{X}\boldsymbol\beta = \mathbf{X}^{T}\mathbf{y}$$ and assuming $\mathbf{X}^{T}\mathbf{X}$ is invertible, $$\hat{\boldsymbol\beta} = (\mathbf{X}^{T}\mathbf{X})^{-1}\mathbf{X}^{T}\mathbf{y}$$

Best Answer

Related Solutions

[Math] Variance of Coefficients in a Simple Linear Regression

[Math] How to find the closed form formula for $\hat{\beta}$ while using ordinary least squares estimation

Related Question