Let $Y_1,Y_2,\ldots,Y_n$ and $X_1,X_2,\ldots,X_m$ be random variables with $E(Y_i)=\mu_i$ and $E(X_j)=\xi_j$. Define $$U_1=\sum_{i=1}^n a_i Y_i\quad\text{and}\quad U_2=\sum_{j=1}^m b_j X_j$$ for constants $a_1,a_2,\ldots,a_n$ and $b_1,b_2,\ldots,b_m$. Then the following hold:
$\quad\textbf a\,\,$ $E(U_1)=\sum_{i=1}^n a_i\mu_i.$
$\quad\textbf b\,\,$
$V(U_1)=\sum_{i=1}^n a_i^2 V(Y_i)+2\sum\sum_{1\leqslant i \leqslant j \leqslant n} a_i a_j \operatorname{Cov}(Y_i,Y_j)$, where the double sum is over all pairs $(i,j)$ with $i\lt j$.$\quad\textbf c\,\,$ $\operatorname{Cov}(U_1, U_2)=\sum_{i=1}^n \sum_{j=1}^m a_i b_j \operatorname{Cov}(Y_i, X_j)$.
I am also given a hint to use this which I do not know how I can apply this to this question?
Suppose that $Y_1, Y_2, \ldots, Y_n$ are independent normal random variables with $E(Y_i) = \beta_0 + \beta_1 x_i$ and $V(Y_i) = \sigma^2$, for $i = 1, 2, \ldots, n$. The maximum-likelihood estimators of $\beta_0$ and $\beta_1$ are the same as the least-squares estimators of 
Find $\operatorname{Cov}(\hat{\beta}_0, \hat{\beta}_1)$.
What I got so far is $$\operatorname{Cov}(\hat{\beta_0}, \hat{\beta}_1) = \operatorname{Cov}(\bar{Y}-\hat{\beta}_1\bar{x}, \hat{\beta}_1)$$
How can I move forward with this?
Best Answer
This assumes that the $x_{i}$ are fixed, the model is quite different if they are not. However, this assumption is reasonable given the expected value and variance of the $Y_{i}$. Usually, for any linear relationship, we have the model $$Y_i=\beta_0+\beta_1 x_i+\epsilon_i,$$ where $\beta_{0},\beta_1 \in \mathbb{R}$, the $x_i$ are fixed, and $\mathbb{E}(\epsilon_i)=0, \text{Var}(\epsilon_i)=\sigma^2$ for a constant $\sigma^{2}$. This is very similar to your context, so I will assume this is the correct context. Denote the random variables $Y_i$ with lower case $y_i$ for consistency with typical regression notation. We have $$\text{Cov}(\hat{\beta}_0,\hat{\beta}_1)=\text{Cov}(\bar{y}-\hat{\beta_{1}}\bar{x},\hat{\beta}_1) \\=\text{Cov}(\bar{y},\hat{\beta}_1)-\text{Cov}(\hat{\beta}_1\bar{x},\hat{\beta}_1)\\=\text{Cov}(\bar{y},\hat{\beta}_1)-\bar{x}\text{Cov}(\hat{\beta}_1,\hat{\beta}_1)\\=\text{Cov}(\bar{y},\hat{\beta}_1)-\bar{x}\text{Var}(\hat{\beta}_1),$$ where the third equality comes from the fact that the $x_i$ are fixed and the fourth equality comes from the definitions of variance and covariance. From here, substitute the given expression for $\hat{\beta}_1$ (this expression and the one given for $\hat{\beta}_0$ can be derived from the usual method of maximum likelihood or from the least-squares estimators, they are equivalent), and use your knowledge about covariances and variances to finish the calculation (some information that you need will come from the proposition - "the following" in your box). Important hint: you will need to write $\hat{\beta}_1$ as a linear combination of the $y_{i}$, this is the key to completing the computation. Note that the covariance function and the vector space of random variables form an inner product space, so $\text{Cov}(\cdot,\cdot)$ is linear in the first argument (and second argument as a result of the symmetry clause of inner products) and thus, $$\text{Cov}\left(\sum_{i=1}^n c_i Z_i,X\right)=\sum_{i=1}^n c_i \text{Cov}(Z_i,X)$$ (this will be very useful as well) for constants $c_1,c_2,\ldots,c_n$ and any random variables $Z_1,Z_2,\ldots,Z_n,X$. Also, here is yet another two useful identities/derivations:$$\sum_{i} x_i(x_i - \bar{x})=\sum_i x_i(x_i - \bar{x}) - \bar{x} \sum_i(x_i - \bar{x})\\=\sum_i(x_i - \bar{x})(x_i-\bar{x})=\sum_i (x_i - \bar{x})^2= S_{xx},$$ and similarly, $$\sum_i x_i(y_i - \bar{y})=\sum_i x_i(y_i - \bar{y}) - \bar{x} \sum_i(y_i - \bar{y})\\=\sum_{i}(x_i - \bar{x})(y_i-\bar{y})= S_{xy},$$ since we know that $\sum_i(x_i-\bar{x})=\sum_i(y_i-\bar{y})=0$.