**Question: In the setup above, are conditions (1) and (2) satisfied?**

Answer: No, in general the conditions are not satisfied.

The following example provides a proof of the answer.

\begin{align*}
X &= \begin{bmatrix}
1 & 0 \\
1 & 0 \\
0 & 1 \\
0 & 1 \\
\end{bmatrix},
Y = \begin{bmatrix}
1 \\2 \\3\\4
\end{bmatrix}, \Sigma = \begin{bmatrix}
1 & 0&0&0 \\
0&5&0&0 \\
0&0&5&0\\
0&0&0&5
\end{bmatrix}.
\end{align*}

Notice that $\Sigma, X'X$ and $X'\Sigma^{-1}X$ are all diagonal matrices with non-zero, positive, elements on the diagonals. Thus, they are all positive definite and have the standard basis vectors as eigenvectors. That is, they satisfy the setup and condition 1). It is easy to check that the OLS and GLS estimates are different (see code below). Thus, condition 2) must not hold. Let's see why.

In this example, $k=2$ so the columns of $H$ are two eigenvectors of $\Sigma$. Let $A=[a_1, a_2]$. Then $X = HA$ implies that $Ha_1 = x_1 = [1,1,0,0]'$. The eigenvectors of $\Sigma$ are the standard basis vectors, say $e_i$, and, thus, it must be that $H = [e_1, e_2]$ up to reordering of the columns. But then $x_2 =[0,0,1,1]' \notin \mathrm{span}(H)$, i.e. we cannot pick $a_2$ to satisfy the requirement that $X=HA$. We conclude condition 2) is not satisfied.

The following R code snippet shows that the GLS estimates, in this case WLS because of the diagonal covariance matrix, differ from the OLS estimates.

```
X <- matrix(c(1,1,0,0,0,0,1,1), ncol = 2); Y <- 1:4; E <- diag(c(1, 5, 5, 5))
coef(lm(Y ~ X - 1)
>X1 X2
>1.5 3.5
coef(lm(Y ~ X - 1, weights = 1/diag(E)))
>X1 X2
>1.66667 3.50000
```

If we assume that $\sigma^2$ is known, the variance of the OLS estimator only depends on $X'X$ because we do not need to estimate $\sigma^2$. Here is a purely algebraic proof that the variance of the estimator decreases with any additional observation if $\sigma^2$ is known. Suppose $X$ is your current design matrix and you add one more observation $x$, which has dimension $1\times (p+1)$. Your new design matrix is $$X_{new} = \left(\begin{array}{c}X \\ x \end{array}\right).$$
You can check that $X_{new}'X_{new} = X'X + x'x$. Using the Woodbury identity we get
$$
(X_{new}'X_{new})^{-1} = (X'X + x'x)^{-1} = (X'X)^{-1} - \frac{(X'X)^{-1}x'x(X'X)^{-1}}{1+x(X'X)^{-1}x'}
$$
Because $(X'X)^{-1}x'x(X'X)^{-1}$ is positive semi-definite (it is the multiplication of a matrix with its transpose) and $1+x(X'X)^{-1}x'>0$, the diagonal elements of the subtracting term are greater than or equal to zero. So, the diagonal elements of $(X_{new}'X_{new})^{-1}$ are less than or equal to the diagonal elements of $(X'X)^{-1}$.

## Best Answer

This is what I ended up doing: $$Var(\beta^{gls}) = Var[(X'\Sigma^{-1}X)^{-1} X'\Sigma^{-1}\varepsilon]\\ =(X'\Sigma^{-1}X)^{-1} X'\Sigma^{-1} Var(\varepsilon) \Sigma^{-1} X(X'\Sigma^{-1}X)^{-1}\\ = \sigma^2 (X'\Sigma^{-1}X)^{-1} X'\Sigma^{-1} \Sigma^{-1} X(X'\Sigma^{-1}X)^{-1}\\ $$

Want to show $Var(\beta^{gls}) - Var(\beta^{ols})$ is psd. $$\implies \sigma^2 (X'\Sigma^{-1}X)^{-1} X'\Sigma^{-1} \Sigma^{-1} X(X'\Sigma^{-1}X)^{-1} - \sigma^2(X'X)^{-1} \geq 0\\ \iff X'X - X'\Sigma^{-1}X( X'\Sigma^{-1} \Sigma^{-1} X)^{-1}X'\Sigma^{-1}X \geq 0\\ \implies X'(I-\Sigma^{-1}X( X'\Sigma^{-1} \Sigma^{-1} X)^{-1}X'\Sigma^{-1})X \geq 0\\ \implies X'MX\geq0 $$ Where $M$ is the residual maker matrix, and since $M$ is psd, the statement holds.