I am looking at a handful of problems where I am trying to fit a linear model using generalized least squares (GLS) where the covariance matrix of the error term is relatively "nice". I was wondering if in these cases the fixed effect parameter estimates are equivalent to those from ordinary least squares (OLS) method.
Specifically, I am looking at performing linear regression with the model:
$Y = \beta X + \epsilon$,
where $\epsilon$ has distribution $N(0, \Sigma)$. $\Sigma$ is a known invertible positive definite covariance matrix. In addition to that, it is known that $X$ has full rank (i.e. rank $k$) — which I think implies that $X'X$ is also positive definite. The question is whether the estimates of $\beta$ will be the same for OLS and GLS (i.e. $\beta$ = $(X'X)^{-1}(X'Y)$)
Furthermore, from Amemiya (1985) and Lu and Schmidt (2011), we have that the OLS and GLS estimates will be the same if
1) $\Sigma$, $X'X$ and $X'\Sigma^{-1}X$ are positive definite, and
2) $X = HA$ for some nonsingular $A$ where the columns of $H$ are $K$ eignvectors of $\Sigma$.
Question: In the setup above, are conditions (1) and (2) satisfied?
It's been a while since I've done much linear algebra, but this seems straightforward, so I think I'm missing something. If $X$ is full rank, then the column vectors span some $k$-dimensional subspace of $R^n$. Furthermore, since $\Sigma$ is also full rank, its eigenspace is all of $R^n$, and its eigeinvectors form a basis of $R^n$. Any selection, $H$, of $k$ eigienvectors will span a $k$-dimensional subspace of $R^k$. Between any two same-dimensional linear subspaces there always exists a homeomorphism, so for some invertible matrix $B$, $B$ will transform $H$ into a basis for the $\text{span}(X)$, and since $X \in \text{span}(X)$, there exists some invertible $A$ to transform $H$ into $A$, i.e. $X = HA$.
But I think there's a good chance I missed something important here.
Some miscellania: $\Sigma$ is an inverse distance matrix. The element $\sigma_{ij} = D – d(i,j)$ where $D$ is the maximum difference between any two elements $i$ and $j$, and $d$ is the distance between $i$ and $j$, always greater than zero if $i \neq j$.
Best Answer
The following example provides a proof of the answer.
\begin{align*} X &= \begin{bmatrix} 1 & 0 \\ 1 & 0 \\ 0 & 1 \\ 0 & 1 \\ \end{bmatrix}, Y = \begin{bmatrix} 1 \\2 \\3\\4 \end{bmatrix}, \Sigma = \begin{bmatrix} 1 & 0&0&0 \\ 0&5&0&0 \\ 0&0&5&0\\ 0&0&0&5 \end{bmatrix}. \end{align*}
Notice that $\Sigma, X'X$ and $X'\Sigma^{-1}X$ are all diagonal matrices with non-zero, positive, elements on the diagonals. Thus, they are all positive definite and have the standard basis vectors as eigenvectors. That is, they satisfy the setup and condition 1). It is easy to check that the OLS and GLS estimates are different (see code below). Thus, condition 2) must not hold. Let's see why.
In this example, $k=2$ so the columns of $H$ are two eigenvectors of $\Sigma$. Let $A=[a_1, a_2]$. Then $X = HA$ implies that $Ha_1 = x_1 = [1,1,0,0]'$. The eigenvectors of $\Sigma$ are the standard basis vectors, say $e_i$, and, thus, it must be that $H = [e_1, e_2]$ up to reordering of the columns. But then $x_2 =[0,0,1,1]' \notin \mathrm{span}(H)$, i.e. we cannot pick $a_2$ to satisfy the requirement that $X=HA$. We conclude condition 2) is not satisfied.
The following R code snippet shows that the GLS estimates, in this case WLS because of the diagonal covariance matrix, differ from the OLS estimates.