I'm having a bit of trouble tying two related ideas together, and I think I'm just missing a silly detail somewhere.
In the standard formulation of linear least-squares, it can be shown that given a full-rank matrix $A$, one can form the projection matrix
$$
P_A = A(A^TA)^{-1}A^T
$$
which acts on vectors by orthogonally projecting them onto the column space of $A$.
On the other hand, given a subspace $S \subseteq V$ of a vector space, where $S = \mathrm{Span}(\mathbf u_1, \mathbf u_2, \cdots \mathbf u_k)$ and the $\mathbf u_i$ form an orthonormal basis of $S$, then one can construct a matrix with each $\mathbf u_i$ in the $i$th column:
$$
U = [\mathbf u_1, \mathbf u_2, \cdots \mathbf u_k].
$$
One can then form the projection
$$
P_S = UU^T
$$
which projects vectors in $V$ onto the subspace $S$.
I have two questions:
-
The matrix $U$ seems to be orthogonal, which would seem to mean it satisfies $UU^T = I$. What exactly am I missing?
-
The column space of $U$ is $S$ by construction, so how would one recover the formula for $P_S$ by letting $A=U$ in the formula for $P_A$? If $U$ is orthogonal, I see how $(A^TA)^{-1} = I$, but by the first question, it would also seem to reduce the remaining $AA^T$ to $I$ as well.
Best Answer
If $k$ is less than the dimension of $V$, then while $U^T U$ will be an identity matrix (because the columns of $U$ are orthonormal), $UU^T$ will generally not be an identity matrix (because there will be no guarantee that the rows of $U$ are orthonormal. In fact, you can show that if $U$ has size $n\times k$ where $n > k$, then $UU^T$ can never be the identity matrix.)
For example, suppose $$ U = \begin{bmatrix} 1/\sqrt{2} & 0 \\ 1/\sqrt{2} & 0\\ 0 & 1\end{bmatrix}. $$
Then $U^TU = I$, while $$ \begin{align*} UU^T &= \begin{bmatrix} 1/\sqrt{2} & 0 \\ 1/\sqrt{2} & 0\\ 0 & 1\end{bmatrix}\begin{bmatrix} 1/\sqrt{2} & 1/\sqrt{2} & 0\\ 0 & 0 & 1\end{bmatrix} \\ &= \begin{bmatrix}1/2 & 1/2 & 0\\ 1/2 & 1/2 & 0 \\ 0 & 0& 1\end{bmatrix}. \end{align*} $$