Let $A\in \mathbb{R^{m\times n}}$ have full column rank, $b\in \mathbb{R^m}$ with $b \neq 0$
Show that $P = A(A^TA)^{-1}A^T$ is a projection matrix. Onto what space does $P$ project?
I tried looking at videos,at textbooks, and lecture notes. Which version best answers the question?
version 1:
Since $ATA$ is an invertible matrix, take the normal equation $A^TAx=A^Tb$ and solve for $x$. We get, $x=(A^TA)^{-1}A^Tb$. For $Ax=Pb$, we have $Ax$ as a projection, and $P$ is an orthogonal projector onto the range of $A$. It follows, $A(A^TA)^{-1}A^Tb = Pb$, $\forall b$. Generally, for any matrices $M$ and $N$ s.t. $Mx=Nx$,$\forall x \Rightarrow M=N$, thus $P=A(A^TA)^{-1}A^T$. $P$ projects a vector $b$ onto the spaced spanned by the columns of $A$.
version 2
Given a matrix $A$, the projection of $b$ onto $C(A)$ is $P=Ax$, where $x$ solves $A^TAx=A^Tb$. Since A has full column rank, $A^TA$ is invertible. We write, the projection matrix is $P=A(A^TA)^{-1}A^T$.
version 3
The vector $Ax$ is always in the column space of $A$. We project b onto a vector $P$ in the column space of $A$ and solve $Ax=P$. In matrix form, $A^T(b-Ax)=0 \Rightarrow A^TAx=A^Tb$. Since $A^TA$ is a square matrix, we multiply by $(A^TA)^{-1}$ In $n$ dimensions, $x=(A^TA)^{-1}A^Tb \Rightarrow P=Ax=A(A^TA)^{-1}A^Tb \Rightarrow P=A(A^TA)^{-1}A^T$
Best Answer
Throughout versions 2 and 3, you confuse the projected vector $p = Pb$ with the matrix $P$; you should avoid using the same symbol to mean both. I ignore that point in the following.
I'd say that the most appropriate version depends on the context. Version 1 is good in that it emphasizes the normal equation, which may be the focus of a few weeks of a class in and of itself. Version 2 is particularly terse, which can be desirable; since you are still studying the material, I would avoid this one. Version 3 exploits the fact that $\mathcal C(A)^\perp = \mathcal N(A^T)$, but fails to make this explicit. You also have some unclear word choice.
All together, I would say that, as written, your first version is the most cohesive and complete. I personally prefer the approach you take in the third, but your current writeup requires a lot of clarification.