least squareslinear algebravector-spacesvectors
When you find the least squares solution you solve $$A^TA = A\vec b$$ but to find the orthogonal projection into the "subspace" A, you multiply this result (the least squares solution) with the original matrix. Why is this?
If you use the analogy with the light shining orthogonally on to the subspace and the orthogonal projection is the shadow in the subspace, isn't this shadow also the least squares solution?
Starting with $$ \mathbf{A}x = b, $$ where the system matrix of rank $\rho$ has $m$ rows, and $n$ columns: $$ \mathbf{A} \in \mathbb{C}^{m\times n}_{\rho}, \quad x \in \mathbb{C}^{n}, \quad b \in \mathbb{C}^{m}. $$ The general least squares problem is defined as $$ x_{LS} = \left\{ x \in \mathbb{C}^{n} \colon \lVert \mathbf{A}x - b\rVert_{2}^{2} \text{ is minimized}\right\} $$ and the solution is $$ x_{LS} = \color{blue}{\mathbf{A}^{\dagger} b} + \color{red}{\left( \mathbf{I}_{n} - \mathbf{A}^{\dagger}\mathbf{A} \right)y}, \quad y\in\mathbb{C}^{n}. $$ Range space components are in blue, nullspace in red.
The set of least squares minimizers is an affine space (dashed red line) passing through the range space of $\mathbf{A}$ at the point $x_{LS} = \mathbf{A}^{\dagger}b,$ as seen in the figure.
Every point on the dashed line is a least squares minimizer. Which point has minimum length? That is, which point is closest to the origin? $$ \lVert x_{LS}(y) \rVert_{2}^{2} = \lVert \color{blue}{\mathbf{A}^{\dagger} b} + \color{red}{\left( \mathbf{I}_{n} - \mathbf{A}^{\dagger}\mathbf{A} \right)y} \rVert_{2}^{2} = \lVert \color{blue}{\mathbf{A}^{\dagger} b} \rVert_{2}^{2} + \lVert \color{red}{\left( \mathbf{I}_{n} - \mathbf{A}^{\dagger}\mathbf{A} \right)y} \rVert_{2}^{2} $$ We can control the nullspace term in red by selecting the vector $y=0$; this choice is the least squares minimizer of minimum norm, $\color{blue}{\mathbf{A}^{\dagger} b}.$
Notice that the nullspace is trivial $\color{red}{\mathcal{N}\left( \mathbf{A} \right)}=\left\{ \mathbf{0}\right\}$ when $n=\rho$. Therefore, the set of minimizers is a point, and the solution is unique.
Best Answer
A least squares solution is not the shadow you refer to in the shining light analogy. This shadow is the orthogonal projection of $b$ onto the column space of $A$, and it is unique. Call this projection $p$. A least squares solution of $Ax = b$ is a vector $x$ such that $Ax = p$. The vector $x$ need not be unique.
Consider the matrix
$$A = \begin{bmatrix} 4 &8 \\ 6 &12 \end{bmatrix}$$ and the vector
$$b = \begin{bmatrix} 5 \\ 1 \end{bmatrix} $$
which is not in $C(A) = \textrm{span} \left( \begin{bmatrix} 4 \\ 6 \end{bmatrix} \right)$. The orthogonal projection of $b$ onto $C(A)$ is given by
$$ p = \frac{\begin{bmatrix} 4 \\ 6 \end{bmatrix} \cdot \begin{bmatrix} 5 \\ 1 \end{bmatrix}}{\begin{bmatrix} 4 \\ 6 \end{bmatrix} \cdot \begin{bmatrix} 4 \\ 6 \end{bmatrix}} \begin{bmatrix} 4 \\ 6 \end{bmatrix} = \begin{bmatrix} 2 \\ 3 \end{bmatrix}$$
A least squares solution of $Ax = b$ is a vector $x$ such that
$$Ax = p$$
This system has infinitely many solutions. The solution set is
$$x = \left\{ t \begin{bmatrix} -2 \\ 1 \end{bmatrix} + \begin{bmatrix} \frac{1}{2} \\ 0 \end{bmatrix}, t \in \mathbb{R} \right\} $$
Therefore, both $x_1 = \begin{bmatrix} \frac{1}{2} \\ 0 \end{bmatrix}$ and $x_2 = \begin{bmatrix} -\frac{7}{2} \\ 2 \end{bmatrix}$, for instance, are least squares solutions, because both $Ax_1 = p$ and $Ax_2 = p$. But neither of these solutions is the "shadow" you refer to in the shining light analogy. Rather, $p$ is the shadow, and $x_1$ and $x_2$ are simply vectors you could multiply $A$ by to get $p$.