least squareslinear algebra
True or false: The unique least norm solution to $Ax=b$ is the orthogonal projection of b onto $R(A)$
So, first isn't this the definition of least squares? The only incorrect thing I can think of is the solution may not be unique.
Also, the detailed answer states: The minimal solution of $Ax = b$ is in $R(A^T)$
Second, shouldn't it be in the $R(A)$, since it is the projection of $b$ onto the column space of $A$?
No. If the columns of $A$ are orthonormal, then $A^T A=I$, the identity matrix, so you get the solution as $A A^T v$.
A least squares solution is not the shadow you refer to in the shining light analogy. This shadow is the orthogonal projection of $b$ onto the column space of $A$, and it is unique. Call this projection $p$. A least squares solution of $Ax = b$ is a vector $x$ such that $Ax = p$. The vector $x$ need not be unique.
Consider the matrix
$$A = \begin{bmatrix} 4 &8 \\ 6 &12 \end{bmatrix}$$ and the vector
$$b = \begin{bmatrix} 5 \\ 1 \end{bmatrix} $$
which is not in $C(A) = \textrm{span} \left( \begin{bmatrix} 4 \\ 6 \end{bmatrix} \right)$. The orthogonal projection of $b$ onto $C(A)$ is given by
$$ p = \frac{\begin{bmatrix} 4 \\ 6 \end{bmatrix} \cdot \begin{bmatrix} 5 \\ 1 \end{bmatrix}}{\begin{bmatrix} 4 \\ 6 \end{bmatrix} \cdot \begin{bmatrix} 4 \\ 6 \end{bmatrix}} \begin{bmatrix} 4 \\ 6 \end{bmatrix} = \begin{bmatrix} 2 \\ 3 \end{bmatrix}$$
A least squares solution of $Ax = b$ is a vector $x$ such that
$$Ax = p$$
This system has infinitely many solutions. The solution set is
$$x = \left\{ t \begin{bmatrix} -2 \\ 1 \end{bmatrix} + \begin{bmatrix} \frac{1}{2} \\ 0 \end{bmatrix}, t \in \mathbb{R} \right\} $$
Therefore, both $x_1 = \begin{bmatrix} \frac{1}{2} \\ 0 \end{bmatrix}$ and $x_2 = \begin{bmatrix} -\frac{7}{2} \\ 2 \end{bmatrix}$, for instance, are least squares solutions, because both $Ax_1 = p$ and $Ax_2 = p$. But neither of these solutions is the "shadow" you refer to in the shining light analogy. Rather, $p$ is the shadow, and $x_1$ and $x_2$ are simply vectors you could multiply $A$ by to get $p$.
Best Answer
Starting with $$ \mathbf{A}x = b, $$ where the system matrix of rank $\rho$ has $m$ rows, and $n$ columns: $$ \mathbf{A} \in \mathbb{C}^{m\times n}_{\rho}, \quad x \in \mathbb{C}^{n}, \quad b \in \mathbb{C}^{m}. $$ The general least squares problem is defined as $$ x_{LS} = \left\{ x \in \mathbb{C}^{n} \colon \lVert \mathbf{A}x - b\rVert_{2}^{2} \text{ is minimized}\right\} $$ and the solution is $$ x_{LS} = \color{blue}{\mathbf{A}^{\dagger} b} + \color{red}{\left( \mathbf{I}_{n} - \mathbf{A}^{\dagger}\mathbf{A} \right)y}, \quad y\in\mathbb{C}^{n}. $$ Range space components are in blue, nullspace in red.
The set of least squares minimizers is an affine space (dashed red line) passing through the range space of $\mathbf{A}$ at the point $x_{LS} = \mathbf{A}^{\dagger}b,$ as seen in the figure.
Every point on the dashed line is a least squares minimizer. Which point has minimum length? That is, which point is closest to the origin? $$ \lVert x_{LS}(y) \rVert_{2}^{2} = \lVert \color{blue}{\mathbf{A}^{\dagger} b} + \color{red}{\left( \mathbf{I}_{n} - \mathbf{A}^{\dagger}\mathbf{A} \right)y} \rVert_{2}^{2} = \lVert \color{blue}{\mathbf{A}^{\dagger} b} \rVert_{2}^{2} + \lVert \color{red}{\left( \mathbf{I}_{n} - \mathbf{A}^{\dagger}\mathbf{A} \right)y} \rVert_{2}^{2} $$ We can control the nullspace term in red by selecting the vector $y=0$; this choice is the least squares minimizer of minimum norm, $\color{blue}{\mathbf{A}^{\dagger} b}.$
Notice that the nullspace is trivial $\color{red}{\mathcal{N}\left( \mathbf{A} \right)}=\left\{ \mathbf{0}\right\}$ when $n=\rho$. Therefore, the set of minimizers is a point, and the solution is unique.