Let $x_1$ and $x_2$ such that $x_1\ne x_2$ are two least-squares solutions to the equation $Ax=b$.
Prove that $Ax_1=Ax_2$, meaning that the projection of $b$ on $Col(A)$ is unique.
My way:
I know that the norm of the residual vectors is the same. I'd like to show that $Ax_1-b$=$Ax_2-b$ but I stuck here.
Best Answer
First, we know that $Ax_1-b$ and $Ax_2-b$ are in the left nullspace of $A$ (i.e. the orthogonal complement of the image of $A$). This is because, if $Ax_1-b$ was not in the left nullspace of $A$, then the residual $\|Ax_1-b\|$ would not be minimized and $x_1$ would not be a least-squares solution.
Now, notice that: $$b=Ax_1-(Ax_1-b)=Ax_2-(Ax_2-b)\implies Ax_1-Ax_2=(Ax_1-b)-(Ax_2-b)$$ The left side of this equation is clearly is in the image of $A$, since $Ax_1$ and $Ax_2$ are both in the image of $A$. Meanwhile, the right side of this equation is in the left nullspace of $A$, since $Ax_1-b$ and $Ax_2-b$ are both in the left nullspace of $A$. Thus, this quantity is in both the image and the left nullspace of $A$, but since these two subspaces are orthogonal complements of each other, the only way a vector can be in both of them at the same time is if that vector is $\vec 0$.
Therefore: $$Ax_1-Ax_2=\vec 0\implies Ax_1=Ax_2$$