I know there always exists a least-square solution $\hat{x}$, regardless of the properties of the matrix $A$. However, I keep finding online that least-square can have infinitely many solutions, if $A$ is not full column rank.
Shouldn't $\hat{x}$ always be unique, as the minimization of a quadratic function (the error) always yields a global minima/maxima? Therefore, regardless of what the matrix $A$ is (even if it is a badly constructed matrix with dependent columns), least-square should find a single 'best' solution $\hat{x}$?
Is there an easy (or intuitive) proof showing why would the least-square method produce infinitely many solutions if there are dependent columns?
Best Answer
Your least squares solution is minimizing $\hat x^T A \hat x$ If $A$ does not have full rank, there is some vector $y$ such that $Ay=0$. Then $(\hat x+y)^TA(\hat x+y)=\hat x^T A \hat x$ so you can add any multiple of $y$ to your solution and get the same product.