$A$ is real valued matrix of size mxn, m > n. SVD decomposition of $A$:
$A=U\Sigma V^T$, where $U$, $V$ are orthogonal and $\Sigma$ is diagonal.
Moore Penrose pseudoinverse $A^+ = V\Sigma^+U^T$. How to show that $\hat{\mathbf{x}}= A^+\mathbf{b}$ is a solution of $A\mathbf{x} = \mathbf{b}$.
$$AA^+\mathbf{b} = U\Sigma V^T V\Sigma^+U^T \mathbf{b}=U\Sigma \Sigma^+U^T\mathbf{b}$$
How to show that the left part is equal $\mathbf{b}$ ?
Best Answer
$A^+b$ is not necessarily a solution of $Ax=b$. It is a solution only when $Ax=b$ is solvable. For instance, consider $$ A=\pmatrix{1\\ 0},\ b=\pmatrix{0\\ 1}. $$ Then $Ax=b$ is not solvable and $\hat{x}=A^+b=\pmatrix{1&0}\pmatrix{0\\ 1}=0$ is not a solution to $Ax=b$.
When $Ax=b$ is solvable, we have $U\Sigma V^Tx=b$ for some $x$ and in turn $\Sigma (V^Tx)=U^Tb$. It follows that if $\Sigma$ has $k$ nonzero singular values, then the last $n-k$ rows of $\Sigma$ and in turn, of $U^Tb$, are zero. Therefore $$\Sigma\Sigma^+U^Tb=(I_k\oplus 0_{(n-k)\times(n-k)})U^Tb=U^Tb=b$$ and hence $U\Sigma\Sigma^+U^Tb=b$.