I'm trying to prove the inverse matrix of $X=U\Sigma^{}V^T$ is $X^{-1}=V\Sigma^{-1}U^T$.
X is invertible and dxd square matrix, U and V are orthogonal matrices, which are dxk, and $\Sigma$ is kxk matrix.
Since the orthogonal matrices are rectangular, $U^TU=V^TV=I\neq VV^T$
When we inverse the matrix X, then
$X^{-1}=(U\Sigma V^{-1})^{-1} = V\Sigma^{-1}U^{-1}$
Since $U^TU=I$, we can say when $U^T$ is multiplied to the left side of U, $U^T=U^{-1}$.
However, in the above $X^{-1}$case, can we say $U^{-1}=U^T$?
Also, can we say $U^{-1}=(U^T)^{-1}$ since $U^TU=I$?
If not, how do we prove $X^{-1}=V\Sigma^{-1}U^T$?
Best Answer
What you're trying to prove is quite wrong. If a matrix is not square, then it is not invertible. No exceptions. Even a square a matrix is only invertible if all of the columns (equivalently, rows) are linearly independent.
The only way to bypass this is to come up with what are known as pseudoinverses. One of the most famous of these is the Moore-Penrose inverse, which for a matrix $A$ is typically denoted as $A^+$ and is of the form:
$$ A^+ \;\; =\;\; A^T\left (AA^T\right )^{-1}. $$
Note that this is only possible if $AA^T$ is invertible (this matrix is square) but $A^+$ only serves as a right inverse since $AA^+ = I$ but $A^+A \neq I$ in general.
Edit
In the case that $A$ is square and invertible, then we find that each piece of the SVD is invertible as well. Hence if $A = U\Sigma V^T$ then the inverse is given by
$$ A^{-1} \;\; =\;\; \left (U\Sigma V^T\right )^{-1} \;\; =\;\; \left (V^T\right )^{-1}\Sigma^{-1}U^{-1} \;\; =\;\; V\Sigma^{-1}U^T, $$
which follows directly from the properties of the inverse.