From Singular Value Decomposition, we know that:
Any $m$ x $n$ matrix A can be factored into $A=U\Sigma V^{T}$ , where $U$ and $V$ are orthogonal, and $\Sigma$ is of the same size as $A$ with all entries zero except down the main diagonal where the successive entries are $\sigma _{1}\geq …\geq \sigma _{k} > 0 $ for some $k$ with $k\leq$ min$(m,n)$.
To find $U$, $\Sigma$, and $V$ , we can consider $AA^{T}$ and $A^{T}A$ which are symmetric matrices.
$AA^{T}=(U\Sigma V^{T})(V\Sigma^{T} U^{T})=U(\Sigma \Sigma^{T}) U^{T}$ ($\because$ V is orthogonal implies $V^{T}V=I_{n}$)
$A^{T}A=(V\Sigma^{T} U^{T})(U\Sigma V^{T})=V(\Sigma^{T} \Sigma)V^{T} $ ($\because$ U is orthogonal implies $U^{T}U=I_{m}$)
From Spectral Theorem, I understand that, since $AA^{T}$ and $A^{T}A$ are symmetric matrices, $U$ must be the eigenvector matrix for $AA^{T}$, and $\Sigma \Sigma^{T}$ is the eigenvalue matrix for $AA^{T}$; whereas $V$ must be the eigenvector matrix for $A^{T}A$, and $\Sigma^{T} \Sigma$ is the eigenvalue matrix for $A^{T}A$.
But, I don't understand why the $k$ singular values on the diagonal of $\Sigma$ are the square roots of the nonzero eigenvalues of both $AA^{T}$ and $A^{T}A$. It seems like this is only true if $\Sigma \Sigma^{T}$=$\Sigma ^{2}$ and $\Sigma^{T} \Sigma$=$\Sigma ^{2}$ . But $\Sigma \Sigma^{T}$ is $m$ x $m$ matrix, whereas $\Sigma^{T} \Sigma$ is $n$ x $n$ matrix. How can both of them be equal to $\Sigma^{2}$ ?
I'm so confused. 🙁
Best Answer
Let's just multiply some matrices. \begin{align*} \begin{pmatrix} 2 & 0 & 0 \\ 0 & 3 & 0 \\ 0 & 0 & 0 \\ 0 & 0 & 0 \end{pmatrix} \begin{pmatrix} 2 & 0 & 0 & 0 \\ 0 & 3 & 0 & 0 \\ 0 & 0 & 0 & 0 \end{pmatrix} &= \begin{pmatrix} 4 & 0 & 0 & 0 \\ 0 & 9 & 0 & 0 \\ 0 & 0 & 0 & 0 \\ 0 & 0 & 0 & 0 \end{pmatrix}, \\ \begin{pmatrix} 2 & 0 & 0 & 0 \\ 0 & 3 & 0 & 0 \\ 0 & 0 & 0 & 0 \end{pmatrix} \begin{pmatrix} 2 & 0 & 0 \\ 0 & 3 & 0 \\ 0 & 0 & 0 \\ 0 & 0 & 0 \end{pmatrix} &= \begin{pmatrix} 4 & 0 & 0 \\ 0 & 9 & 0 \\ 0 & 0 & 0 \end{pmatrix}. \end{align*} See how the non-zero diagonal entries of $\Sigma\Sigma^T$ and $\Sigma^T\Sigma$ agree?