I know that a unitary matrix is a matrix whose inverse equals its conjugate transpose (or that multiplying it by its conjugate transpose yields the identity), but I don't have a deep intuition about it (I just accept the definition). So for example, when I encounter the statement that the left and right singular decompositions $U$ and $V$ in the SVD are unitary, I don't get the significance. I would appreciate if somebody could enlighten me to connect the dots and how to feel when encountering unitary matrices. I have the feeling there is something unwritten that I'm missing.
[Math] What’s the interpretation of a unitary matrix
linear algebramatricesunitary-matrices
Best Answer
Structurally, unitary matrices are rotations and reflections. Perhaps it's more clear to first picture unitary diagonalization before the singular value decomposition. Suppose we unitarily diagonalize $$A = UDU^{\dagger}$$ In unitary diagonalization, we first rotate (and possibly reflect) from our standard basis into our new orthonormal basis. This is the action of $U^{\dagger}$. Then we perform stretches by the magnitudes of the eigenvalues in the respective basis directions. This is the action of the diagonal matrix $D$. Finally we rotate back to our original basis, which is the action of $U$ which reverses $U^\dagger$.
The action of a singular value decomposition is virtually identical, except that the "diagonal" matrix $\Sigma$ does not necessarily map the same space to itself, so that the rotations happen in different vector spaces.