I am dealing with a problem similar to principal component analysis. Aka, I have a matrix and I want to recover the 'most efficient basis' to explain the matrix variability. With a square matrix these are the eigenvectors, weighted by the eigenvalues.
Originally, I was dealing with square matrices, and I used eigendecomposition to recover the eigenvectors, as explained above. Now however, I am dealing with rectangular matrices and using SVD to recover the efficient basis, i.e. A=USV' where the vectors of U are the recovered basis weighted by S, the singular values.
In my particular application, the sign of the eigenvalues/singular values makes a difference.
Here is my question: with eigendecomposition and square matrices, the eigenvalues will be positive/negative. With SVD, the singular values re contrained to be the absolute value of the eigenvalues, i.e. $s_i$ = |$\lambda_i$|.
Is there anyway to recover the 'sign' of the eigenvalues through SVD?
Thanks!
Best Answer
What you say, that is, the singular values are the absolute values of the eigenvalues, makes only sense for normal matrices, e.g., Hermitian ones. Say, $A\in\mathbb{C}^{n\times n}$ is normal then there exists a unitary $U$ and diagonal $\Lambda$ (with eigenvalues on the diagonal) such that $A=U\Lambda U^*$. The SVD can be obtained simply as $$A=WSV^*=(UD_1^*)(D_1\Lambda D_2^*)(UD_2^*)^*,\tag{1}$$ where $D_1$ and $D_2$ are diagonal matrices such that $D_i^*D_i=1$ (that is, the diagonal elements have unit absolute value) and such that $D_1\Lambda D_2^*$ has non-negative diagonal (for simplicity, I assume the eigen/singular values to be distinct). Therefore, the singular vectors are scalar multiples of the eigenvectors.
Now consider the SVD $A=WSV^*$ is given and let $w_i=We_i$ be the $i$th column of $W$. What happens if you compute the Rayleigh quotient $w_i^*Aw_i$? Using (1), we have $$ \begin{split} w_i^*Aw_i&=e_i^TW^*WSV^*We_i=e_i^T(D_1\Lambda D_2^*)(D_2U^*)(UD_1^*)e_i\\ &=e_i^TD_1\Lambda D_1^*e_i=e_i^TD_1^*D_1\Lambda e_i=e_i^T\Lambda e_i=\lambda_i, \end{split} $$ that is, the Rayleigh quotient associated with the $i$th left (right would work as well) singular vectors gives you the eigenvalue $\lambda_i$. That's not surprising at all since the eigenspace associated with the eigenvalue $\lambda_i$ is the same as the singular vector space associated with the singular value $|\lambda_i|$ and hence the Rayleigh quotients of singular vectors must be equal to the eigenvalues.
This of course does not hold for general matrices but normal ones.