I am trying to decompose the following matrix using the Singular Value Decomposition (SVD): $$A = \begin{bmatrix}
4 & 4\\
-3 & 3\\
\end{bmatrix} = U\Sigma V^T$$
Here is my work (I know this is far from the most efficient way to do SVD, but please follow along my way):
Finding $\Sigma$ and $V$:
$$A^T A = \begin{bmatrix}25 & 7\\7 & 25\\\end{bmatrix} \quad\text{with } \lambda_1 = 32, v_1 = \begin{bmatrix}1/\sqrt2\\1/\sqrt2\\\end{bmatrix} \quad\text{and } \lambda_2 = 18, v_2 = \begin{bmatrix}1/\sqrt2\\-1/\sqrt2\\\end{bmatrix}$$
So, $$V = \begin{bmatrix}1/\sqrt2 & 1/\sqrt2\\1/\sqrt2 & -1/\sqrt2\\\end{bmatrix} \quad \text{and} \quad \Sigma = \begin{bmatrix} \sqrt{32} & 0 \\ 0 & \sqrt{18}\\\end{bmatrix} $$
Finding $U$:
$$A A^T = \begin{bmatrix}32 & 0\\0 & 18\\\end{bmatrix} \quad\text{with } \lambda_1 = 32, u_1 = \begin{bmatrix}1\\0\\\end{bmatrix} \quad\text{and } \lambda_2 = 18, u_2 = \begin{bmatrix}0\\1\\\end{bmatrix}$$
So, $$U = \begin{bmatrix}32 & 0\\0 & 18\\\end{bmatrix} $$
However, $$U \Sigma V^T = \begin{bmatrix}4 & 4\\3 & -3\\\end{bmatrix} \neq A$$
Did I do something wrong?
Next attempt:
This time, I used $u_2 = \begin{bmatrix}0\\-1\\\end{bmatrix}$ instead of $\begin{bmatrix}0\\1\\\end{bmatrix}$. So, $U = \begin{bmatrix}1 & 0\\0 & -1\\\end{bmatrix}$.
Now, it seems to work: $$U \Sigma V^T = \begin{bmatrix}4 & 4\\-3 & 3\\\end{bmatrix} = A.$$
So my question is: Does it matter which singular vectors you choose for $U$ and $V$? In other words, if you find a singular vector $x$ with unit length, how do you know to choose $x$ or $-x$? I know that in Eigenvalue decomposition, it didn't matter because you can change the diagonal matrix $\Lambda$ accordingly. What about in SVD?
Best Answer
Short Answer: The convention of choosing non-negative singular values locks our answer for $U$.
Explanation: The square of the singular values ($\sigma_i$) of a matrix $A$ is equal to the eigenvalues of $A^T A$ ($\lambda_i$), which are all non-negative (since $A^T A$ is positive semidefinite). There are two solutions to $\sigma_i^2 = \lambda_i$: the positive root ($\sigma_i = \sqrt\lambda_i$), and the negative root ($\sigma_i = -\sqrt\lambda_i$). If we were free to choose either one, then we could also choose either $u_i$ or $-u_i$ as the $i^{th}$ left singular vector, since we can adjust the sign of $\sigma_i$ in the diagonal matrix $\Sigma$ to match the final calculation for $A = U \Sigma V^T$. However, since it is a convention to use the positive root in $\Sigma$, our choice for $u_i$ is locked by $A$ and $v_i$: $$u_i = \frac{A v_i}{\sigma_i}.$$
In the example above, $U = \begin{bmatrix} 1 & 0\\ 0 & 1\\ \end{bmatrix}$ works fine if we choose $\sigma_2 = -\sqrt{18}$. Then: $$U \Sigma V^T = \begin{bmatrix} 1 & 0\\ 0 & 1\\ \end{bmatrix} \begin{bmatrix} \sqrt{32} & 0\\ 0 & -\sqrt{18}\\ \end{bmatrix} \begin{bmatrix} 1/\sqrt2 & 1/\sqrt2\\ 1/\sqrt2 & -1/\sqrt2\\ \end{bmatrix} = \begin{bmatrix} 4 & 4\\ -3 & 3\\ \end{bmatrix} = A$$
However, since we choose the positive root for $\sigma_2$ by convention, we would have to change the sign of either $u_2$ or $v_2$ (the latter would flip the sign of the second row of $V^T$): $$U \Sigma V^T = \begin{bmatrix} 1 & 0\\ 0 & -1\\ \end{bmatrix} \begin{bmatrix} \sqrt{32} & 0\\ 0 & \sqrt{18}\\ \end{bmatrix} \begin{bmatrix} 1/\sqrt2 & 1/\sqrt2\\ 1/\sqrt2 & -1/\sqrt2\\ \end{bmatrix} = \begin{bmatrix} 4 & 4\\ -3 & 3\\ \end{bmatrix} = A$$
$$\text{or}$$ $$U \Sigma V^T = \begin{bmatrix} 1 & 0\\ 0 & 1\\ \end{bmatrix} \begin{bmatrix} \sqrt{32} & 0\\ 0 & \sqrt{18}\\ \end{bmatrix} \begin{bmatrix} 1/\sqrt2 & 1/\sqrt2\\ -1/\sqrt2 & 1/\sqrt2\\ \end{bmatrix} = \begin{bmatrix} 4 & 4\\ -3 & 3\\ \end{bmatrix} = A$$