I was trying to solve the following problem:
Suppose that rectangular matrix A has orthogonal columns w1…wn, where the lengths of these columns are a1…an. What would be the SVD result of this matrix?
I solved the problem, but I can't understand why the singular values will be equal to the length of the column vectors of A? Why these orthogonal column vectors (normalized) will be the left singular vectors of matrix U? What is the intuition behind?
Best Answer
Since you have orthogonal columns, I assume your matrix $A$ is tall matrix and has full column-rank (all columns are independent).
From the SVD we have $A=U \Sigma V^\top $. Now, since matrix $A$ is full column-rank, its vectors are a basis of its own columns space. This is exactly what the left-singular vectors associated with non-zero singular values tell us: they reveal a basis for the column space. Notice also that the column space of $A$ is equal to the columns space of $U$, since they have the same rank.
So both the column vectors of $A$ and the one of $U$ (when removing the $m-n$ vectors associated to the null space) are basis vectors. The only difference is that the vectors $[u_1, \ldots, u_n]$ need to be unitary, i.e., with norm $1$.
You can readily see that it holds:
$$A = \big[\frac{w_1}{\|w_1\|_2}, \ldots, \frac{w_n}{\|w_n\|_2}\big] \begin{bmatrix} \|w_1\|_2 & 0 & 0 \\ 0 & \ddots & \|w_n\|_2 \\ 0 & 0 & 0 \\ 0 & 0 & 0 \end{bmatrix} I$$
In other words: $A e_i=\sigma_iu_i \rightarrow A e_i=\frac{w_e}{\|w_i\|_2}\|w_i\|_2=w_i $ where $e_i$ is the $i$th canonical vector.