I was reading the Wikipedia article on Singular Value Decomposition. It shows a nice visualisation where the SVD of a matrix $M = U\Sigma V^*$ allows us interpret M as a rotation $V^*$, followed by a scaling $\Sigma$, and a second and final rotation $U$. That makes perfect sense when $M$ is a square.
Now, how can I interpret the decomposition of a non-square matrix? If I take as a simple example a real row matrix $M$, then $U$ always seems to be a scalar equal to 1 or -1 (which also makes total sense), and $\Sigma$ has a single non-zero entry, namely the first SV. And what does the matrix $V^*$ represent?
Best Answer
First we will assume that $M$ is a real matrix. In the complex case, the geometry of the SVD is a little harder to visualize because taking inner products also requires conjugation.
Also, let us define the SVD as follows: if $M$ is $m \times n$, then the SVD is given by $$ M = U \Sigma V^*, $$ where $U,V$ have orthonormal columns, $\Sigma$ is diagonal with non-negative entries, and $U$ is $m \times r$, $\Sigma$ is $r \times r$, and $V^*$ is $r \times n$, and $r = \mathrm{rank}(M)$. I believe this definition of SVD is often called the "thin" or "reduced" version.
With these assumption, it's best to make sense of $U$ and $V$ as storing orthonormal bases for the range and corange (also called column space and row space when referring to matrices), respectively, when $M$ is thought of as a linear map from $\mathbb{R}^n$ to $\mathbb{R}^m$.
So the correct picture would be, first take the $n$-dimensional ball. Then look at its image under $M$; this will be an $m$-dimensional ellipsoid (and its interior). The lengths of the axes of this resulting ellipsoid are the singular values, the normalized directions of these axes are the columns of $U$, and the preimages of those axes are the columns of $V$.
For $n = m$, this corresponds to some rotation and stretching since we're mapping an $n$-sphere to an $n$-ellipsoid.
For your example where $M$ is just a row matrix, then the row matrix $V^{\ast}$ is just the normalized version of $M$.