I am reading a tutorial on PCA. In there the auther makes the claim, that
The fact that the orthonormal basis $\mathbf{U}^\top$ transforms column vectors means that $\mathbf{U}^\top$
is a basis that spans the columns of $\mathbf{X}$. Bases that span the columns are termed the column space of $\mathbf{X}$.
I don't understand what he means by that. I know the column space of a matrix as the set of all linear combinations of the columns of the matrix.
What does the column space of $\mathbf{X}$ in $\mathbf{U}^\top \mathbf{X}$ have to do with $\mathbf{U}^\top$? Is there some theorem I am unaware of the author is indirectly referencing?
You can find the statement on page 8, left column, bottom paragraph:
Best Answer
It's not very clearly written, whatever it's saying. Clearly, $\mathbf{U}^\top$ is a matrix, not a basis, and the columnspace of $X$ is the span of the columns of $\mathbf{X}$, not a catch-all term for the bases of said spaces.
I do have a guess as to what the author is referring to here.
Another way to write the columnspace of $\mathbf{X}$ is $$\{\mathbf{X}v : v \in \Bbb{R}^n\}.$$ Why? It's a simple fact of matrix multipliation that multiplying a matrix by a column vector produces a linear combination of the columns of the matrix, using the entries of the vector as coefficients. More explicitly, $$\left(\begin{array}{c|c} &&& \\ \mathbf{x}_1 & \mathbf{x}_2 & \cdots & \mathbf{x}_n \\ &&& \end{array}\right)\begin{pmatrix} v_1 \\ v_2 \\ \vdots \\ v_n \end{pmatrix} = v_1 \mathbf{x}_1 + v_2 \mathbf{x}_2 + \ldots + v_n \mathbf{x}_n$$ So, the set of all $\mathbf{X}v$ is indeed the same as the set of linear combinations of columns of $\mathbf{X}$.
Here's another relevant identity:
$$\mathbf{A}\left(\begin{array}{c|c} &&& \\ \mathbf{x}_1 & \mathbf{x}_2 & \cdots & \mathbf{x}_n \\ &&& \end{array}\right) = \left(\begin{array}{c|c} &&& \\ \mathbf{A}\mathbf{x}_1 & \mathbf{A}\mathbf{x}_2 & \cdots & \mathbf{A}\mathbf{x}_n \\ &&& \end{array}\right).$$
So, in this way, going from $\mathbf{X}$ to $\mathbf{U}^\top \mathbf{X}$, we are essentially replacing the $\mathbf{x}_i$s in each linear combination
$$v_1 \mathbf{x}_1 + v_2 \mathbf{x}_2 + \ldots + v_n \mathbf{x}_n$$
with $\mathbf{U}^\top \mathbf{x}_i$s, yielding
$$v_1 \mathbf{U}^\top \mathbf{x}_1 + v_2 \mathbf{U}^\top \mathbf{x}_2 + \ldots + v_n \mathbf{U}^\top \mathbf{x}_n.$$
In this way there's kind of a change in basis happening now, as we are interpreting coordinates the were originally made with respect to the columns of $\mathbf{X}$, now in terms of the column vectors of $\mathbf{U}^\top \mathbf{X}$, which are indeed transformed versions of the $\mathbf{x}_i$s.
That's all I think the author means by this.