[Math] Projecting data onto a vector

eigenvalues-eigenvectorslinear algebravectors

I have learned that projecting a vector a onto a vector b is done by multiplying the orthogonal projection of a (say $\mathbf{a_o}$) with the unit vector $\mathbf{\hat{b}}$ in the direction of b =

  • Orthogonal projection of a onto b = $\mathbf{a_o}$ * $\mathbf{\hat{b}}$

However in the context of principal component analysis, I often read that the projection of the data D onto an eigenvector v is simply $\mathbf{v}^{T}$*$D$ (and so we don't take the orthogonal components of the data vectors of D..?) – why is that?

Thanks

Best Answer

For two vectors $a,b \in \mathbb R^n$, the orthogonal projection of $a$ onto the span of $b$ is given by $P_b(a) = \langle a ,b\rangle \frac{b}{\langle b,b\rangle}$, where the brackets denote the scalar product, i.e. $\langle a, b\rangle = a^Tb = b^Ta$. If $b$ has norm $1$, i.e. $\langle b,b\rangle =1,$ this simplifies to $P_b(a) = \langle a ,b\rangle b = (b^Ta)b$.

If you want to plot this vector, the vector part ($b$) tells you in which direction to go and the scalar factor ($\langle a ,b\rangle = (b^Ta)$) tells you how far. If you already know in which direction to go, the only interesting information is the distance.

The data matrix $D$ in your example consists of a lot of vectors like $a$, written down in columns.

So if you just consider a (normed) vector $b$ and look at $b^T D, $ this will be a 1xn column vector.

What does this vector tell us? The first entry is the scalar factor for the first column, the second for the second, and so on. What's missing is the vector $b$ to multiply it with.

That is: $b^T D$ is the column vector which contains the coordinates of the columns of $D$ with respect to $b$. If you want to plot this, you can forget about the actual direction of $b$, as the aim is to consider $b$ as the 'new x' and the orthogonal complement of $b$ as the new 'y'. We don't care about the direction, only the coordinates: If $c$ is orthogonal to $b$ and has norm 1, then we can calculate the coordinates of the columns of $D$ as $c^TD$. If we plot the first entry of $b^TD$ against the first entry of $c^TD$, we will have the the data point corresponding to the first column represented in the new coordinate system, and so on for all the other columns of $D$.

Related Question