I understand that Multidimensional scaling (MDS) is same as doing Principal Components analysis (PCA) if Euclidean distance is used, this is known as Metric MDS. But I came across this in a book that "it has been shown (Chatfield and Collins 1980) that the eigenvalues of $XX^T$ (unnormalised outer product matrix) are equal to the eigenvalues $X^TX$ (unnormalised inner product matrix) and eigenvectors of $XX^T$ and $X^TX$ are related by a linear transformation. "
Note, that X denotes a matrix of data, with $n$ features (rows) and $m$ instances (columns).
Now I am unable to get this Chattield and Collins book anywhere, and I can understand that the eigenvalues are equal. But how are the eigenvectors of PCA and metric MDS related to each other ?
Best Answer
I find it helpful to consider the singular value decomposition for questions like this with the assumption that $X$ is a real matrix. Writing $X = UDV^T$, we can see that $XX^T = UD^2U^T$ and $X^TX = VD^2V^T$. As we can see, the eigenvalues of both $XX^T$ and $X^TX$ are contained in the diagonal matrix $D^2$ and are indeed equal. Also, we see that the matrix of eigenvectors of $XX^T$ is $U$, while the matrix of eigenvectors of $X^TX$ is $V$.
Because $$ X^TU = VDU^TU = VD, $$ we have the relationship $X^TU = VD$, similar to that pointed out by whuber.
Assuming $D$ is nonsingular, two additional properties that prove to be quite useful are:
$$ \begin{align} U &= XVD^{-1}\\ V &= X^TUD^{-1}. \end{align} $$