What is the relationship between SVD and factor analysis? How can use singular values and other matrices from SVD to perform factor analysis or cluster document-term matrix without using other clustering techniques?
Solved – Can SVD be used to perform factor analyis
clusteringfactor analysissvd
Related Solutions
We have implemented this (along with a power iteration refinement) in the scikit-learn python package.
Our implementation is able to find the exact same singular values and vectors if k + p > rank(M) as demonstrated in the tests.
If you cut (k + p) before reaching near zero singular values (i.e. in the k + p < rank(M) case) then the singular vectors are indeed different from the ones you get with the un-truncated version but they might still be very useful in practice for features extraction in machine learning: for instance 'truncated' eigenfaces at 150 work as good for face recognition task with SVM as the top 150 first singular vectors of the full decomposition even though the rank of my faces dataset seems to be much higher.
This randomized / truncated SVD method looks really interesting in practice: it can really cut down the computation time as shown in this benchmark:
Yes, the two errors amount to the same thing. They're telling you (roughly) that two or more of your manifest variables are linearly dependent (like $y_1 = ay_2 + b$ for scalars $a, b$). These two variables (dimensions) would be "redundant", meaning that the sample covariance matrix is not invertible (ie is singular) and therefore not positive definite either.
As for what you ought to do about it, that depends. First I would try to find out which variables are giving you the trouble; a scatterplot matrix might be enough to tell you that. Then you can decide what to do from there - most likely dropping some redundant variables.
Best Answer
Google brought me here, and I dislike how the comments just assume everyone knows that FA and PCA are related. So to answer your question: yes. See Tipping and Bishop, Probabilistic principal component analysis, 1996. This paper is great because: