I know how to perform PCA (principal component analysis), but I would like to know steps that should be used for factor analysis.
To perform PCA, let us consider some matrix $A$, for instance:
3 1 -1
2 4 0
4 -2 -5
11 22 20
I have calculated its correlation matrix B = corr(A):
1.0000 0.9087 0.9250
0.9087 1.0000 0.9970
0.9250 0.9970 1.0000
Then I have done eigenvalue decomposition [V,D] = eig(B), resulting in eigenvectors:
0.5662 0.8209 -0.0740
0.5812 -0.4613 -0.6703
0.5844 -0.3366 0.7383
and eigenvalues:
2.8877 0 0
0 0.1101 0
0 0 0.0022
General idea behind the PCA is to choose significant components, form new matrix which have columns eigenvectors, then we need to project original matrix (in PCA it is zero centered). But in factor analysis, for instance we should choose components that have higher that $1$ singular value, also then we are using rotation of factors, please tell me how it is done? For instance in this case.
Please help me understand factor analysis steps, as compared to the PCA steps.
Best Answer
This answer is to show concrete computational similarities and differences between PCA and Factor analysis. For general theoretical differences between them, see questions/answers 1, 2, 3, 4, 5.
Below I will do, step by step, Principal Component analysis (PCA) of iris data ("setosa" species only) and then will do Factor analysis of the same data. Factor analysis (FA) will be done by Iterative principal axis (PAF) method which is based on PCA approach and thus makes one able to compare PCA and FA step-by-step.
Iris data (setosa only):
We have 4 numeric variables to include in our analyses: SLength SWidth PLength PWidth, and the analyses will be based on covariances, which is the same as to say that we analyse centered variables. (If we chose to analyse correlations that would be analysing standardized variables. Analysis based on correlations produce different results than analysis based on covariances.) I will not display the centered data. Let's call these data matrix
X.PCA steps:
FA (iterative principal axis extraction method) steps:
After the extraction (shown above), optional rotation may take place. Rotation is frequently done in FA. Sometimes it is done in PCA exactly the same way. Rotation rotates loading matrix A into some form of "simple structure" which facilitates interpretation of factors greatly (then rotated scores can be recomputed). Since rotation is not what differentiates FA from PCA mathematically and because it is a separate large topic, I won't touch it.