Solved – How to interpret the loadings of the second principal component

correlationdescriptive statisticsinterpretationpca

I perform the following experiment:
each observer performs 3 tasks to evaluate his strength (S variables), additionally each of them perform 2 IQ tasks.
So, I have a dataset with 18 rows for observers and 5 columns: 3 for S and 2 for IQ variables.

I do PCA and for PC1 all 5 loadings are of the same sign, and have similar absolute value (0.4-0.5). For PC2 loadings are in the same absolute value range, but loadings for S and loadings for IQ have different signs.

How can I interpret it?

Can I say that S and IQ are positively correlated (according to PC1)? Or negatively correlated (according to PC2)?

UPDATE:

As I understand from comments and answers I have situation similar to this:

Is it correct interpretation?

Technical details below

1) I perform PCA and study first 3 components for significance. To do it I sample all variables independently with replacement. Thus, I obtain a null-distribution of explained residual Variance for each component (for first component it was obviously explained total variance). I found that observed values for first and second components are very improbable for null-distribution (p-value<<0.01). Thus, I conclude, that 1 and 2 principle components are significant.

2) I check whether I have "noisy" variables in my dataset. To address this question I perform sampling of each variable separately and obtain the null-distribution for case that this variable is independent from others. I found that all variables has similar input in explained residual variance for both first and second components. Perfect.

3) I study the loading of the 1 and 2 principle components. For first all 5 loadings were of the same sign, and has similar absolute value (0.4-0.5). For second principle component loading were in the same absolute value range, but loading for S and loadings for IQ have different sign, and always have same sig inside group. (To simplify S variables loadings always were positive, IQ variables loadings always were negative). To evaluate the stability of this effect i perform sampling of observers with replacement. And obtain null-distribution for each loading for different samples from general populations of observers. They were stable (in less then 1% of samples loadings cross 0).

Thus, I conclude, (A) that first and second component explain significant amount of residual variance, (B) first component loading are of the same size, second component loading dissect 5 variable to 2 groups, which are of a different sign. These groups are the same as biology under variables (physical or mental abilities).

Now comes a most hard part. I should explain these results to my grandmother (boss, student, reviewer… I am not sure what will be easier).

I may say that more muscles you have more clever You are, because all 5 variables projections on first component are cocorrelated (or something like this). But I have no ideas how to explain different sign for "muscles" and "brain" on the second component.

If I will have different signs for a first component loadings everything will be easier.

Best Answer

First of all, you should do a scatter plot of the projection of your individuals on the first two PCs. If instead of seeing a single ellipsis, they cluster in different groups, you'll find an easy interpretation of your data.

If they fall in a single elipsis, you can interpret the PCA as giving you low dimensional (approximate) models of your data.

If you decide to keep only the first PC, you consider that the individuals are roughly distributed along one axis (the long axis of the mentioned ellipsis), given by this PC. In your case, you can interpret this axis as a "good performer/bad performer" axis. As all your loadings have similar values, this means that you consider that a typical individual will have similar scores at all five tests, and the coordinate of an individual on this axis is approximately proportional to its mean score.

If you decide to keep the two first PCs, you consider that they are distributed in a plane. The first axis is as previously; the second axis, orthogonal to the first one, materializes the differences between people with the same coordinate on the first axis. In you case, this means that among people with similar scores, some are more of the muscular type, the others are more intellectuals.

The decision to keep one, two, or more PCs, to give a good description of your data, should rely in particular on the eigenvalues associated to the PCs (or on the proportion of explained variance...).

Related Solutions

Solved – Polychoric PCA and component loadings in Stata

Although you can store all scores in variables, you cannot display the weights for all of them. But as they are important for a meaningful interpretation of the components, you could use the generated variables containing the score to get back on the weights.

Solved – How to interpret PCA loadings

Loadings (which should not be confused with eigenvectors) have the following properties:

Their sums of squares within each component are the eigenvalues (components' variances).
Loadings are coefficients in linear combination predicting a variable by the (standardized) components.

You extracted 2 first PCs out of 4. Matrix of loadings $\bf A$ and the eigenvalues:

A (loadings)
         PC1           PC2
X1   .5000000000   .5000000000 
X2   .5000000000   .5000000000 
X3   .5000000000  -.5000000000 
X4   .5000000000  -.5000000000
Eigenvalues:
    1.0000000000  1.0000000000

In this instance, both eigenvalues are equal. It is a rare case in real world, it says that PC1 and PC2 are of equal explanatory "strength".

Suppose you also computed component values, Nx2 matrix $\bf C$, and you z-standardized (mean=0, st. dev.=1) them within each column. Then (as point 2 above says), $\bf \hat {X}=CA'$. But, because you left only 2 PCs out of 4 (you lack 2 more columns in $\bf A$) the restored data values $\bf \hat {X}$ are not exact, - there is an error (if eigenvalues 3, 4 are not zero).

OK. What are the coefficients to predict components by variables? Clearly, if $\bf A$ were full 4x4, these would be $\bf B=(A^{-1})'$. With non-square loading matrix, we may compute them as $\bf B= A \cdot diag(eigenvalues)^{-1}=(A^+)'$, where diag(eigenvalues) is the square diagonal matrix with the eigenvalues on its diagonal, and + superscript denotes pseudoinverse. In your case:

diag(eigenvalues):
1 0
0 1

B (coefficients to predict components by original variables):
    PC1           PC2
X1 .5000000000   .5000000000 
X2 .5000000000   .5000000000 
X3 .5000000000  -.5000000000 
X4 .5000000000  -.5000000000

So, if $\bf X$ is Nx4 matrix of original centered variables (or standardized variables, if you are doing PCA based on correlations rather than covariances), then $\bf C=XB$; $\bf C$ are standardized principal component scores. Which in your example is:

PC1 = 0.5*X1 + 0.5*X2 + 0.5*X3 + 0.5*X4 ~ (X1+X2+X3+X4)/4

"the first component is proportional to the average score"

PC2 = 0.5*X1 + 0.5*X2 - 0.5*X3 - 0.5*X4 = (0.5*X1 + 0.5*X2) - (0.5*X3 + 0.5*X4)

"the second component measures the difference between the first pair of scores and the second pair of scores"

In this example it appeared that $\bf B=A$, but in general case they are different.

Note: The above formula for the coefficients to compute component scores, $\bf B= A \cdot diag(eigenvalues)^{-1}$, is equivalent to $\bf B=R^{-1}A$, with $\bf R$ being the covariance (or correlation) matrix of variables. The latter formula comes directly from linear regression theory. The two formulas are equivalent within PCA context only. In factor analysis, they are not and to compute factor scores (which are always approximate in FA) one should rely on the second formula.