Your interpretation is mostly correct. The first PC accounts for most of the variance, and the first eigenvector (principal axis) has all positive coordinates. It probably means that all variables are positively correlated between each other, and the first PC represents this "common factor". The second PC (looks like it has much smaller variance) contrasts b5
and b7
from everything else.
Is there a connection between the blue vectors direction and the position of the scores? Meaning, do variable vectors which end close to some scores have something to do with those same scores?
Here is one way to look at it. Imagine that you had a data point with original coordinates $(1,0,0,\ldots)$, i.e. only one variable is equal to $1$ and others to zero. Then this imaginary data point would have PC scores as the end-point of your b1
vector. The same goes for other vectors as well.
Having said that, as the original 6D space is projected on 2D, many different points can be projected to the same 2D point, so if one blue vector, e.g. b1
has an end-point near one particular red point, it does not necessarily mean that this data point had coordinates $(1,0,0,\ldots)$.
I should add that the above is true only for this particular normalization of a biplot, when blue lines correspond to eigenvectors, and the PC scores are not standardized.
If data are well clustered, how do I interpret the results of the PCA in terms of what variable has a major (or minor) influence on the system?
I don't really understand this question. I would not call these data "well clustered", it rather looks like a unimodal distribution. And all variables seem to be pretty similar in your case, positively correlated between each other and contributing similarly to PC1/PC2.
There are many different ways to produce a PCA biplot and so there is no unique answer to your question. Here is a short overview.
We assume that the data matrix $\mathbf X$ has $n$ data points in rows and is centered (i.e. column means are all zero). For now, we do not assume that it was standardized, i.e. we consider PCA on covariance matrix (not on correlation matrix). PCA amounts to a singular value decomposition $$\mathbf X=\mathbf{USV}^\top,$$ you can see my answer here for details: Relationship between SVD and PCA. How to use SVD to perform PCA?
In a PCA biplot, two first principal components are plotted as a scatter plot, i.e. first column of $\mathbf U$ is plotted against its second column. But normalization can be different; e.g. one can use:
- Columns of $\mathbf U$: these are principal components scaled to unit sum of squares;
- Columns of $\sqrt{n-1}\mathbf U$: these are standardized principal components (unit variance);
- Columns of $\mathbf{US}$: these are "raw" principal components (projections on principal directions).
Further, original variables are plotted as arrows; i.e. $(x,y)$ coordinates of an $i$-th arrow endpoint are given by the $i$-th value in the first and second column of $\mathbf V$. But again, one can choose different normalizations, e.g.:
- Columns of $\mathbf {VS}$: I don't know what an interpretation here could be;
- Columns of $\mathbf {VS}/\sqrt{n-1}$: these are loadings;
- Columns of $\mathbf V$: these are principal axes (aka principal directions, aka eigenvectors).
Here is how all of that looks like for Fisher Iris dataset:
![Fisher Iris biplots, PCA on covariance](https://i.stack.imgur.com/6ddZg.png)
Combining any subplot from above with any subplot from below would make up $9$ possible normalizations. But according to the original definition of a biplot introduced in Gabriel, 1971, The biplot graphic display of matrices with application to principal component analysis (this paper has 2k citations, by the way), matrices used for biplot should, when multiplied together, approximate $\mathbf X$ (that's the whole point). So a "proper biplot" can use e.g. $\mathbf{US}^\alpha \beta$ and $\mathbf{VS}^{(1-\alpha)} / \beta$. Therefore only three of the $9$ are "proper biplots": namely a combination of any subplot from above with the one directly below.
[Whatever combination one uses, it might be necessary to scale arrows by some arbitrary constant factor so that both arrows and data points appear roughly on the same scale.]
Using loadings, i.e. $\mathbf{VS}/\sqrt{n-1}$, for arrows has a large benefit in that they have useful interpretations (see also here about loadings). Length of the loading arrows approximates the standard deviation of original variables (squared length approximates variance), scalar products between any two arrows approximate the covariance between them, and cosines of the angles between arrows approximate correlations between original variables. To make a "proper biplot", one should choose $\mathbf U\sqrt{n-1}$, i.e. standardized PCs, for data points. Gabriel (1971) calls this "PCA biplot" and writes that
This [particular choice] is likely to provide a most useful graphical aid in interpreting multivariate matrices of observations, provided, of course, that these can be adequately approximated at rank two.
Using $\mathbf{US}$ and $\mathbf{V}$ allows a nice interpretation: arrows are projections of the original basis vectors onto the PC plane, see this illustration by @hxd1011.
One can even opt to plot raw PCs $\mathbf {US}$ together with loadings. This is an "improper biplot", but was e.g. done by @vqv on the most elegant biplot I have ever seen: Visualizing a million, PCA edition -- it shows PCA of the wine dataset.
The figure you posted (default outcome of R biplot
function) is a "proper biplot" with $\mathbf U$ and $\mathbf{VS}$. The function scales two subplots such that they span the same area. Unfortunately, the biplot
function makes a weird choice of scaling all arrows down by a factor of $0.8$ and displaying the text labels where the arrow endpoints should have been. (Also, biplot
does not get the scaling correctly and in fact ends up plotting scores with $n/(n-1)$ sum of squares, instead of $1$. See this detailed investigation by @AntoniParellada: Arrows of underlying variables in PCA biplot in R.)
PCA on correlation matrix
If we further assume that the data matrix $\mathbf X$ has been standardized so that column standard deviations are all equal to $1$, then we are performing PCA on the correlation matrix. Here is how the same figure looks like:
![Fisher Iris biplots, PCA on correlations](https://i.stack.imgur.com/rMf6y.png)
Here the loadings are even more attractive, because (in addition to the above mentioned properties), they give exactly (and not approximately) correlation coefficients between original variables and PCs. Correlations are all smaller than $1$ and loadings arrows have to be inside a "correlation circle" of radius $R=1$, which is sometimes drawn on a biplot as well (I plotted it on the corresponding subplot above). Note that the biplot by @vqv (linked above) was done for a PCA on correlation matrix, and also sports a correlation circle.
Further reading:
Best Answer
PCA tries to project your data onto a new set of dimensions where the variances in your data are captured such that you can classify/cluster them visually or by using a hopefully simple algorithm.
The variance plot tells you how much the new set of dimensions capture variances in decreasing order. Biplot is the projection of your data on the first two principal components (where the variances are the highest).