I'm reading this interesting paper on application of ICA to gene expression data.
The authors write:
[T]here is no requirement for PCA components to be statistically independent.
That is true, but the PCs are orthogonal, are they not?
I am a bit fuzzy as to what is the relationship between statistical inedpendence and orthogonality or linear independence.
It is worth noting that while ICA also provides a linear decomposition of the data matrix, the requirement of statistical independence implies that the data covariance matrix is decorrelated in a non-linear fashion, in contrast to PCA where the decorrelation is performed linearly.
I don't understand that. How does lack of linearity follow from statistical independence?
Question: how does statistical independence of components in ICA relate to linear independence of components in PCA?
Best Answer
This is likely to be a duplicate of some older question(s), but I will briefly answer is nevertheless.
For a non-technical explanation, I find quite helpful this figure from the Wikipedia article on Correlation and dependence:
The numbers above each scatter plot show correlation coefficients between X and Y. Look at the last row: on each scatter plot the correlation is zero, i.e. X and Y are "linearly independent". However they are obviously not statistically independent: if you know the value of X, you can narrow down the possible values of Y. If X and Y were independent, it would mean that knowing X tells you nothing about Y.
The purpose of ICA is to try to find independent components. In PCA you only get uncorrelated ("orthogonal") components; correlation between them is zero but they can very well be statistically dependent.