independenceindependent component analysisnormal distributionpca
$X = AS$ where $A$ is my mixing matrix and each column of $S$ represents my sources. $X$ is the data I observe.
If the columns of $S$ are independent and Gaussian, will the components of PCA be extremely similar to that of ICA? Is this the only requirement for the two methods to coincide?
Can someone provide an example of this being true when the $cov(X)$ isn't diagonal?
The answer to your question is given by the mapping at the bottom of Page 6 of the original Laplacian Eigenmaps paper :
$x_i \rightarrow (f_1(i), \dots, f_m(i))$
So for instance, the embedding of a point $x_5$ in, say, the top 2 "components" is given by $(f_1(5), f_2(5))$ where $f_1$ and $f_2$ are the eigenvectors corresponding to the two smallest non-zero eigenvalues from the generalized eigenvalue problem $L f = \lambda D f$.
Note that unlike in PCA, it is not straightforward to obtain an out-of-sample embedding. In other words, you can obtain the embedding of a point that was already considered when computing $L$, but not (easily) if it is a new point. If you're interested in doing the latter, look up this paper.
This is likely to be a duplicate of some older question(s), but I will briefly answer is nevertheless.
For a non-technical explanation, I find quite helpful this figure from the Wikipedia article on Correlation and dependence:
The numbers above each scatter plot show correlation coefficients between X and Y. Look at the last row: on each scatter plot the correlation is zero, i.e. X and Y are "linearly independent". However they are obviously not statistically independent: if you know the value of X, you can narrow down the possible values of Y. If X and Y were independent, it would mean that knowing X tells you nothing about Y.
The purpose of ICA is to try to find independent components. In PCA you only get uncorrelated ("orthogonal") components; correlation between them is zero but they can very well be statistically dependent.
Best Answer
PCA will be equivalent to ICA if all the correlations in the data are limited to second-order correlations and no higher-order correlations are found. Said another way, when the covariance matrix of the data can explain all the redundancies present in the data, ICA and PCA should return same components.