dimensionality reductionpca
Suppose I have a 2D point dataset and I want to detect the directions of all the local maxima's of variance in the data, for example:
PCA does not help in this situation as it is an orthogonal decomposition and therefore cannot detect both the lines I indicated in blue, rather its output may look like the one shown by green lines.
Please recommend any technique which might be suitable for this purpose. Thanks.
Are by chance your scales different?
How do you principal components look like?
Most likely you did not perform appropriate normalization beforehand.
A side effect of doing PCA is that it also will apply z-score normalization to your data set. This gives all three dimensions roughly equal weight. And as you can imagine, using all three attributes and not just one will usually result in better performance.
This may be only a partial answer because I don't think the plot that you expect is really what is in the data, especially the "parallelity and continuity" of the intermediate signals. I will speculate on reasons for that below.
But I think I was able to get to what you look for in terms of the four basal signals A1, A5, E1, E5. Namely that they lie on the edge of the embedded manifold, that opposite signals lie more or less diametrical to each other (that is A1, E5 and E1, A5 respectively) and that neighbouring signals are preserved (so A1, A5 and E1, E5 respectively).
Generally, I think standard (i.e. with an input weight matrix consiting of only 1's) MDS doesn't really give you what you want because you are actually looking for nonlinear dimension reduction that has some localization feature, namely that larger "distances" should not be preserved but local distances should. There are a number of algorithms that do that. A rather popular one that is often used for cases like yours is called t-SNE for t-Distributed Stochastic Neighbor Embedding. On the linked homepage you will find quite some information.
I calculated the t-SNE embedding for your kappa distance. Note that I use a high perplexity to force a shaping that looks like the one you aim for. It can be obtained by
set.seed(1)
library(tsne)
tsne.coor1 <- tsne(res,perplexity=25)
rownames(tsne.coor1) <- c("A1","A2","A3","A4","A5",
"B1","B2","B3","B4","B5",
"C1","C2","C3","C4","C5",
"D1","D2","D3","D4","D5",
"E1","E2","E3","E4","E5")
plot(tsne.coor1[,1], tsne.coor1[,2], type="n", xlab="", ylab="")
text(tsne.coor1[,1], tsne.coor1[,2],
labels=row.names(tsne.coor1), cex=0.8)
abline(h=0,v=0,col="gray75")
and looks like this
The basic signals are labeled in red. As you can see, the similarity structure as captured by your preprocessing and distance measure does not suggest that the intermediate signals are actually at all as you expect them to be. For example, the path from A1 to E1 has intermediate signals E4, D5, A3 and not B1 through D1. But this is what your data tell you! So, the similarity between the signals that are within the convex hull of the embedded manifold suggests that the clear-cut pattern is not preserved.
There are two obvious explanations:
[\Edit] (thanks @ttnphns!)
To investigate the last point, I tried other measures of similarity for binary matrices. For this high perplexity it led to no discernably different results in the shape, but it did so in the arrangements (I used a gravity model similarity and an asymmetric binary similarity, the one returned by dist(x,method="binary")). For lower perplexity the effects on the visualisation the kappa distance is small, but for the other distances it is not. To illustrate for the asymmetric distances the results are:
res <- dist(M,method="binary")
set.seed(1)
tsne.coor2 <- tsne(res,perplexity=25)
tsne.coor3 <- tsne(res,perplexity=3)
rownames(tsne.coor2)<-rownames(tsne.coor3) <- c("A1","A2","A3","A4","A5",
"B1","B2","B3","B4","B5",
"C1","C2","C3","C4","C5",
"D1","D2","D3","D4","D5",
"E1","E2","E3","E4","E5")
par(mfrow=c(1,2))
plot(tsne.coor2[,1], tsne.coor2[,2], type="n", xlab="", ylab="")
text(tsne.coor2[,1], tsne.coor2[,2],
labels=row.names(tsne.coor3), cex=0.8)
abline(h=0,v=0,col="gray75")
plot(tsne.coor3[,1], tsne.coor3[,2], type="n", xlab="", ylab="")
text(tsne.coor3[,1], tsne.coor3[,2],
labels=row.names(tsne.coor2), cex=0.8)
abline(h=0,v=0,col="gray75")
and here's the results
So, for high perplexity the difference in the shape of the projection of the signals is rather similar for both distance measures. For lower perplexity, the results change. Note that while it is looking different than what you originally intended when using the asymmetric binary distance, the transition from basis signal through the other states seems to be better preserved in the low perplexity plot, particularly column-wise! . This makes the speculation 2 of using kappa being a not very suitable distance measure more likely. You may try the distances dist(x, method="binary") or cluster::daisy(x, metric= "gower")) (which is the dice coefficient, I think).
[\end Edit]
Note the t-SNE has random initializations, so it might look a bit different at your end --- not sure whether set.seed has an effect in the R implementation. The random initialization is actually something that might get you closer to what you need anyway. As the authors put it:
In contrast to, e.g., PCA, t-SNE has a non-convex objective function. The objective function is minimized using a gradient descent optimization that is initiated randomly. As a result, it is possible that different runs give you different solutions. Notice that it is perfectly fine to run t-SNE a number of times (with the same data and parameters), and to select the visualization with the lowest value of the objective function as your final visualization.
You may play around with the above mentioned distances and some of the parameters, especially perplexity, to perhaps get you closer to what you need into one direction or the other. Hope this helps as a first start.
Best Answer
Independent Component Analysis should be able to provide you with s good solution. It is able to decompose non-orthogonal components (like in your case) by assuming that your measurements result from a mixture of statistically independent variables.
There are plenty of good tutorials in Internet, and quiet a few freely available implementations to try out (for example in scikit or MDP).
When does ICA not work?
As other algorithms, ICA is optimal when the assumptions for which it was derived apply. Concretely,
ICA returns an estimation of the mixing matrix and the independent components.
When your sources are Gaussian then ICA cannot find the components. Imagine you have two independent components, $x_{1}$ and $x_{2}$, which are $N(0,I)$. Then, $$ p(x_{1}, x_{2}) = p(x_{1})p(x_{2}) = \frac{1}{2\pi}\exp \left( -\frac{x_{1}^{2}+x_{2}^{2}}{2} \right) = \frac{1}{2\pi}\exp -\frac{||\mathbf{x}||^{2}}{2} $$
where $||.||$. is the norm of the two dimensional vector. If they are mixed with an orthogonal transformation (for example a rotation $R$), we have, $||R\mathbf{x}|| = ||\mathbf{x}||$, which means that the probability distribution does not change under the rotation. Hence, ICA cannot find the mixing matrix from the data.