Solved – RandomForest – MDS plot interpretation

classificationmultidimensional scalingrrandom forest

I used randomForest to classify 6 animal behaviours (eg. Standing, Walking, Swimming etc.) based on 8 variables (different body postures and movement).

The MDSplot in the randomForest package gives me this output and I have problems in interpreting the result. I did a PCA on the same data and got a nice seperation between all the classes in PC1 and PC2 already, but here Dim1 and Dim2 seem to just seperate 3 behaviours. Does this mean that these three behaviours are the more dissimilar than all other behaviours (so MDS tries to find the greatest dissimilarity between variables, but not necessarily all variables in the first step)? What does the positioning of the three clusters (as e.g in Dim1 and Dim2) indicate? Since I'm rather new to R I also have problems plotting a legend to this plot (however I have an idea what the different colours mean), but maybe somebody could help? Thanks a lot!!

RandomForest MDSplot of 6 different animal behaviours

I add a plot made with the ClassCenter function in RandomForest. This function also uses the proximity matrix (same as in the MDS Plot) for plotting the prototypes. But just from looking at the datapoints for the six different behaviours, I can't understand why the proximity matrix would plot my prototypes as it does. I also tried the classcenter function with the iris data and it works. But it seems like it doesn't work for my data…

Here is the code I used for this plot

be.rf <- randomForest(Behaviour~., data=be, prox=TRUE, importance=TRUE)
class1 <- classCenter(be[,-1], be[,1], be.rf$prox)
Protoplot <- plot(be[,4], be[,7], pch=21, xlab=names(be)[4], ylab=names(be)[7], bg=c("red", "green", "blue", "yellow", "turquoise", "orange") [as.numeric(factor(be$Behaviour))])
points(class1[,4], class1[,7], pch=21, cex=2, bg=c("red", "green", "blue", "yellow", "turquoise", "orange"))

My class column is the first one, followed by 8 predictors. I plotted two of the best predictor variables as x and y.

RandomForest ClassCenter plot for 6 different animal behaviours

Best Answer

The function MDSplot plots the (PCA of) the proximity matrix. From the documentation for randomForest, the proximity matrix is:

A matrix of proximity measures among the input (based on the frequency that pairs of data points are in the same terminal nodes).

Based on this description, we can guess at what the different plots mean. You seem to have specified k=4, which means a decomposition of the proximity matrix in 4 components. For each entry (i,j) in this matrix of plots, what is plotted is the PCA decomposition along dimension i versus the PCA decomposition along dimension j.

I did a PCA on the same data and got a nice seperation between all the classes in PC1 and PC2 already, but here Dim1 and Dim2 seem to just seperate 3 behaviours. Does this mean that these three behaviours are the more dissimilar than all other behaviours (so MDS tries to find the greatest dissimilarity between variables, but not necessarily all variables in the first step)?

MDS can only base its analysis on the output of your randomForest. If you're expecting a better separation, then you might want to check the classification performance of your randomForest. Another thing to keep in mind is that your PCA is mapping from 9-dimensional data to 2 dimensions, but the MDS is mapping from an NxN-dimensional proximity matrix to 2 dimensions, where N is the number of datapoints.

What does the positioning of the three clusters (as e.g in Dim1 and Dim2) indicate?

It just tells you how far apart (relatively) these clusters are from each other. It's a visualisation aid, so I wouldn't over-interpret it.

Since I'm rather new to R I also have problems plotting a legend to this plot (however I have an idea what the different colours mean), but maybe somebody could help?

The way R works, there's no way to plot legend after-the-fact (unlike in say Matlab, where this information is stored inside the figure object). However, looking at the code for MDSplot, we see that relevant code block is:

palette <- if (require(RColorBrewer) && nlevs < 12) brewer.pal(nlevs, "Set1")

...

plot(rf.mds$points, col = palette[as.numeric(fac)], pch = pch, ...)

So the colours will be taken from that palette, and mapped to the levels (behaviours) in whichever order you've given them. So if you want to plot a legend:

legend(x,y,levels(fac),col=brewer.pal(nlevs, 'Set1'), pch=pch)

would probably work.

Related Solutions

R Multivariate Analysis – How to Interpret MDS Factor Plot

I'm answering my own question for 2 reason:1) I want to be clear what I've understood is correct or not. 2) If somebody is looking for the same reason he/she should find it here.I hardly found book that gives a clear explanation of interpretation of MDS biplots. I'll also give few references where people can read more about interpretation of MDS ploting to better understand it.

This answer is divided in few parts: Part 1: The axis of the biplot are the principal components. x-axis has the PC 1, which reflect the max variance in the dataset. y-axis has the PC 2, whichreflect 2nd most variance. E.g. in my example x-axis represent 72% of the variance, while y-axis represent 16% of the variance in the data.

 PC1      PC2      PC3      PC4 
0.727891 0.166721 0.070320 0.003048

Part 2: The arrows reflect how the variables are loaded in each PCs. E.g. in my example "uncluttered" & "visualization" is highly negatively loaded to PC 2, hence y-axis. Similarly, "no water","fast relief" & "convinient" is highly plositively loaded to PC 2, hence x-axis.This gives us a visualization about how variables are loaded in different PCs.

                    NMDS1     NMDS2
Safe                      0.616967 -0.786989
Highly.efficacious       -0.135565  0.990768
Same.side.effect.profile  0.822707 -0.568466
Fast.Relief               0.988621 -0.150428
No.Water                  0.990893  0.134648
Convenient                0.989206  0.146534
Convincing                0.763225 -0.646133
Visually.appealing        0.154414 -0.988006
Very.novel                0.900984  0.433853
Noticeable                0.691596  0.722284
Likely.to.be.read         0.887028 -0.461715
Uncluttered               0.031498 -0.999504
Interesting               0.872584 -0.488465
Credible                  0.620556 -0.784162
Prescribe.Recommend       0.809955 -0.586492

part 3: Concept points tells us how dissimilar they are from the each other. So, in my example Concept 1 & Concept 2 are very different from rest of them. Concept 2 is both bad in terms of visual appeal as well as convenience. Whereas concept 3 & 4 are more alike. They are also good in terms of visualization as well as convenience.

Reference: 1) Greenacre, M. (2010). Biplots in Practice
           2) Everitt & Hothorn: An Introduction to Multivariate Analysis with R(Chapter 4).
           3) Hair: Multivariate Data Analysis

Solved – RandomForest MeanDecreaseAccuracy interpretation

Variable inportance(VI), is (1)training (2)predict OOB (2)permuting each variable in turns and re-predict OOB. (4) substract OOB-performance'2' OOB-performances in'3'. VI,(%mse or %class error) will only under some conditions approximate the loss of prediction performance if omitting a given variable from training. To understand when and when not, try imagine that two variables can hold some redundant and/or complimentary information. In fact, if these two variable variables are completely redundant e.g. by having a Spearman correlation = 1(non-parametric Pearson), these two variables will be interchangeable during growing the forest. Thus an equal amount of splits will rely on these two variables. The VI of both variables are the same. But if growing a new forest with one of the redundant variables omitted, the prediction performance could be almost unchanged, whereas the VI of the remaining redundant variable would double(if no other redundancies). The prediction performance would first decline, if regrowing the forest a third time omitting both redundant variables.

Variables can also be complimentary or even inter-dependent. E.g. a variable(lab-source) describing which lab produced a test result may be completely unuseful knowledge unless the test result itself(lab-result) is known also. If omitting the "lab-result" variable before training, then the 'lab-source' variable would have a lower variable importance. When a RF model essentially have captured a stong pair-wise variable interaction, VI can understate the loss of prediction performance by omitting one of the variables, as it is infact rendering another variable unuseful.

Such deviations between prediction performance and VI due to omitting/permuting variables before and after growing the forest can be used to screen for pair-wise variable redundancy and 'reliance'.

VI is mainly used for variable selection by ranking. The absolute value of VI is often regarded as arbitrary.

Best Answer

Related Solutions

R Multivariate Analysis – How to Interpret MDS Factor Plot

Solved – RandomForest MeanDecreaseAccuracy interpretation

Related Question