Solved – setting max_features to none in random forest

cartmachine learningrandom forestscikit learn

For sklearn's Random forest classification module, setting max_features to none takes into consideration all the n features for building each tree. In this case, how is it different from applying bagging to simple CART. Also isn't feature sampling the USP of random forest.

Best Answer

In bagging, the only parameter we tune is the number of trees. In Random Forest, we tune both the number of trees and the number of input variables $m \leq p$ considered for splitting each node. If $m = p$, we are bagging. Typically we choose $m \ll p$.

I assume by "USP" you mean "unique selling proposition?" If so, yes, the ability to consider a subset of features at each node is the main feature differentiating Random Forest from Bagging. The fact that one could set $m=p$ doesn't mean one should, and it doesn't invalidate the random feature subsetting as a "selling point" of Random Forest.

One typically uses $\sqrt{p}$ features, at least for classification problems.

Related Solutions

Solved – Classification and regression trees (cart)

It looks to me like classregtree is just building a tree, not using any of these methods, all of which are supplementary to tree building. That is, classregtree is implementing the methods described in Breiman et al., per the reference given in the documentation. It builds a tree and then (by default) prunes it.

Solved – Random forest: which features led to a certain prediction

In the canonical implementation of random forest (R's randomForest package), there is a way to produce a local importance matrix that tells you which feature(s) have contributed to the model's prediction.

library(randomForest)
set.seed(71)
iris.rf <- randomForest(Species ~ ., data=iris, importance=TRUE,
                        localImp=TRUE,
                        proximity=TRUE)

locImp = iris.rf$localImportance
dim(locImp)
[1]   4 150

The rows of locImp are the features, columns the observations. So locImp[,1] gives,

Sepal.Length  Sepal.Width Petal.Length  Petal.Width 
  0.02564103   0.01025641   0.32307692   0.37435897

That says Petal.Width has the most weight in predicting setosa on the first observation.

Best Answer

Related Solutions

Solved – Classification and regression trees (cart)

Solved – Random forest: which features led to a certain prediction

Related Question