Solved – Conditional Inference Random Forest

partyrrandom forest

I use cforest, a function of the R package Party, to realize a conditional inference random forest. However I don't understand how this function compute the predict variable for a regression problem.

Could you explain me how this work please ?
Can I get in output the median of the mean final tree nodes ?
Can I get the more significant explanatory variables with this function ?

Best Answer

The cforest function constructs a forest of conditional inference trees, see help("cforest", package = "party") for further details and references. In short, the conditional inference trees (Hothorn et al. 2006a) are grown "in the usual way" on bootstrap samples or subsamples with only a subset of variables available for splitting in each node. For predictions a suitably weighted mean of the observed responses is constructed (Hothorn et al. 2006b). You could also use the forest to get other types of aggregations such as medians or other quantiles. However, this is not provided by default.

While conditional inference trees employ significance tests for determining the split variables and split points, there are no classical significances for the explanatory variables. However, various flavors of variable importance measures are available (Strobl et al. 2007, 2008).

References:

Torsten Hothorn, Kurt Hornik, Achim Zeileis (2006a). Unbiased Recursive Partitioning: A Conditional Inference Framework. Journal of Computational and Graphical Statistics, 15(3), 651-674.
Torsten Hothorn, Peter Bühlmann, Sandrine Dudoit, Annette Molinaro, Mark Van Der Laan (2006b). Survival Ensembles. Biostatistics, 7(3), 355-373.
Carolin Strobl, Anne-Laure Boulesteix, Achim Zeileis, Torsten Hothorn (2007). Bias in Random Forest Variable Importance Measures: Illustrations, Sources and a Solution. BMC Bioinformatics, 8(25).
Carolin Strobl, Anne-Laure Boulesteix, Thomas Kneib, Thomas Augustin, Achim Zeileis (2008). Conditional Variable Importance for Random Forests. BMC Bioinformatics, 9(307).

Related Solutions

Solved – Does party package in R provide out-of-bag estimates of error for Random Forest models

The caret package has a method for getting that. You can use train as the interface. For example:

> mod1 <- train(Species ~ ., 
+               data = iris, 
+               method = "cforest", 
+               tuneGrid = data.frame(.mtry = 2),
+               trControl = trainControl(method = "oob"))
> mod1
150 samples
  4 predictors
  3 classes: 'setosa', 'versicolor', 'virginica' 

No pre-processing
Resampling: 

Summary of sample sizes:  

Resampling results

  Accuracy  Kappa
  0.967     0.95 

Tuning parameter 'mtry' was held constant at a value of 2

Alternatively, there is an internal function that can be used if you want to go straight to cforest but you have to call it using the namespace operator:

> mod2 <- cforest(Species ~ ., data = iris,
+                 controls = cforest_unbiased(mtry = 2))
> caret:::cforestStats(mod2)
 Accuracy     Kappa 
0.9666667 0.9500000

HTH,

Max

Solved – Pruning Conditional Inference Trees

In those situations where p-values work well (e.g., in small to moderately sized samples), the pre-pruning strategy employed in conditional inference trees typically works well. (Pre-pruning means you stop growing the tree when some condition is fulfilled - rather than first growing a larger tree and then pruning it back.)

However, it is, of course possible, to treat the significance level as a tuning parameter and choose its value based on cross-validation or out-of-bag performance etc. This can be useful for large datasets where essentially all p-values are significant in order to avoid overfitting. The strategy is implemented in the caret package as train(..., method = "ctree").

Finally, it would be conceivable to first grow a large tree (with low mincriterion) and then prune it based on information criteria or cost-complexity etc. But I think it's not readily available for conditional inference trees in an R package at the moment. If you're doing binary classification, you might consider a logit glmtree() which offers AIC- and BIC-based post-pruning. This is in the partykit package which also contains the recommended re-implementation of ctree().

Best Answer

Related Solutions

Solved – Does party package in R provide out-of-bag estimates of error for Random Forest models

Solved – Pruning Conditional Inference Trees

Related Question