Solved – Pruning Conditional Inference Trees

classificationparty

I am trying to build a prediction model using classification trees. While I tried the "rpart" package, the results were not entirely satisfactory. Hence, I thought of exploring conditional inference trees as well ("party" package in R)

Now, under the documentation for "ctree" function they have mentioned the following – "For example, when mincriterion = 0.95, the p-value must be smaller than 0.05 in order to split this node. This statistical approach ensures that the right sized tree is grown and no form of pruning or cross-validation or whatsoever is needed"

However, with the default mincriterion value of 0.95, I end up with just 1 split. Would it make sense, if I vary the mincriterion value (say from 0.95 to 0.90), cross validate the resulting models and pick the one with the lowest CV error?

If yes, is there a function within the party package which can help me do this? (roughly analogous to a "plotcp/printcp" function that we have in rpart)

Thanks!

Best Answer

In those situations where p-values work well (e.g., in small to moderately sized samples), the pre-pruning strategy employed in conditional inference trees typically works well. (Pre-pruning means you stop growing the tree when some condition is fulfilled - rather than first growing a larger tree and then pruning it back.)

However, it is, of course possible, to treat the significance level as a tuning parameter and choose its value based on cross-validation or out-of-bag performance etc. This can be useful for large datasets where essentially all p-values are significant in order to avoid overfitting. The strategy is implemented in the caret package as train(..., method = "ctree").

Finally, it would be conceivable to first grow a large tree (with low mincriterion) and then prune it based on information criteria or cost-complexity etc. But I think it's not readily available for conditional inference trees in an R package at the moment. If you're doing binary classification, you might consider a logit glmtree() which offers AIC- and BIC-based post-pruning. This is in the partykit package which also contains the recommended re-implementation of ctree().

Related Solutions

Solved – How do Conditional Inference Trees do binary classification

By default ctree() uses a certain generalized type of association test to determine the split variable to be used. You can easily replicate this with the coin package that implements the same class of association tests for direct use:

library("coin")
independence_test(t20.a ~ p20, teststat = "quad")
##         Asymptotic General Independence Test
## 
## data:  t20.a by p20
## chi-squared = 14.286, df = 1, p-value = 0.0001571

Once the variable is selected, the optimal split point is selected by maximizing the corresponding two-sample association test. More details and further references are given in this discussion: What is the test statistics used for a conditional inference regression tree?

Optionally, it is possible to also use the maximally-selected statistic for the split variable already. By default this is not used because it is somewhat less powerful for monotonic associations and computationally more intensive. However, as your pat363 example illustrates: It can miss certain completely balanced patters (like the XOR or chessboard example). However, any splitting strategy you use will perform better on some patterns and worse on others...

Solved – Conditional Inference Random Forest

The cforest function constructs a forest of conditional inference trees, see help("cforest", package = "party") for further details and references. In short, the conditional inference trees (Hothorn et al. 2006a) are grown "in the usual way" on bootstrap samples or subsamples with only a subset of variables available for splitting in each node. For predictions a suitably weighted mean of the observed responses is constructed (Hothorn et al. 2006b). You could also use the forest to get other types of aggregations such as medians or other quantiles. However, this is not provided by default.

While conditional inference trees employ significance tests for determining the split variables and split points, there are no classical significances for the explanatory variables. However, various flavors of variable importance measures are available (Strobl et al. 2007, 2008).

References:

Torsten Hothorn, Kurt Hornik, Achim Zeileis (2006a). Unbiased Recursive Partitioning: A Conditional Inference Framework. Journal of Computational and Graphical Statistics, 15(3), 651-674.
Torsten Hothorn, Peter Bühlmann, Sandrine Dudoit, Annette Molinaro, Mark Van Der Laan (2006b). Survival Ensembles. Biostatistics, 7(3), 355-373.
Carolin Strobl, Anne-Laure Boulesteix, Achim Zeileis, Torsten Hothorn (2007). Bias in Random Forest Variable Importance Measures: Illustrations, Sources and a Solution. BMC Bioinformatics, 8(25).
Carolin Strobl, Anne-Laure Boulesteix, Thomas Kneib, Thomas Augustin, Achim Zeileis (2008). Conditional Variable Importance for Random Forests. BMC Bioinformatics, 9(307).

Best Answer

Related Solutions

Solved – How do Conditional Inference Trees do binary classification

Solved – Conditional Inference Random Forest

Related Question