Solved – Tuning adaboost

adaboostboostingmachine learning

The boosting algorithm Adaboost (when using a tree) has three core parameters:

  • number of weak learners to train
  • learning rate
  • max nb of splits (depth of tree)

What are good practices, perhaps proven empirically, for finding the appropriate value for these parameters?

One option I can think of is to do a grid search via cross validation and get out the validation accuracy (1-generalization error).

Best Answer

Number of weak learners

Train many, many weak learners. Then look at a test-error vs. number of estimators curve to find the optimal number.

Learning rate

Smaller is better, but you will have to fit more weak learners the smaller the learning rate. During initial modeling and EDA, set the learning rate rather large (0.01 for example). Then when fitting your final model, set it very small (0.0001 for example), fit many, many weak learners, and run the model over night.

Maximum number of splits

There is no a-priori best answer. You need to grid search this one.

Related Question