Boosting – Why Adaboost Requires a Weak Base Classifier

adaboostboosting

In Boosting/Adaboost, why the base classifier must be weak classifier?

Best Answer

It is not required to have a "weak classifier" for base classifier, in theory you can even choose neural network as base classifier. However, weak classifier is recommended to avoid overfitting and better control of the model complexity.

The reason is that boosting will increase the model "variance" step by step over iterations. If the base classifier has high variance (say neural network), we may jump to far from one step to another step over iterations. The result would be it is very hard to control the model complexity. In the end, we may have a overfitting problem.

On the other hand, if we chose a simple / trivial base, such as decision stump, we can use number of iterations to control the model complexity easily, and if we increase number of iterations, we still will have a sufficient complex model.

I Would suggest you to read my other question and answers, there are many visualizations in there.

Is a decision stump a linear model?

How does linear base leaner works in boosting? And how it works in xgboost library?

Boosting: why is the learning rate called a regularization parameter?