As I've heard of the AdaBoost classifier repeatedly mentioned at work, I wanted to get a better feel for how it works and when one might want to use it. I've gone ahead and read a number of papers and tutorials on it which I found on Google, but there are aspects of the classifier which I'm still having trouble understanding:
-
Most tutorials I've seen speak of AdaBoost as finding the best weighted combination of many classifiers. This makes sense to me. What does not make sense are implementations (i.e. MALLET) where AdaBoost seems to only accept one weak learner. How does this make any sense? If there's only one classifier provided to AdaBoost, shouldn't it just return back that same classifier with a weight of 1? How does it produce new classifiers from the first classifier?
-
When would one actually want to use AdaBoost? I've read that it's supposed to be one of the best out-of-the-box classifiers, but when I try boosting a MaxEnt classifier I was getting f-scores of 70%+ with, AdaBoost murders it and gives me f-scores of something like 15% with very high recall and very low precision instead. So now I'm confused. When would I ever want to use AdaBoost? I'm looking for more of an intuitive rather than strictly statistical answer, if possible.
Best Answer
Adaboost can use multiple instances of the same classifier with different parameters. Thus, a previously linear classifier can be combined into nonlinear classifiers. Or, as the AdaBoost people like to put it, multiple weak learners can make one strong learner. A nice picture can be found here, on the bottom.
Basically, it goes as with any other learning algorithm: on some datasets it works, on some it doesn't. There sure are datasets out there, where it excels. And maybe you haven't chosen the right weak learner yet. Did you try logistic regression? Did you visualize how the decision boundaries evolve during adding of learners? Maybe you can tell what is going wrong.