Solved – XGBoost – Weights in classifcation tree 0 or 1

boostingclassificationweights

I've read the paper on the XGBoost algorithm and one thing is not quite clear to me. The regularization term is defined as below where $w_{j}$ corresponds to the weight in end node $j$. For the regression case this would be the predicted value in the end node and for the classification case this would be either a $0$ or a $1$ I would suspect. Then the regularization term seems intuitive to me in the case of regression for penalizing large predictions however in the classification case it wouldn't add much as the $w_{j}$'s will always be a $0$ or a $1$ right?

enter image description here

Best Answer

XGBoost does not produce a decision tree/trees with leaf node values of 0 or 1.

Instead, it uses multiple regression trees with continuous value "weights" on its leaf nodes e.g. in the range 0 ~ 1. Hence, regularization is applied in the same manner as for similar regression based loss functions.

The tree boosting algorithm applies the trees additively i.e. sums the weights for all trees to arrive at the final value for the given input.

This continuous valued "score" needs to be interpreted by you as a class label by applying a cutoff, e.g. predicted class = 1 if $\hat y > 0.5 $, if cutoff = 0.5.