XGBoost Optimization – XGBoost CV GridSearch vs Early Stopping

Question

I am using the XGBoost Gradient Boosting Algorithm for a sales prediction dataset. I am planning to tune the parameters regularly with CVGridSearch. Now I am wondering if it makes sense to still specify the Early Stopping Parameter if I regularly tune the algorithm. From my understanding, the Early Stopping option does not provide such an extensive cross validation than the CVGridSearch method would. Does anyone have any suggestions or recommendations from a similar implementation?

Answer 1

It makes perfect sense to use early stopping when tuning our algorithm. We are not a faced with a "GridSearch vs Early Stopping" but rather with a "GridSearch and Early Stopping" situation. :)

We can readily combine CVGridSearch with early stopping. We can go forward and pass relevant parameters in the fit function of CVGridSearch; the SO post here gives an exact worked example. Notice that we can define a cross-validation generator (i.e. a cross-validation procedure) in our CVGridSearch. Using early stopping when performing hyper-parameter tuning saves us time and allows us to explore a more diverse set of parameters. We need to be a bit careful to pull the relevant parameters from our classifier object (i.e. get the best_iteration directly from the fitted object instead of relying on the parameter grid values because we might have hit the early stopping beforehand) but aside that, everything should be fine.