Solved – Is it possible that the AIC and BIC give totally different model selections

aicbicmodel selectionpoisson-regression

I'm performing a Poisson Regression model with 1 response variable and 6 covariates. Model selection using AIC results in a model with all covariates as well as 6 interaction terms. The BIC however, results in a model with only 2 covariates and no interaction terms. Is it possible that the two criteria, that look very similar, yield totally different model selections?

Best Answer

It is possible indeed. As explained at https://methodology.psu.edu/AIC-vs-BIC, "BIC penalizes model complexity more heavily. The only way they should disagree is when AIC chooses a larger model than BIC."

If your goal is to identify a good predictive model, you should use the AIC. If your goal is to identify a good explanatory model, you should use the BIC. Rob Hyndman nicely summarizes this recommendation at
https://robjhyndman.com/hyndsight/to-explain-or-predict/:

"The AIC is better suited to model selection for prediction as it is asymptotically equivalent to leave-one-out cross-validation in regression, or one-step-cross-validation in time series. On the other hand, it might be argued that the BIC is better suited to model selection for explanation, as it is consistent."

The recommendation comes from Galit Shmueli’s paper “To explain or to predict?”, Statistical Science, 25(3), 289-310 (https://projecteuclid.org/euclid.ss/1294167961).

Addendum:

There is a third type of modeling - descriptive modeling - but I don't know of any references on which of AIC or BIC is best suited for identifying an optimal descriptive model. I hope others here can chime in with their insights.