Solved – BIC selection yields much smaller model than AIC – can I use the likelihood ratio test to compare

aicbicdeviancelikelihood-ratiomodel selection

I'm trying to model the data (not make predictions) and am NOT using lasso for this, just want to know if my plan is somewhat reasonable here:

I'm modelling for a "yes/no" response variable, so I used logistic regression and stepwiseAIC for variable selection. The results gives me 13 parameters: 8 covariates with 5 interaction terms (several parameters are not significant on their own but have a significant interaction).

When I instead used stepwise based on BIC criteria, I only got two covariates and their interaction. Much simpler of course, but the deviance increased quite a bit. Since all the parameters in the small model were also in the large one, I considered the small one to be nested, so I did the deviance test (likelihood ratio test) and it gave me a p-value of nearly 0, indicating that the larger model is better.

Am I doing it right?

Best Answer

Using variable selection procedures like minimizing AIC or BIC impacts p-values of subsequent hypothesis tests (as well as impacting other things such as bias of parameter estimates and standard errors). Testing a hypothesis based on two different model selection procedures doesn't have the nominal properties; finding statistical significance for a difference of the two models is neither surprising nor necessarily informative about the relative worth of the two models.

BIC and AIC are based on different assumptions about the situation; the two aren't consistent with each other -- if $n>7$ then using BIC to select a model will always penalize larger models more harshly than AIC.

[If you use AIC to do 1 variable-at-a-time stepwise selection, it's equivalent to doing ordinary stepwise model selection based on a significance level of 15.7%; BIC would correspond to reducing the significance level with larger sample size.]

If you're going to then use hypothesis testing to choose between them, you're essentially falling back on p-values for variable selection. AIC works out "better" one way; BIC a different way, and the likelihood ratio test a different way a again.

I don't think there's a good argument for the procedure you have adopted.