I've got a problem choosing the right model. I have a model with various variables (covariables and dummy variables). I was trying to find the best size for this model, so I first started by comparing different models with AIC. From this it followed, that the minimum AIC was reached when allowing all variables to stay in the model (with the whole bunch to interact with all dummies). When I compute the summary of the model, all effects are absolutely not significant and the standard errors are very high. I was a bit confused, when comparing the "best" (on AIC) model with a smaller model with any interaction. The smaller model had small standard errors and nice p-values… But the AIC is higher compared to the big model. What might be the problem? Overspecification?
I really need help in this, because I have absolutely no idea how to handle this!
Thanks a lot
Best Answer
The AIC and standard error measure different things, and if you are trying to minimize standard error, a cross-validation approach may be better to use. Another alternative is the Bayesian information criterion (BIC), which is more parsimonious than the AIC.
Also, here's a good article comparing the relations between various evaluation metric for supervised machine learning: Data mining in metric space: an empirical analysis of supervised learning performance criteria.