Starting from an "a priori" set of models based on my knowledge about potential relations between my dependent variable and the independent variables considered, I use AIC to find best models. More specifically I calculate Akaike weights then Evidence Ratio (ER) and consider that models with a ER < 2 are equally likely.
But the same problem remain each time I do that. I selected the best models from a set of them, but I don't know if those models are efficient to predict (or at least represent) my data.
I can have selected the best element(s) of the list of the worst models.
I do not use $R^2$ in model selection because of the fact that including more variables generally increase $R^2$ value.
But ! When the selection by AIC is done and I can consider that models with a ER < 2 are equally likely. Do you find it is correct to calculate $R^2$ or pseudo-$R^2$ for the best "set of models" in order to have an idea of the representativeness of those models and use this value to select the more efficient model?
I would be glad to hear your opinions about this!
Note: Thanks for suggestions about cross-validation, I will try this. Unfortunatly, I do not have an external dataset to test my models.
Best Answer
I agree cross validation + good measure of error should be the way to go. Maybe you can too use ridge and lasso instead of AIC to extract strong predictors in order to get a robust model.