Model Selection – Is AIC Suitable for VAR Model Given Sample Size Variations with Lags?

aicmodel selectionvector-autoregression

Given that the sample size of a VAR (or a similar model: VARX, SVAR etc.) reduces by $1$ for each extra dependent-variable lag that I introduce (since we need to drop the empty rows, or NaN's in programming speak), is the AIC an appropriate measure for model selection?

To be more specific, in this response to a previous question of mine, it was kindly pointed out that the dependent variable of the model needs to be the same across the models that are being compared for the AIC to be valid. Well, although it is true that the dependent variable stays the the same, the data does change because of the reduced sample size. Does this matter? (It is clear that this likely does not make a difference in the large-sample limit, but what about in a small sample?)

Finally, are any of the points above good reasons to consider using the AICc over the AIC for model selection?

Best Answer

No, if your number of records changes, AIC is not appropriate. There's multiple idea for fixing this:

  1. deal with the missing informaiton to have the same number of records so that missing lags are no longer an issue (e.g. multiple imputation, single imputation of some form perhaps with a missing flag for those records, etc. or use an algorithm that intrinsically can deal with missingness like xgboost)
  2. use the smallest subset you can use all methods for (if you don't want to do (1) and still want to compare using AIC)
  3. do not use AIC (e.g. use cross-validation or some past-vs.-future split if this is a time series problem, i.e. not evaluating on the training data).

AIC vs. AICc is sort of irrelavant for this topic. AICc is meant to be better for small sample situations (although there seems to be disagreement on whether it truly is).

Finally, there's the question whether model selection is a sensible thing to do (esp. if the models are so close to each other that it's not really possible to be sure whether one is truly better than the other) or whether you want model averaging instead.

Related Question