Solved – Stepwise AIC – Does there exist controversy surrounding this topic

aicarimaforecastingpredictive-modelsstepwise regression

I've read countless posts on this site that are incredibly against the use of stepwise selection of variables using any sort of criterion whether it be p-values based, AIC, BIC, etc.

I understand why these procedures are in general, quite poor for the selection of variables. gung's probably famous post here clearly illustrates why; ultimately we are verifying a hypothesis on the same dataset we used to come up with the hypothesis, which is just data dredging. Furthermore, p-values are affected by quantities such as collinearity and outliers, which heavily skew results, etc.

However, I've been studying time series forecasting quite a bit lately and have come across Hyndman's well respected textbook in which he mentions here the use of stepwise selection to find the optimal order of ARIMA models in particular. In fact, in the forecast package in R the well known algorithm known as auto.arima by default uses stepwise selection (with AIC, not p-values). He also criticizes p-value based feature selection which aligns well with multiple posts on this website.

Ultimately, we should always cross validate in some way at the end if the goal is to develop good models for forecasting/prediction. However, surely this is somewhat of a disagreement here when it comes to the procedure itself for evaluation metrics other than p-values.

Does anyone have any opinions on the use of stepwise AIC in this context, but also in general out of this context? I have been taught to believe any stepwise selection is poor, but to be honest, auto.arima(stepwise = TRUE) has been giving me better out of sample results than auto.arima(stepwise = FALSE) but perhaps this is just coincidence.

Best Answer

There are a few different issues here.

  • Probably the main issue is that model selection (whether using p-values or AICs, stepwise or all-subsets or something else) is primarily problematic for inference (e.g. getting p-values with appropriate type I error, confidence intervals with appropriate coverage). For prediction, model selection can indeed pick a better spot on the bias-variance tradeoff axis and improve out-of-sample error.
  • For some classes of models, AIC is asymptotically equivalent to leave-one-out CV error [see e.g. http://www.petrkeil.com/?p=836 ], so using AIC as a computationally efficient proxy for CV is reasonable.
  • Stepwise selection is often dominated by other model selection (or averaging) methods (all-subsets if computationally feasible, or shrinkage methods). But it's simple and easy to implement, and if the answer is clear enough (some parameters corresponding to strong signals, others weak, few intermediate), then it will give reasonable results. Again, there's a big difference between inference and prediction. For example if you have a couple of strongly correlated predictors, picking the incorrect one (from a "truth"/causal point of view) is a big problem for inference, but picking the one that happens to give you the best AIC is a reasonable strategy for prediction (albeit one that will fail if you try to forecast a situation where the correlation of the predictors changes ...)

Bottom line: for moderately sized data with a reasonable signal-to-noise ratio, AIC-based stepwise selection can indeed produce a defensible predictive model; see Murtaugh (2009) for an example.

Murtaugh, Paul A. "Performance of several variable‐selection methods applied to real ecological data." Ecology letters 12, no. 10 (2009): 1061-1068.

Related Question