I have a time series that I'm fitting models to, using R.
I have chosen an ARIMA model based on minimising the AIC_C values.
The ETS model (ets()) was chosen based on minimising the model accuracy errors and the rolling forecast accuracy errors (in the latter, I chose the best 3 from the former, and the ets() model was competing against hw() and hw() with multiplicative seasonality, assuming that exponential models are comparable).
Now, I'd like to know which of the two best models is overall best?
How do I compare them?
And generally, if I fit another type of model, is there a "rule of thumb" for comparing the performance of different models with each other?
Thanks!
Edit: The ARIMA model's residuals fail the Shapiro test but pass the Box-Ljung test. The ets() model's residuals pass both tests.
Best Answer
If your goal is to forecast the variables, then the pseudo-out-of-sample analysis is a good way to choose the best model. Apparently that is what you've done between the ETS models, but you can add the ARIMA to the horse-race.
You will be interested in analyzing some forecasting performance measures, such as RMSE and MAPE, and choosing the model that minimizes those measures.
Usually in practice you may find similar RMSEs or MAPEs between models. Then, a formal way to test if the models provide statistically different forecast errors is to perform the Dielbold-Mariano test.
Note that the package 'forecast' by professor Hyndman has the accuracy() function for calculating the forecasting measures (both in- and out-of sample). It also has the dm.test(), for Diebold-Mariano test.