Which attribute should consider as best fitted model AICc or RMSE in auto.arima?
I am having a case of
best$aicc <- Inf
for(i in 1:25){
r = fourier(timeSeries,K=i)
ar <- auto.arima(anotherTimeseries, xreg= r, seasonal = false)
if(ar$aicc < best$aicc){
best = ar /* this part is reached only once for my data, the only first value is set as best and the aicc value keep on increasing*/
}else{
acc <- accuracy(ar)
acc1 <- accuracy(best)
/* check the RMSE accuracy rate and the set lowest RMSE value to best*/
}
forecast(best, xreg = (best i value from pervious value), h= 104)
}
Now, The doubt is whether I need to choose a best fitting model based on aicc value or RMSE value check (in the else part). Which approach will be proper?
AICC RMSE
1642.857 acc- 233.6344
acc1 - 234.3495
1651.623 acc- 233.3246
acc1 - 234.3495
acc- 232.7801
1656.273 acc1- 234.3495
RMSE value decreases in every step but AICC value increases. Which one would be better arima model ? Thanks in advance for the suggestions
Best Answer
forecast::accuracy
will give you in-sample accuracy measures which are useless for model selection. By construction, a model with some additional Fourier terms will beat a model without them in sample. But we are normally interested in generalization performance, i.e. out of sample.On the other hand, AICc estimates expected likelihood out of sample (as explained in this answer and can be found in Hastie et al. "The Elements of Statistical Learning") and is a sound criterion for model choice, especially if the goal is forecasting. You should pick the model with the lowest AICc.