Solved – Which is the better method to do forecast..1-step or h-step ahead

arimacross-validationexponential-smoothingforecastingr

I am using forecast() package in R to predict future values. I have a time series data
for approx 6-7 years.

First, I split the data into training set and test
set. Test set contained values of the last 12 months of original data. Then, applied ets() and auto.arima() on the training set. This can be done by iterating 12 times using 1-step ahead OR by running once to get 12 points at the same time, which is my question no. 1

Then, the output received is compared with the test set to check which of the two (ets or auto.arima) gives minimal error in comparison with test set. The selected model class is then applied on the original data set to predict future 12 values. So, here arises my question no. 2

In Short, I have two things to do

  1. Compare accuracy measures for test set of ets and auto.arima to select one
  2. Predict future values using the derived model from the first step

Questions

  1. Should I use one-step ahead or 12-step ahead forecasts to generate data for test set?

  2. Should I use one-step ahead or 12-step ahead forecasts to predict future values?

  3. Should the method be same in both the steps mentioned above?

Kindly help.

Best Answer

Your questions are still not very clear, but if you want to compare the forecast performances of the two methods maybe you can do it like this?

library(forecast)
data<-USAccDeaths
tr.data<-window(data,statrt=c(1973,1),end=c(1977,12)) # Training data
test.data<-window(data,statrt=c(1978,1),end=c(1978,12)) # Test data

Models:

arima_model<-auto.arima(tr.data)
ets_model<-ets(tr.data)

Forcasts with the models:

arima_f<-forecast(arima_model,h=12)
ets_f<-forecast(ets_model,h=12)

Extracting the 12-step-ahead forecast errors from the models:

arima_f_e<-test.data-arima_f$mean
    ets_f_e<-test.data-ets_f$mean

Calculation of MSE for the forecasts:

mean((ets_f_e<-test.data-ets_f$mean)^2)
72275.59 => MSE of ets-forecast

mean((arima_f_e<-test.data-arima_f$mean)^2) 
83421.72 => MSE of arima-forecast


Application of dm.test from forecast-package to compare predictive accuracy:

dm.test(arima_f_e,ets_f_e)
=>p-value = 0.6145 
although  the MSE of ets is smaller, hypothes of equal performance cannot be rejected for the 12-step forecasts

You can also compare the errors within the training period:

dm.test(arima_model$res,ets_model$res) 
p-value = 3.548e-05 
=> the hypothesis of equal performance of one-step forecasts is rejected!

If you modify the alternative hypothesis, that ets outperforms arima, you get strong     signal that it does:

dm.test(arima_model$res,ets_model$res,alternative="greater")
p-value = 1.774e-05
alternative hypothesis: greater

I wonder obout this result, because the aic criteria for the arima models were much lower than for ets.

I hope it helps you a little bit.

Related Question