Solved – Which is the best accuracy measuring criteria among rmse, mae & mape

arimaforecastingmaemaperms

I have created training set and test set from my data. Then I performed auto.arima() and ets() in R on the training set to predict one-step ahead forecasts. These were then compared with the test set values to measure error, namely RMSE, MAPE & MAE.

This is the output of both ets and auto.arima

 RMSE.ets
 [1] 3767.561
 RMSE.ar
 [1] 3776.308
 MAE.ets
 [1] 2885.112
 MAE.ar
 [1] 2624.482
 MAPE.ets
 [1] 0.04232065
 MAPE.ar
 [1] 0.03857747

Which criteria should be ideally used to select one of the two models (ets or auto.arima) for future predictions. Or is there any other criteria that I am missing out on.

Kindly help.

Best Answer

I have to agree with Glen.

It is axiomatic in control system's engineering that there is no such thing as "best" without a measure of goodness.

Some (weak) examples of candidate bests include:

Best = robust indicator of central tendency
Best = robust indicator of variation around central tendency
Best = fastest to compute

Personally, when trying to select models, I like to use AICc because it is "good enough". It accounts for over-fitting, has a fair basis in statistics, and is comprised using figures of merit that many systems have as outputs.

Here is some info on it: http://www4.ncsu.edu/~shu3/Presentation/AIC.pdf

One of its family members is BIC (Bayes Information Criterion): link1,link2. You might want to explore "Information Criterion" for model selection.

You might consider using "Akaike weights" to combine your models for better predictive power.

Related Solutions

Solved – Which is the better method to do forecast..1-step or h-step ahead

Your questions are still not very clear, but if you want to compare the forecast performances of the two methods maybe you can do it like this?

library(forecast)
data<-USAccDeaths
tr.data<-window(data,statrt=c(1973,1),end=c(1977,12)) # Training data
test.data<-window(data,statrt=c(1978,1),end=c(1978,12)) # Test data

Models:

arima_model<-auto.arima(tr.data)
ets_model<-ets(tr.data)

Forcasts with the models:

arima_f<-forecast(arima_model,h=12)
ets_f<-forecast(ets_model,h=12)

Extracting the 12-step-ahead forecast errors from the models:

arima_f_e<-test.data-arima_f$mean
    ets_f_e<-test.data-ets_f$mean

Calculation of MSE for the forecasts:

mean((ets_f_e<-test.data-ets_f$mean)^2)
72275.59 => MSE of ets-forecast

mean((arima_f_e<-test.data-arima_f$mean)^2) 
83421.72 => MSE of arima-forecast


Application of dm.test from forecast-package to compare predictive accuracy:

dm.test(arima_f_e,ets_f_e)
=>p-value = 0.6145 
although  the MSE of ets is smaller, hypothes of equal performance cannot be rejected for the 12-step forecasts

You can also compare the errors within the training period:

dm.test(arima_model$res,ets_model$res) 
p-value = 3.548e-05 
=> the hypothesis of equal performance of one-step forecasts is rejected!

If you modify the alternative hypothesis, that ets outperforms arima, you get strong     signal that it does:

dm.test(arima_model$res,ets_model$res,alternative="greater")
p-value = 1.774e-05
alternative hypothesis: greater

I wonder obout this result, because the aic criteria for the arima models were much lower than for ets.

I hope it helps you a little bit.

Solved – time series forecasting using auto.arima and exponential smoothing

Seasonality is probably not very strong. Different algorithms will give different results, unless seasonality is glaringly obvious.
The best measure is always to compare forecast accuracy on a holdout set: hold back the last $n$ observations, fit your models to all other observations, forecast into the last $n$ time periods with both models, then compare forecast accuracy using your error measure of choice (see 5 below).
Yes, this is a common complaint. I don't think there is an easy way to get the in-sample fit. But you can get the residuals: auto.arima(WWWusage)$residuals. Best to look into the code of auto.arima() to see whether you need to add or subtract them from the original series to get the fit. I'd say you have to subtract ("actuals=model+residuals"), but better check.
I recommend a good forecasting textbook. This is a very good start. Otherwise, read through the help pages.
The appropriate error measure will depend on your personal loss function. Is your pain symmetric, and will it increase more strongly with larger errors? Then use MSE. Is your pain proportional to absolute errors? Then use MAE. Best to look at multiple error measures.

One tip: averaging forecasts will usually improve accuracy. Consider taking the average of your two models' forecasts per future time bucket.
auto.arima() apparently fits no drift, even if you allow it.

Best Answer

Related Solutions

Solved – Which is the better method to do forecast..1-step or h-step ahead

Solved – time series forecasting using auto.arima and exponential smoothing

Related Question