So I've been learning how to forecast over this summer and I've been using Rob Hyndman's book Forecasting: principles and practice. I've been using R, but my questions aren't about code. For the data I've been using, I've found that an average forecast of multiple models has produced higher accuracy levels that any sole model by itself.
Recently I read an blog that talked about averaging forecasting methods and assigning weights to them. So in my case, lets say I assign 11 different models to my set of data (Arima, ETS, Holt Winters, naive, snaive, and so forth) and I want to average a few of these to get a forecast. Has anyone had any experience with this or can point me to an article that might give some insight on the best way of going about this?
As of right now, I'm using cross validation and Mean Absolute Error to figure out which models perform best and which perform worst. I can even use this to identify the top k # of models.
I guess my questions are
1) How many models would you suggest selecting? (2,3,4,5,6, etc)
2) Any ideas on weights? (50% to the best, 25% to the second best, 15% third best, 10% to the 4th best, etc)
3) Are any of these forecasting models redundant and shouldn't be included?
(Arima, snaive, naive, HW's "additive", ETS, HoltWinters exponential smoothing, HoltWinters smoothing w/ trend, HoltWinters w/ trend/seasonality, multiple regression)
Best Answer
The answers to your questions in order
How Many Models
Usually as many as you want, but this can be limited by the amount of data you have. Also depends on the method you are using to derive the weights (which I explain more below)
How to Assign Weights
There are many, here are the five most popular off the top of my head, though non of them use Mean absolute error.
Are Any of the Forecasting Models Redundant/Excludable
The Holt-Winters models are likly to be similar so maybe throw a couple of those out. Averaging forecasts is like diversifying a financial portfolio, you want your models to be diverse. With some of the above averaging techniques it doesn't hurt to include redundant models, with others it does.
You can also find a friendly introduction here, with couple of more good ways to average forecasts (Constrained Least Squares for example) along with an R implementation.