After looking at Hyndman and Koehler, 2006 and applying the metric to my own data, I have been convinced that MASE is a better metric for evaluating forecast error than the method I had been previously using (MAPE), at least for short horizon forecasts. However, I am struggling with how to interpret the value of the MASE at longer horizons.
At short horizons it makes sense, it is effectively quantifying how much better (or worse) your forecast did than one of the simplest forecasting methods (the naive/seasonal naive). If your MASE is greater than 1 then, you may want to be suspect that your forecasting method is adding value (assuming there have been no major changes in your data between the in sample and out of sample values).
Where I get confused is for longer horizon forecasts. Imagine I have a situation like below where I have a random walk series I want to forecast (black). For convenience, I will be looking at in sample errors. In cyan, I have plotted the rolling horizon 20 naive forecast for the series as well at the one step (horizon 1) naive forecast in orange which would be used to compute the denominator of the MASE.
The horizon 20 forecast is much worse for obvious reasons, it has less information than the horizon 1 forecast. Accordingly, the MASE reflects this and has a value of 2.88. However, we know that the naive forecast is the best forecast we can make for a random walk series, so it is the forecast we should be using even though the MASE is "bad".
It is easy to know in this circumstance that the MASE being greater than 1 is ok because we have a theoretical grounding, but this will not be the case for most series we want to evaluate. How do we properly interpret the MASE for these longer horizon applications then? Should we create a hMASE which would be a horizonal dependent MASE to match the horizon of interest changing it from
$$
MASE = \frac{\sum_{t=1}^{T} \left| e_t \right|}{\frac{T}{T-1}\sum_{t=2}^T \left| Y_t-Y_{t-1}\right|}
$$
to
$$
hMASE = \frac{\sum_{t=1}^{T} \left| e_t \right|}{\frac{T}{T-h}\sum_{t=h+1}^T \left| Y_t-Y_{t-h}\right|}
$$
or would we need to also calculate the MASE for the naive forecast and use that as a comparison? Is there some other method I am not thinking of? What If I am evaluating both short and long term forecasts? Would I include multiple errors for each in sample data point (one for each horizon) in the denominator of my MASE?
Best Answer
The MASE compares "your" forecast against a naive benchmark forecast calculated in-sample.
The original paper by Hyndman & Kohler (2006) chose the one-step ahead random walk forecast as this benchmark, but this is not set in stone. For instance, if you have an obviously seasonal time series, then the random walk makes little sense as a benchmark, and it would be better to use a seasonal naive forecast (i.e., the value from one season back), like Hyndman does in this paper mentioned in this earlier CV question about the MASE. Rob Hyndman doesn't write so explicitly in his online textbook, but he has repeatedly made this point in discussions about the MASE.
In your case, I'd use a naive 20-step-ahead forecast as the benchmark, i.e., your $hMASE$. Just make it clear in your writeup what the benchmark is that you are comparing "your" forecast against.