Solved – Diebold-Mariano test for multiple prediction horizons

accuracydiebold-mariano-testforecastingr

I am trying to compare two forecasts using the Mariano Diebold test in R. Both forecasts are for 150 days ahead; that is, on day $t$ I forecast $t+1, t+2, \dotsc, t+150$.

I deduced from this post that my forecast horizon $h=150$. Using that, the Diebold-Mariano test (implemented using function dm.test in "forecast" package in R) gives a p-value of 1 no matter what forecasts I compare.

I looked into the code of this function, and I figured that this is caused by the following 3 lines of code (d is the vector of loss-differential series):

n <- length(d)
k <- ((n + 1 - 2 * h + (h/n) * (h - 1))/n)^(1/2)   
STATISTIC <- STATISTIC * k

Since in my case n = h, the variable k will always be 0 and therefore the test statistic is always zero.

Questions:

  1. Does this mean that we cannot use the Diebold-Mariano test when we are forecasting an entire period at once? I could not find any evidence of this in their paper.
  2. How should I proceed to find a model-based way to compare my forecasts, rather than for example simply taking the MSE?

Best Answer

Background
Your forecast horizon is not $150$ but rather a whole vector $h=(h_1,\dotsc,h_{max})=(1,\dotsc,150)$. You only seem to have one observation of forecast errors from the different forecasts for each horizon $h_i$, for $i=1,\dotsc,150$. You cannot compare two forecasts $f_{1,i}$ and $f_{2,i}$ with any confidence because you only have one observation for $f_{1,i}$ and $f_{2,i}$. (Testing equality of means from a sample of size 1 won't work well.)

Answers

  1. You cannot directly use the standard Diebold-Mariano test in this setting.
  2. If you went for the standard Diebold-Mariano setting and collected multiple forecast errors for one horizon (e.g. using rolling windows within the original sample), you could use the standard Diebold-Mariano test.
Related Question