Solved – Diebold-Mariano in the context of volatility forecasting: What is the ultimate aim of this test

diebold-mariano-testforecastingrmsvolatility-forecasting

Perhaps I'm missing a simple conceptual point here. But do the error statistics (RMSE, MAE) not tell which is the best forecast by presenting the lowest figures between the forecast and the actual figures? Why then do we need to compare two forecasts directly, given we've already established which is one is superior via RMSE etc? Isn't our interest in establishing which one replicates values closest to the actual figures?

Secondly how would one go about interpreting the following Diebold Mariano test I've conducted. Would the below be correct?

<>prob

H0: Both forecasts have the same accuracy
H1: The forecasts do not have the same accuracy

$<0.05$ reject null, accept alternative hypothesis
$>0.05$ accept null, both forecasts have the same accuracy

>Prob

H0: Both forecasts have the same accuracy
H1: Forecast 1 is less accurate than forecast 2

$<0.05$ reject null, accept alternative hypothesis
$>0.05$ accept null, Both Forecasts have the same accuracy

<.Prob

H0: Both forecasts have the same accuracy
H1: Forecast 2 is more accurate than Forecast 1

$<0.05$ reject null, accept alternative hypothesis
$>0.05$ accept null, Both Forecasts have the same accuracy

enter image description here

This can then ultimately be written as something close to the following: The table shows the Diebold-Mariano test statistics for different sub-periods. It is employed to test whether the RMSE performance between the Short Moving Average of 5 days and other volatility forecasting models are statistically different from one another. A negative value indicates that the Short Moving Average of 5 days is better than the second model.

i.e. Negative values of the Diebold–Mariano test show that the squared errors of the model listed first (short moving average) are lower than those of the model listed last.

Best Answer

Why then do we need to compare two forecasts directly, given we've already established which is one is superior via RMSE etc? Isn't our interest in establishing which one replicates values closest to the actual figures?

Let Diebold himself answer this for you (Diebold, 2015, second paragraph in the Introduction):

The need for formal tests for comparing predictive accuracy is surely obvious. We’ve all seen hundreds of predictive horse races, with one or the other declared the “winner” (usually the new horse in the stable), but with no consideration given to the statistical significance of the victory. Such predictive comparisons are incomplete and hence unsatisfying. That is, in any particular realization, one or the other horse must emerge victorious, but one wants to know whether the victory is statistically significant. That is, one wants to know whether a victory “in sample” was merely good luck, or truly indicative of a difference “in population.”

(emphasis mine)

Secondly how would one go about interpreting the following Diebold Mariano test I've conducted. Would the below be correct?

Strictly speaking, you never accept a hypothesis (be it null or alternative), you can only fail to reject.
Also note that Forecast 1 is less accurate than forecast 2 is the same as Forecast 2 is more accurate than Forecast 1 (of course you did not mean that).
Other than that, I do not know what the particular software means by >prog (with a spelling error) or <prob, but it indeed looks as if they refer to outcomes of two one-sided $t$-tests. You should be able to figure out which forecast is better by looking at the loss values, then you will know clearly what forecasts these test result correspond to.

References:

Related Question