I am trying to compare the accuracies of two time series forecasts. I read about the Diebold-Mariano (DM) Test, which tests the null hypothesis of $E[d_t]=0$, where $d_t=g(e_{it})-g(e_{jt})$ is the loss differential between the two forecast errors ($e_t=y_t-\hat{y}_t$, where $y_t$ is the actual and $\hat{y}_t$ is the forecast). One could set $g(e_{it})=e_{it}^2$ or $g(e_{it})=\vert{}e_{it}\vert$ to obtain a measure based on the differential of MSE or MAE respectively.
However, I am wondering whether the result of this test also applies to other accuracy measures which are not based on the forecast error. For instance, consider Mean Directional Accuracy (MDA), defined as:
where $F_t$ is the forecast and $X_t$ is the actual value at time $t$. MDA is thus the proportion of directions correctly predicted by the forecasts.
If a DM test fails to reject $H_0$ (i.e. two competing forecasts have the same precision on average) but the two forecasts have very different MDAs, does this mean that one of the forecasts is more precise (in terms of MDA) or do the conclusions of the DM test also apply to other measures different from MAE and MSE?
Best Answer
I think Diebold-Mariano test can work on the indicator of directional accuracy instead of the usual absolute error or squared error. If the sample is large enough for the central limit theorem to kick in, you should be able to use the test in the standard way.
Does the superiority of one forecast with respect to one accuracy measure (e.g. absolute error) imply the same forecast's superiority w.r.t. another forecast measure (e.g. squared error or directional accuracy indicator)? No, not in general. Thus you would pick a measure and analyze the different forecasts' performance w.r.t. it separately from other measures, not assuming that the result will be the same for all measures.