Solved – Diebold Mariano test (in R)

accuracydiebold-mariano-testforecastingmodel comparisontime series

As I asked in my answer to this question: does anyone know if the DM test (in R in this case) is supposed to be made with h=h-1?

If not, am I supposed to make several prediction sets (with h observations each) and perform h-step predictions on them in order to have more degrees of freedom?
That is, am I supposed to make n h-step predictions for the DM test?

When I attempt to perform the test with h=18 (there are 18 obs.) it says that DM variance is 0 (it consumes the degrees of freedom?).

Error in dm.test(error.311arima, error.013arima, :
Variance of DM statistic is zero

The residuals are different for the time series predictions.

EDIT

I have "checked under the hood" of the dm.test function in R, and apparently what happens is that when it executes the following operation:

m.test
function (e1, e2, alternative = c("two.sided", "less", "greater"), 
    h = 1, power = 2) 
{
    alternative <- match.arg(alternative)
    d <- c(abs(e1))^power - c(abs(e2))^power
    d.cov <- acf(d, na.action = na.omit, lag.max = h - 1, type = "covariance", 
        plot = FALSE)$acf[, , 1]
    d.var <- sum(c(d.cov[1], 2 * d.cov[-1]))/length(d)
    dv <- d.var
    if (dv > 0) 
        STATISTIC <- mean(d, na.rm = TRUE)/sqrt(dv)
    else stop("Variance of DM statistic is zero")

the d.var part yields a negative number, i.e. the variance estimator is negative???

I presume it is due to a lack of covariance stationarity on the errors, but then what should I do? I can only think of manually differencing the series (or the error of the series) and then use those forecasts, which should be in turn, covariance stationary.

error 311arima  error 013arima
-0.8758334  -0.8623334
-0.571964484    -0.614464484
-1.5118295  -1.3041295
-2.034418946    -1.918818946
-1.422745494    -1.805045494
-1.777216545    -2.240316545
-1.671314551    -2.260914551
-2.61727251 -3.40957251
-4.29605616 -5.10285616
-4.87524124 -5.78274124
-6.29766468 -7.27736468
-7.458823814    -8.455323814
-0.983687537    -1.110287537
-0.49127652 -0.92037652
0.652971123 0.048671123
2.52594615  1.73144615
0.651429228 -0.409670772
3.416551133 2.252951133
3.302610752 2.023910752
0.929007023 -0.460192977
1.959733687 0.522733687
-0.710917583    -2.217517583
-2.09341202 -3.64921202
-2.320514415    -3.910114415

The answer to this question can be found in a later question here: Diebold-Mariano test for multiple prediction horizons

We cannot compare forecasts for VECTORS of differing (in this case increasing) points in time, we must use a rolling window approach with a fixed h to compare these two models' h-point forecasts along time

Best Answer

The answer to this question can be found in a later question here: Diebold-Mariano test for multiple prediction horizons

We cannot compare forecasts for VECTORS of differing (in this case increasing) points in time, we must use a rolling window approach with a fixed h to compare these two models' h-point forecasts along time

Related Question