When applying the Diebold-Mariano test to test for predictive accuracy we need to specify a loss differential. For instance the loss differential $d$ in terms of the mean absolute error (MAE) is
$d = abs(e_1) – abs(e_2)$
where $e_1$, $e_2$ are the errors of two competing forecasts and are nobs-by-1 matrices.
When we want to compare the forecasts in terms of root mean square error (RMSE), how the loss differential should be defined? My intuition says
$d = (e_1^2)^{0.5} – (e_2^2)^{0.5},$
but this is exactly as the MAE above. Do we maybe need to use the MSE in this case?
What's missing?
Best Answer
I think root mean squared error (RMSE) is not a valid loss function when defined on a single forecast error, as there it reduces to the absolute error (as you note yourself).
The Diebold-Mariano test considers two random variables $e_1$ and $e_2$ that generate forecast errors $e_{1,i}$ and $e_{2,i}$ for $i=1,\dots,n$ where $n$ is the number of times a forecast is made (and an error committed). The test aims at assessing whether the expectations of some function (absolute value, square, ...) of $e_1$ and $e_2$ are equal.
Meanwhile, the "mean" in "root mean squared error" refers to averaging over a number of elements, thus it takes more than one error as its argument. As such it does not fit into the framework of the Diebold-Mariano test.
But you could also think as follows. Consider a group of errors (e.g. $5$ consecutive errors) as a single element; thus instead of two scalar random variables $e_1$ and $e_2$ you have two vector-valued random variables $\varepsilon_1$ and $\varepsilon_2$. Take RMSE as your loss function. If you have a sample of $n$ realizations of $\varepsilon_1$ and $\varepsilon_2$ (corresponding to $5n$ "individual errors"), then you can use the Diebold-Mariano test to assess whether the population RMSEs of $\varepsilon_1$ and $\varepsilon_2$ are the same.