RMSE – How to Test if Two RMSE Values are Significantly Different

diebold-mariano-testrmsstatistical significance

Say I have two models for a regression task and from each model I get a RMSE. One RMSE is smaller than the other, however I wish to test if the difference is statistically significant in order to be able to say that one model is better than the other. How can I do it?

Best Answer

To test whether two (root) mean squared prediction errors are significantly different, the standard test is the Diebold-Mariano test (Diebold & Mariano, 1995, Journal of Business and Econonomic Statistics). We have a diebold-mariano tag, which may be useful. I also recommend Diebold's (2015, Journal of Business and Econonomic Statistics) personal perspective on uses and abuses twenty years later.

Related Solutions

Solved – How to test if two regression models are significantly different in R

Yes you can only use ANOVA for nested models. That means you can compare models of the form Energy ~ HDD + House and Energy ~ HDD, but not across the same model used with different samples. Instead, take a look at the response to this question: Test equivalence of non-nested models.

Taken from that thread, you can use some different functions:

library(lmtest)
coxtest(fit1, fit2)
jtest(fit1, fit2)

Coxtest is the Cox LR test, and jtest is the Davidson-MacKinnon J test.

Solved – Is it possible to compare the RMSE of different models with different datasets

RMSE is a scale-dependent measure of forecast accuracy. If the scales of your dependent variables differ across the cases, RMSEs from the different cases will not be comparable. For example, if you predict price in dollars in one case and price in thousands of dollars in another case, comparing RMSEs directly will not make sense.

If you want to compare RMSEs for completely different dependent variables, such as daily values of S&P500 stock price index and monthly temperature in Chicago, you have to be aware of the fact that some things are intrinsically easier to predict than other. For example, if you guess the direction of change in S&P500 correctly 55% of the time, that might not seem impressive on absolute terms but it would be very impressive given the subject-matter context. Meanwhile, if you guess the direction of change in temperature in Chicago correctly only 55% of the time, that would be really low as almost any other forecast source would do better.

Best Answer

Related Solutions

Solved – How to test if two regression models are significantly different in R

Solved – Is it possible to compare the RMSE of different models with different datasets

Related Question