I have 5 datasets and for each one of them I created 2 prediction models. For such task I divided each dataset in training and testing set (70/30%). Then I assessed the RMSE for each one and calculated the mean of the 5 RMSE for each model.
My question is can I compare my results with other results given that they used another dataset(s)? How can I say that my results are significantly better than the others?
Best Answer
RMSE is a scale-dependent measure of forecast accuracy. If the scales of your dependent variables differ across the cases, RMSEs from the different cases will not be comparable. For example, if you predict price in dollars in one case and price in thousands of dollars in another case, comparing RMSEs directly will not make sense.
If you want to compare RMSEs for completely different dependent variables, such as daily values of S&P500 stock price index and monthly temperature in Chicago, you have to be aware of the fact that some things are intrinsically easier to predict than other. For example, if you guess the direction of change in S&P500 correctly 55% of the time, that might not seem impressive on absolute terms but it would be very impressive given the subject-matter context. Meanwhile, if you guess the direction of change in temperature in Chicago correctly only 55% of the time, that would be really low as almost any other forecast source would do better.