I had two datasets that I wanted to cross validate using xgb.cv. I used a pandas dataframe, split each it into 2 where X = what I wanted to predict, y = features.
dtrain = xgb.DMatrix(X, label=y)
I ran it through xgb.cv and these are the results for the first dataset:
Results for the second dataset:
I noticed that the first dataset had a lower RMSE std while the second one had a lower RMSE mean. What is the differece? I thought RMSE gets a higher score when the more the prediction is far off from the actual result, so how can the mean rmse be lower while the rmse standard deviation is higher?
Best Answer
You are doing $k$-fold cross-validation, so $k$ times you train a model on different train set and validate on different test set. After each such run, you calculate an error metric, in this case RMSE. So, you end up with $k$ values of RMSE, the table shows average RMSE and standard deviation of the metric. Standard deviation tells you how variable, or spread, are the values. Preferably, you would like the standard deviation to be ac close to zero as possible, since if the metric varies a lot depending on the training and testing set, this means that it is harder to judge the exact performance of the model on unseen data.