I am struggling with understanding how the final performance on an independent test set is obtained in a time series forecasting scenario.
My understanding is that rolling CV is used (as seen as in the picture) to prevent future datapoints to leak into the model. The ensemble of scores can than be used to deduce performance.
This method results in many models (in this case, four). What is the final model then selected for further use? Is it the final model, trained on most of the data? Is the performance of this model then described by the scores of the model itself, or by the ensemble of itself and all preceding models?
Best Answer
This does not have to result in 4 models. A common restriction is that the model is the same in all training samples, only its parameters are reestimated in each of them. The model is evaluated on a joint test set which is comprised of the four individual test sets.
If model selection needs to be done, the above is carried out for each model under consideration. The model that yields the best performance on the joint test set gets selected. The model's performance on the joint test set serves as an estimate of its performance on unseen future data. (The latter holds regardless of whether any model selection is going on at all.) The model's parameters are then reestimated on the entire data set, and genuine out-of-sample forecasting begins.