Not sure if this post belongs here or if stack overflow would be more appropriate.
I am starting to familiarize with the caret package in R which seems very powerful for the purpose of optimizing and implementing various machine learning methods. According to my understanding the key idea of the package is to train a model across different parameter sets and resampling methods and to select the optimal calibration based on a certain performance measure. This optimized model can subsequently be used to compute predictions on the test data.
Does the package also allow computing predictions for all trained models other than the optimal model?
If this is possible a minimum working example would be nice, but not essential.
The reason for my question is that I am interested in checking the predictive performance of the optimal model relative to the other trained models on the test data. Moreover, I would like to evaluate the performance of forecast combination schemes based on multiple model calibrations.
Best Answer
No, in the returned
model
caret does only providefinalModel
as the determined best parametrization trained again on all training data without resampling or similar. Thereby, the final training is the same as if you would have trained this parametrization withtrainControl(method='none')
.Therefore, what you can do: train those parametrizations you would like to get a test set performance by hand, using
trainControl(method='none')
and all training data. You could then apply all those models to your test set usingpredict(model, ...)
. But keep in mind that you should not compare multiple models based on only the test set performance.Update: caret provides a good explanation on how to compare multiple models with partitioning + resampling. This could boil down to something like: