Solved – How to retrieve model-specific information when using the caret package

caretr

CARET is a convenient tool that wraps dozens of modelling methods. CARET allows the user to access performance statistics such as RMSE, confusion matrices, etc. in a generic way without having to recall the usage of various modeling methods.

However, some models have unique reporting outcomes. For example, with a boosted tree model one can compare the in-sample vs. out-sample error-rate as a function of the number of trees to test for overfitting. Naturally this report would not make sense with other methods that do not use trees.

Is there a way I can use CARET to access these model-specific reports (maybe by accessing the modeling object generated directly)?

Best Answer

The train function in caret returns an S3 list object. On of the items in this list is called finalModel. This item is just the fitted object in it's native form. So if you trained a gbm using train and stored it in an object called my.fit, then summary(my.fit$finalModel) would use the summary.gbm method from the gbm package and show you the relative variable importance.

Related Solutions

Solved – the best strategy to train and validate classification using PLS-[classifier] in caret package

I don't have time right now to answer all your questions, but here's a start:

yes you can optimize whatever hyperparameters you have to optimize with the same optimization set. The important thing is to make sure that the test set used for the final measurement of prediction performance is kept independent of all kinds of training data, and the optimization set is part of the training data.
See nested or double cross/resampling/out-of-bootstrap validation for search terms.
Model selection as in comparing a number of models and then keeping the "best" is a data-driven optimization. It belongs into the optimization stage, and in order to get the performance of the chosen model, you need independent test data!
As you already say that your sample size is quite small (though I'd be happy to have that many patients, the more so, as I have far more variates...), it is probably better to restrict yourself to not spend samples on optimization - the more so, as you probably cannot do any meaningful model comparison with that sample size anyways. And optimization (as in "pick the best") usually boils down to a massive multiple comparison situation.
bootstrap vs. cross validation: boostrap is preferred by some, cross validation by other disciplines. For my type of data, iterated k-fold cross validation and out-of-bootstrap had very similar total error:
Beleites, C. et al.: Variance reduction in estimating classification error using sparse datasets, Chemom Intell Lab Syst, 79, 91 - 100 (2005).
See also Kim, J.-H.: Estimating classification error rate: Repeated cross-validation, repeated hold-out and bootstrap , Computational Statistics & Data Analysis , 53, 3735 - 3745 (2009). DOI: 10.1016/j.csda.2009.04.009 for similar findings.
Comparing different PLS surrogate models generated during resampling validation:
- Centering: you don't have to mean center your data. That is just a default. One alternative that would be more stable across the surrogate models would be if among your classes you have e.g. a control group. Then the mean of the controls would make a nice center which is also biologically more meaningful.
- The same for scaling: variance scaling again is just some default procedure. But if you can derive a meaningful scaling by external knowledge that is fine, and possibly even constant for all surrogate models.
- Even if you stay with mean centering and variance scaling: you can compare the mean and scaling vectors of the different surrogate models. If they are not reasonably stable, that is a strong indicator that your sample size is too small to derive a meaningful predictive model.
- Once the center and scale is (approximately) constant, and if your dummy coding of the dependent stays the same, then (unlike PCA loadings and the PLS X loadings) the PLS coefficients should be stable as well. LDA or PLS-LDA final coefficients have some degrees of freedom (flipping and rotation) that don't affect the prediction. Therefore you can and should align the surrogate models accordingly to avoid seeing meaningless variation. See Beleites, C. et al.: Raman spectroscopic grading of astrocytoma tissues: using soft reference information, Anal Bioanal Chem, 400, 2801-2816 (2011). DOI: 10.1007/s00216-011-4985-4 for more explanations and an example.
Q4: such constraints are known as stratification.
as for caret and PLS preprocessing, maybe a glance at the code for PCA pre-processing allows you to define a PLS preprocessing? Otherwise, you'll probably have to set up custom PLS-xxx models

Solved – Understanding the output of C5.0 classification model using the CARET package

Thanks for the plug =]

1) The winnowing process is erroneously removing predictors that can improve the accuracy of the model. Within the cross-validation loop, the winnowing process thinks that it is improving the accuracy, but that is not holding up once other samples are used to evaluate performance. Sometimes it helps and other times is doesn't

2) There is no graph of the tree yet (but it is on my list). Try using the summary function:

> set.seed(1)
> mod <- train(Species ~ ., data = iris, method = "C5.0")
> ## This data set liked rules over trees but it works the same for trees
> summary(mod$finalModel)

Call:
<snip>
-----  Trial 0:  -----

Rules:

Rule 0/1: (50, lift 2.9)
        Petal.Length <= 1.9
        ->  class setosa  [0.981]

Rule 0/2: (48/1, lift 2.9)
    Petal.Length > 1.9
    Petal.Length <= 4.9
    Petal.Width <= 1.7
    ->  class versicolor  [0.960]
<snip>
Evaluation on training data (150 cases):

Trial           Rules     
-----     ----------------
      No           Errors

   0         4    4( 2.7%)
   1         5    8( 5.3%)
   2         3    6( 4.0%)
   3         6   12( 8.0%)
   4         4    5( 3.3%)
   5         7    3( 2.0%)
   6         3    8( 5.3%)
   7         8   15(10.0%)
   8         4    3( 2.0%)
   9         5    5( 3.3%)
boost             0( 0.0%)   <<


   (a)   (b)   (c)    <-classified as
  ----  ----  ----
    50                (a): class setosa
          50          (b): class versicolor
                50    (c): class virginica


Attribute usage:

100.00% Petal.Length
 66.67% Petal.Width
 54.00% Sepal.Width
 46.67% Sepal.Length


Time: 0.0 secs

HTH,

Max

Best Answer

Related Solutions

Solved – the best strategy to train and validate classification using PLS-[classifier] in caret package

Solved – Understanding the output of C5.0 classification model using the CARET package

Related Question