Solved – How to implement a hold-out validation in R

caretclassificationcross-validationmachine learningr

Let's say I'm using the Sonar data and I'd like to make a hold-out validation in R. I partitioned the data using the createFolds from caret package as folds <- createFolds(mydata$Class, k=5).

I would like then to use exactly the fold mydata[i] as test data and train a classifier using mydata[-i] as train data.

My first thought was to use the train function, but I couldn't find any support for hold-out validation. I mean, I don't want to partition my data and then test all the combinations for estimating the mean accuracy, I want to train it only once. Am I missing something here?

Also, I'd like to be able to use exactly the pre-defined folds as parameter, instead of letting the function partition the data. Does anyone have any thoughts?

Thanks in advance

Best Answer

It is hard to tell exactly what you are interested in. If it is a single hold-out, you can use

trainControl(method = "LGOCV", p = .8, number = 1)

and 80% will be used for training.

There is also method = "none" that will just fit the model for a single tuning parameter value (using the entire training set).

Also, if you want to use your own hold-out set(s), see the index argument of trainControl.

Max

Related Solutions

Cross-Validation – Hold-Out Validation vs. Cross-Validation

NOTE: This answer is old, incomplete, and thoroughly out of date. Its was only debatably correct when it was posted in 2014, and I'm not really sure how it got so many upvotes or how it became the accepted answer. I recommend this answer instead, written by an expert in the field (and with significantly more upvotes). I am leaving my answer here for historical/archival purposes only.

My only guess is that you can Hold-Out with three hours of programming experience; the other takes a week in principle and six months in practice.

In principle it's simple, but writing code is tedious and time-consuming. As Linus Torvalds famously said, "Bad programmers worry about the code. Good programmers worry about data structures and their relationships." Many of the people doing statistics are bad programmers, through no fault of their own. Doing k-fold cross validation efficiently (and by that I mean, in a way that isn't horribly frustrating to debug and use more than once) in R requires a vague understanding of data structures, but data structures are generally skipped in "intro to statistical programming" tutorials. It's like the old person using the Internet for the first time. It's really not hard, it just takes an extra half hour or so to figure out the first time, but it's brand new and that makes it confusing, so it's easy to ignore.

You have questions like this: How to implement a hold-out validation in R. No offense intended, whatsoever, to the asker. But many people just are not code-literate. The fact that people are doing cross-validation at all is enough to make me happy.

It sounds silly and trivial, but this comes from personal experience, having been that guy and having worked with many people who were that guy.

Solved – Feature selection + classification in Caret

You should be able to accomplish everything you want with the sbf function instead. I originally assumed it worked the same way you are, but the functionality given by sbf is apparently more like a super set of what's available in train.

For example, something like this sounds like what you're getting at:

fit <- sbf(
  form = response ~ .,
  data = d, method = "glmnet", 
  tuneGrid=expand.grid(.alpha = .01, .lambda = .1),
  preProc = c("center", "scale"),
  trControl = trainControl(method = "none"),
  sbfControl = sbfControl(functions = caretSBF, method = 'cv', number = 10) 
)

This would run 10 outer folds and fit a single glmnet model to each, using only a feature subset. You could also specify some number of cv folds for trControl and a parameter grid to do training on inner folds.

Best Answer

Related Solutions

Cross-Validation – Hold-Out Validation vs. Cross-Validation

Solved – Feature selection + classification in Caret

Related Question