I'm doing some Machine Learning in R using the nnet package. I want to estimate the generalisation performance of my classifier by using k-fold cross-validation.
How should I go about doing this? Are there some built-in functions that do this for me? I've seen tune.nnet() in the e1071 package, but I'm not sure that does quite what I want.
Basically I want to do the cross-validation (split into 10 groups, train on 9, test on the other 1, repeat) and then from that obtain some sort of measure of how well my classifier generalises – but I'm not sure what measure that should be. I guess I want to look at the average of the accuracies across the different cross-validation examples, but I'm not sure how to do that with the tune.nnet() function above.
Any ideas?
Best Answer
If you are planning to tune the network (e.g. select a value for the learning rate) on your training data and determine the error generalization on that same data set, you need to use a nested cross validation, where in each fold, you are tuning the model on that 9/10 of the data set (using 10 fold cv). See this post where I asked a similar question and got very good answers (post).
I am not sure if there is an R function to accomplish this - maybe the ipred package could be used by passing the tune function with the call - not sure. It is rather trivial to simply write a loop to accomplish this whole process though.
As far as what metric to examine, that depends on your problem (classification versus regression) and if you are interested in accuracy (classifications rate or kappa) or how well the model ranks (e.g. lift) or the MAE or RMSE for regression.