Solved – Evaluating Classifiers k fold CV or ROC

cross-validationmachine learningmodel-evaluationroc

I've been doing a project to determine the 'best' classifier for classification on a dataset from UCI.
I used 10 fold stratified cross validation to calculate the mean accuracy. However it was suggested that I use ROC AUC instead.

My questions are:

1)Which is better cross validation or ROC?

2) Do you perform ROC on the test set, the training set or the whole dataset?

3) If it is the training set or test set. Do I perform it using each fold from the cross validation or should I just split the data once?

Best Answer

AUROC is a measure, just like accuracy, both can be used in a CV setting or outside (validation set, test set).

  1. Depends on the objective and your data. You said you used accuracy, could it be that your classes are unbalanced? ROC as a measure is usually used when you have imbalanced classes, this may be why they recommended you use it instead of accuracy
  2. Since you mentioned CV (which is done on the training set) then you would perform the ROC on the training set also, that is using CV but estimating ROC instead of accuracy. If you had a test set you would have no need for CV, you would therefore use the test set to evaluate your model (accuracy, ROC, ...)
  3. CV is a better option then using the same exact training set that you used to build your model, so you would still do CV with a ROC estimate.
Related Question