MATLAB: How to create an ROC plot from the set of cross-validation models using Statistics Toolbox 7.5 (R2011a)

curvenaivebayesrocStatistics and Machine Learning Toolbox

I am using the CROSSVAL command to validate the performance of 'NaiveBayes' model for my data. I am computing the metrics, misclassification rate and confusion matrix using stratified 10-fold cross-validation to analyze the performance of the classifier. I intend to create an ROC curve from the validation results. Is there any way to create ROC plot from the cross-validation results of my model using Statistics Toolbox 7.5 (R2011a)?

Best Answer

In order to plot ROC curve, the decision boundary should be a value of the posterior probabilities that you compute using ‘NaiveBayes’. You can plot ROC curves using PERFCURVE, which computes the ROC curve for a vector of classifier predictions, given the true class. More details about PERFCURVE are available via the following link:
http://www.mathworks.com/help/toolbox/stats/perfcurve.html
Here is a small example of how to plot an ROC curve without cross-validation:
       load fisheriris
       x = meas(51:end,1:2);
       y = species(51:end);
       nb = NaiveBayes.fit(x,y);
       p = posterior(nb,x);
       [X,Y] = perfcurve(y,p(:,1),'versicolor');
       plot(X,Y)
       xlabel('False positive rate'); ylabel('True positive rate')
       title('ROC for classification by naïve Bayes')
To use 10-fold cross-validation, you can fit the model on 90% of the data, and compute predicted posteriors for the remaining 10% of data which was not used for fitting. You can then loop over each of the 10 subsets to plot the ROC curves for individual runs.
Related Question