Solved – Is it valid to select a model based upon AUC

machine learningmodel selectionrroc

I have plot ROC for several models. These models were used to classify my samples into 2 classes.

Using these commands, I can obtain sensitivity vs. specificity plots for each model:

perf <- performance(pred, "sens", "spec")
plot(perf)

Should I rely on the area under the curve (AUC) for each model to conclude which model is better? Other than AUC, should we consider other results so as to conclude which model is better?

If yes, how to get AUC with R? Am I right in assuming that "the smaller it is the better is the classification power of the model?"

Best Answer

AUROC is one of many ways of evaluating the model -- in fact it judges how good ranking (or "sureness" measure) your method may produce. The question whether to use it rather than precision recall or simple accuracy or F-measure is only depending on a particular application.

Model selection is a problematic issue on its own -- generally you should also use the score you believe fits application best, and take care that your selection is significant (usually it is not and some other factors may be important, like even computational time).

About AUC in R -- I see you use ROCR, which makes nice plots but it is also terribly bloated, thus slow and difficult in integration. Try colAUC from caTools package -- it is rocket fast and trivial to use. Oh, and bigger AUC is better.

Related Question