Solved – ROC curve for knn model

rroc

I am using ROCR package and I was wondering how can one plot a ROC curve for knn model in R? Is there any way to plot it all with this package?

I don't know how to use the prediction function of ROCR for knn. Here's my example, i am using isolet dataset from UCI repository where i renamed the class attribute as y:

cl <- factor(isolet_training$y)
knn_isolet <- knn(isolet_training, isolet_testing, cl, k=2, prob=TRUE)

Now my question is, what are the arguments to pass to the prediction function of ROC. I tried the 2 below alternatives which are not working:

library(ROCR)
pred_knn <- prediction(knn_isolet$y, cl)
    pred_knn <- prediction(knn_isolet$y, isolet_testing$y)

Best Answer

ROCR plots ROCs for binary classification only, and ISOLET has 26 classes.

Related Solutions

Imbalanced Data – Area Under the ROC Curve vs PR Curve for Imbalanced Data

The question is quite vague so I am going to assume you want to choose an appropriate performance measure to compare different models. For a good overview of the key differences between ROC and PR curves, you can refer to the following paper: The Relationship Between Precision-Recall and ROC Curves by Davis and Goadrich.

To quote Davis and Goadrich:

However, when dealing with highly skewed datasets, Precision-Recall (PR) curves give a more informative picture of an algorithm's performance.

ROC curves plot FPR vs TPR. To be more explicit: $$FPR = \frac{FP}{FP+TN}, \quad TPR=\frac{TP}{TP+FN}.$$ PR curves plot precision versus recall (FPR), or more explicitly: $$recall = \frac{TP}{TP+FN} = TPR,\quad precision = \frac{TP}{TP+FP}$$

Precision is directly influenced by class (im)balance since $FP$ is affected, whereas TPR only depends on positives. This is why ROC curves do not capture such effects.

Precision-recall curves are better to highlight differences between models for highly imbalanced data sets. If you want to compare different models in imbalanced settings, area under the PR curve will likely exhibit larger differences than area under the ROC curve.

That said, ROC curves are much more common (even if they are less suited). Depending on your audience, ROC curves may be the lingua franca so using those is probably the safer choice. If one model completely dominates another in PR space (e.g. always have higher precision over the entire recall range), it will also dominate in ROC space. If the curves cross in either space they will also cross in the other. In other words, the main conclusions will be similar no matter which curve you use.

Shameless advertisement. As an additional example, you could have a look at one of my papers in which I report both ROC and PR curves in an imbalanced setting. Figure 3 contains ROC and PR curves for identical models, clearly showing the difference between the two. To compare area under the PR versus area under ROC you can compare tables 1-2 (AUPR) and tables 3-4 (AUROC) where you can see that AUPR shows much larger differences between individual models than AUROC. This emphasizes the suitability of PR curves once more.

Solved – R – Plotting a ROC curve for a Naive Bayes classifier using ROCR. Not sure if I’m plotting it correctly

The problem may lies here

pr <- prediction(pred, realResults)

You may transform "pred" and "realResults" to 0-1 vector by :

predvec <- ifelse(pred=="Lost", 1, 0)
realvec <- ifelse(realResults=="Lost", 1, 0)

and using:

pr <- prediction(predvec, realvec)

the problem may be solved.

Bonus part:

you can plot roc curve with more information by:

plot(ROCRperf, colorize=TRUE, print.cutoffs.at=seq(0,1,by=0.1), text.adj=c(-0.2,1.7))

And a simple way to get auc:

as.numeric(performance(ROCRpred, "auc")@y.values)

Best Answer

Related Solutions

Imbalanced Data – Area Under the ROC Curve vs PR Curve for Imbalanced Data

Solved – R – Plotting a ROC curve for a Naive Bayes classifier using ROCR. Not sure if I’m plotting it correctly

Related Question