Solved – The relation between a confusion matrix and a ROC curve

confusion matrixrroc

As an example I have a confusion matrix that shows good accuracy but poor performance on sensitivity because of imbalanced classes. I made this fictive table for a presentation.

                actual          actual
               positive        negative
pred. positive      6              20
pred. negative      6              986

I have two questions:

First, is it possible to draw a ROC curve that matches with this confusion matrix and how to do this without having the original data? Is there an easy/short way or should I somehow reconstruct some "fictive" data.

Second, could different ROC curves potentially match with the same confusion matrix? I thought that classification thresholds may differ and therefore may result in different ROC curves with similar confusion matrix (see Fawcett 2006.

For implementation I am using the ROCR package in R.

Best Answer

This matrix is just a point on your ROC curve obtained for the threshold you picked. You can compute a value of sensitivity and specificity with your matrix, this is where you point is. $$sensitivity=6/12=1/2$$ $$1-specificity=986/1006=0.98$$ Many different ROC curves could then cross this point. If you can move this threshold, you can draw your ROC curve.

The whole point of ROC curves is to see how sensitivity and specificity vary across various threshold.

Related Solutions

Machine Learning – What Does High AUC Score but Poor F1 Indicate for Imbalanced Dataset?

Out of context, it’s really hard to say how good performance is. While your AUC around $0.8$ could be quite good, it could be that your performance is rather pedestrian or that even a value of $0.55$ is excellent.

A key point to remember for the $F1$ score is that it requires a threshold, while AUC is calculated over all thresholds, and your software is using a default threshold of $0.5$ that might be wildly inappropriate. You might find it informative to write a bit of code that calculates the $F1$ over a range of thresholds, something like:

for threshold in (0.1, 0.2, 0.3,…0.8, 0.9):
    Map probability outputs to categories
    Calculate F1

I suspect you will find a better $F1$ score at a different threshold. With a few more lines, you can plot the $F1$ as a function of the threshold, which you might find useful.

This question at least alludes to the decision-making aspect of the problem, too, where a hard decision about a category must be made. I, along with plenty of high-reputation members here, would argue that, unless you know what you gain from correct classifications and what you lose from incorrect classifications, you have no business making hard categorical predictions and should be predicting probabilities. Nonetheless, the code exercise above should show that you can tweak the threshold as needed to make hard classifications once the gains and losses are known.

I will close with some of the usual links I post on this topic. Having seen several of your posts here and on Data Science, I highly recommend Frank Harrell’s blog posts.

Are unbalanced datasets problematic, and (how) does oversampling (purport to) help?

https://www.fharrell.com/post/class-damage/

https://www.fharrell.com/post/classification/

https://stats.stackexchange.com/a/359936/247274

Proper scoring rule when there is a decision to make (e.g. spam vs ham email)

Best Answer

Related Solutions

Machine Learning – What Does High AUC Score but Poor F1 Indicate for Imbalanced Dataset?

Related Question