Solved – How to calculate TPR and FPR and plot ROC curves for object detection

model-evaluationroc

According to its Wikipedia page, receiver operating curves are created by plotting the TPR vs. the FPR at various discrimination thresholds where:

TPR = TP / (TP + FN)

FPR = FP / (FP + TN)

What would be the process of plotting this ROC curve with an object detection model? Can this only be accomplished with a multi-class object detection model and not single class?

To my understanding, we must pit a class against all other classes to come up with an ROC curve (at least specifically with obj det models). That is, if we have classes A, B, and C, then an ROC curve for class A would be class A vs B and C, and so on. It is impossible to have an ROC curve otherwise because we cannot calculate TN's and FN's with object detection of a single class.

If the above is correct, would the below be reasonable to calculate the ROC curve of class A vs B, C? (Specifically, is my understanding of FP's, FN's, TP's, TN's and TPR and FPR in this context at each iteration correct?)

  1. Run inference on a test dataset, saving predictions in a list in the order of confidence scores (descending). A single prediction consists of a confidence score, bounding box, and label.

  2. Iterate through the list of predictions. Calculate FP, FN, TP, TN at each iteration where:

    a. False Positive = model predicted an object as A when the object is not A (could be B, C, or nothing) or the A bounding box does not exceed the IOU thresh.

    b. False Negative = model predicted an object as B or C but it was not B or C, or the B or C prediction was correct but the bounding box did not exceed the IOU thresh.

    c. True Positive = model correctly predicted class A and the A bounding box exceeds the IOU thresh.

    d. True Negative = model correctly predicted class B or C and the B or C bounding box exceeds the IOU thresh.

  3. With these values at each iteration, we then add them to a global count of each. For eg. global_false_positive += iter_false_positive).

  4. With each global value at each iteration, we can calculate the TPR and FPR up to that confidence score (row). This (TPR, FPR) pair is a coordinate in the ROC curve.

  5. Plot FPR vs TPR. This gives us the ROC curve for Class A vs B and C. Repeat the two for B vs A and C and C vs A and B.

Best Answer

True Negative (TN): Does not apply. It would represent a corrected misdetection. In the object detection task there are many possible bounding boxes that should not be detected within an image. Thus, TN would be all possible bounding boxes that were corrrectly not detected (so many possible boxes within an image). That's why it is not used by the metrics.

So, no TN, no FPR, no ROC curve!

Origin at https://github.com/rafaelpadilla/Object-Detection-Metrics.

Related Question