Solved – How is the optimal probability cut-off in a ROC defined by the R package Epi

aucclassificationrroc

The plot below was created by R package Epi::ROC for a binary classification problem. There is an optimal cut off or threshold for the classification probability at which sensitivity and 1-specifitiy are balanced in some optimal way (called ps3 in the plot). My question is how this point is defined mathematically in this case and if there are other ways to define it.enter image description here

Best Answer

Epi::ROC defines optimal cut-off as a point for which sum of Sensitivity and Specificity is maximized.

See, that Sensitivity and Specificity play similar roles here. But, in general, they don't have to. Sometimes were are more interested in finding highly sensitive test and don't care about Specificity that much (or vice versa). This is the case when we know that False Positive result is much (or less) "bad" than False Negative.

Then we need other expression to maximize. It can be some weighted average of Sensitivity and Specificity.

I never found any general guidelines of how to determine desired balance between Sensitivity and Specificity (that, of course, doesn't mean it doesn't exist). I read a few articles in which authors calculated cost of False Positive and False Negative result of test and derived weights from it. But, I think it is rarely doable...