Solved – must ROC curve be concave

roc

How can I decide whether the attached is a correct ROC curve?

The underlying algorithm consists in calculating an ODE system and storing the number of certain cancer cells calculated by this model at the end of the simulation course.
These end points are compared with experimental results for different thresholds.

It looks to me as if the model is not performing better then random guess, do I see this correctly?

enter image description here

Edits

I rerun the dataset with pROC package – the monotonicity is restored but the rest looks as it was symmetric wrt identity line compared to the original plot (which is worrisome because it means I cannot compute the ROC curve by myself).
enter image description here

Best Answer

must ROC curve be concave?

No, a ROC curve can be concave, convex, or a mix of those on different segments (see this question on SO).

How can I decide whether the attached is a correct ROC curve?

A ROC curve always increases monotonically, so the curve you posted is clearly not a ROC curve.

More generally, making a ROC curve is pretty hard and it is easy to get some details and special cases wrong. So if you want to try your own code I would recommend cross-checking with one of the numerous dedicated functions/packages that already exist and have been thoroughly checked.

It looks to me as if the model is not performing better then random guess, do I see this correctly?

It would seem so visually. Try to add a diagonal line and you will see that parts of your curve are above (better than random) and other below (worse than random) the diagonal. However at times even a small signal might be significant, so you want to check for instance with the DeLong test whether the area under the curve AUC is different from 0.5 (H0: AUC = 0.5).