I understand for highly imbalanced dataset – we need to look for precision-recall vs ROC AUC to better judge the model.
My question is what is the range for PR AUC below which the model is bad? My current model has an ROC AUC of >90% while PR AUC is only 40%. Is the model bad due to low PR AUC or range for PR AUC is different than ROC AUC?
Best Answer
You are correct to be suspicious of your results. While indeed it is relatively easy to somewhat simplistically dismiss an AUCROC as "bad" if it is close to $0.50$ (roughly speaking the probability that the model ranks a random positive example more highly than a random negative example), the same rationale is not relevant to the case of AUCPR. That is because the baseline of an AUCPR is not $0.50$ but rather it is dictated by the proportion of positives in our sample. That means that when dealing with an imbalanced sample our actual base-line might extremely low; one can read a more detail exposition on this matter on the CV.SE thread here: What is "baseline" in precision recall curve.
If we want a more informative interpretation of the P-R analysis, we can use what is knows as Precision-Recall Gain curves; these allows us to view the AUCPRG as the expected $F_1$ score. Details on the CV.SE thread gere: Area Under the Precision Recall curve -similar interpretation to AUROC?.
So to recap, a model with AUCROC ~ $90\%$ and AUCPR ~ $40\%$ is not bad, or good for that matter. Without a reference point for performance these numbers do not much match and especially the AUCPR does not lent itself to simple direct interpretations either.