Because I have a very imbalanced dataset (9% positive outcomes), I decided a precision-recall curve was more appropriate than an ROC curve. I obtained the analogous summary measure of area under the P-R curve (.49, if you're interested) but am unsure of how to interpret it. I've heard that .8 or above is what a good AUC for ROC is, but would the general cutoffs be the same for the AUC for a precision-recall curve?
Solved – a good AUC for a precision-recall curve
aucclassificationprecision-recall
Related Solutions
The question is quite vague so I am going to assume you want to choose an appropriate performance measure to compare different models. For a good overview of the key differences between ROC and PR curves, you can refer to the following paper: The Relationship Between Precision-Recall and ROC Curves by Davis and Goadrich.
To quote Davis and Goadrich:
However, when dealing with highly skewed datasets, Precision-Recall (PR) curves give a more informative picture of an algorithm's performance.
ROC curves plot FPR vs TPR. To be more explicit: $$FPR = \frac{FP}{FP+TN}, \quad TPR=\frac{TP}{TP+FN}.$$ PR curves plot precision versus recall (FPR), or more explicitly: $$recall = \frac{TP}{TP+FN} = TPR,\quad precision = \frac{TP}{TP+FP}$$
Precision is directly influenced by class (im)balance since $FP$ is affected, whereas TPR only depends on positives. This is why ROC curves do not capture such effects.
Precision-recall curves are better to highlight differences between models for highly imbalanced data sets. If you want to compare different models in imbalanced settings, area under the PR curve will likely exhibit larger differences than area under the ROC curve.
That said, ROC curves are much more common (even if they are less suited). Depending on your audience, ROC curves may be the lingua franca so using those is probably the safer choice. If one model completely dominates another in PR space (e.g. always have higher precision over the entire recall range), it will also dominate in ROC space. If the curves cross in either space they will also cross in the other. In other words, the main conclusions will be similar no matter which curve you use.
Shameless advertisement. As an additional example, you could have a look at one of my papers in which I report both ROC and PR curves in an imbalanced setting. Figure 3 contains ROC and PR curves for identical models, clearly showing the difference between the two. To compare area under the PR versus area under ROC you can compare tables 1-2 (AUPR) and tables 3-4 (AUROC) where you can see that AUPR shows much larger differences between individual models than AUROC. This emphasizes the suitability of PR curves once more.
Area under the ROC curve is equivalent to concordance (aka $c$-statistic) (not accuracy!). This can be interpreted as the probability that a random positive is assigned a higher score than a random negative. Unfortunately, area under the PR curve has no such interpretation (that I'm aware of).
The relationship between ROC and PR curves stems from the fact that both are based on the same source: contingency tables for every possible decision value threshold. Every threshold $T$ leads to a contingency table (e.g. $TP^{(T)}$, $FP^{(T)}$, $TN^{(T)}$, $FN^{(T)}$).
Every point in ROC space is based on a certain decision threshold $T$, and therefore coincides with a point in PR space. If a given model's ROC/PR curve dominates another, that model's PR/ROC curve will also dominate (cfr. Davis & Goadrich).
Also, for example, if I obtain an optimal threshold for classifier in ROC curve (like the one that minimize the error), can I use that optimal threshold and calculate the Precision and Recall in PR-curve?
Two remarks: if you want to select the threshold which minimizes the error (maximizes accuracy), ROC curves are not necessary (in fact they don't even show that). Secondly, if you have decided on a threshold you can just use the corresponding contingency table to get whatever other measures you want directly. Don't bother computing a full PR curve to then select 1 point of it.
Keep in mind that neither ROC or PR curves show you which threshold yields a certain point in the given space. They just show you the possible tradeoffs the model is capable of. That said, you can obviously map a point in ROC/PR space to a threshold if you retain a record of what thresholds they correspond to (most software packages to do this anyway).
Best Answer
There is no magic cut-off for either AUC-ROC or AUC-PR. Higher is obviously better, but it is entirely application dependent.
For example, if you could successfully identify profitable investments with an AUC of 0.8 or, for that matter anything distinguishable from chance, I would be very impressed and you would be very rich. On the other hand, classifying handwritten digits with an AUC of 0.95 is still substantially below the current state of the art.
Furthermore, while the best possible AUC-ROC is guaranteed to be in [0,1], this is not true for precision-recall curves because there can be "unreachable" areas of P-R space, depending on how skewed the class distributions are. (See this paper by Boyd et al (2012) for details).