Solved – How to choose between ROC AUC and F1 score

machine learningmodelingrocscoring-rules

I recently completed a Kaggle competition in which roc auc score was used as per competition requirement. Before this project, I normally used f1 score as the metric to measure model performance. Going forward, I wonder how should I choose between these two metrics? When to use which, and what are their respective pros and cons?

Btw, I read the article here What are the differences between AUC and F1-score?, but it doesn't tell me when to use which.

Thanks in advance for any help!

Best Answer

Calculation formula:

  • Precision TP/(TP+FP)
  • Recall: TP/(TP+FN)
  • F1-score: 2/(1/P+1/R)
  • ROC/AUC: TPR=TP/(TP+FN), FPR=FP/(FP+TN)

ROC / AUC is the same criteria and the PR (Precision-Recall) curve (F1-score, Precision, Recall) is also the same criteria.

Real data will tend to have an imbalance between positive and negative samples. This imbalance has large effect on PR but not ROC/AUC.

So in the real world, the PR curve is used more since positive and negative samples are very uneven. The ROC/AUC curve does not reflect the performance of the classifier, but the PR curve can.

If you just do the experiment in research papers, you can use the ROC, the experimental results will be more beautiful. On another hand, PR curve use in the real problem, and it has better interpretability.

Related Question