Solved – Is an Average Precision of 60% acceptable output in a fraud detection machine learning algorithm? What does it signifies

average-precisionfraud-detectionmachine learning

First question here, I am new to machine learning and wanted to understand the following:
I used decision trees, boosting to classify fraud users and I am getting average precision around 60% on my test set. What does it signifies and how well am I able to do the classification.

Best Answer

Precision and recall numbers depend on the actual probability of fraud in your data. Generally fraudulent cases are very small in number => Precision might be low, but recall should be high.

Acceptable or not can only be defined by measuring the impact of the results in real-life. I present two examples:

  1. Let's say for your business the goal is to avoid fraud at all costs, then one would strive to achieve as high recall values as possible. But if it matters to you that you don't predict non-fraud cases as fraud, maybe because it hurts customer-sentiment, precision is more important.
  2. Try quantifying the loss due to misclassification. Lets say a False Positive(hurts sentiment) and a False Negative(allows fraud) cause you losses of $l1$ and $l2$. As you strive to improve one of these, the other might get impacted. Thus, you have to exercise the trade-off well in order to get the right model.
Related Question