I'm training a deep learning model in PyTorch. The first two images I posted here make perfect sense as they are the classical idea of overfitting. The training loss keeps decreasing while the validation loss reaches a minimum and then starts to increase again. I assume this is where it would make sense to stop training the model.
What I don't understand is why the AUC on the validation set would keep increasing monotonically the entire time. Shouldn't it also reach a maximum and then start to decrease due to overfitting?
Best Answer
The simple answer is that binary cross entropy and AUC are different metrics that measure different things. This article gives a example of a data distribution where log-loss (similar to binary cross entropy) and AUC give different results.
The intuition is what these metric care about. AUC only cares about order. if $x_i$ is properly ordered below $x_j$ then AUC is happy. The exact predicted values for the two data points could change, but as long as they don't flip orders with other data points, the AUC remains the same.
Your loss function does care about values, not just order. So, in your case, what is most likely happening is the distributions of the two data (one for each classification case) sets are "shrinking" towards 0 & 1. This causes it to have bigger errors when it miss-classifies, because instead of giving point i a 70% chance of being in group 1, it gives it a 99% chance, as an example. This will destroy your loss. However, this only marginally effects your AUC.
So, your model appears to be getting better at classification in general if all you care about is correct classification. However, it's getting real cocky and overconfident, causing your loss to increase.