I'm training a 2-layer CNN model on audio samples, represented as CQT. There are ≈160k samples, many that are very similar since they originate from the same instrument and/or audio file. 10% have been split out beforehand for validation. My question is, why does my validation loss go up, while the validation accuracy goes up as well. A typical example can be seen in the image below.
The model roughly looks like (conv/pool/relu)x2 -> flatten/dense -> dense/softmax. Categorical crossentopy as cost function.
The phenomena occurs both when validation split is randomly picked from training data, or picked from a completely different dataset. The only way I managed it to go in the "correct" direction (i.e. loss goes down, acc up) is when I use L2-regularization, or a global average pooling instead of the dense layers.
Best Answer
There could be a lot going on here I'm going to give a layman's answer.
The accuracy is going to be how many correct observations out of all observations. If this is a classification problem and the classes are not balanced (120k : class1, 40k : class 2) then it is easily able to get a high accuracy by just picking class 1 more often. This is simple math 120k/160k.
The loss function can be more complex and can be more/less robust to class imbalances.
What I'm trying to say is that the functions for accuracy and loss aren't the same and thus can and will deviated depending on which once you are comparing. It will also depend on how well the classifier is doing for each class. It might be good to look at a confusion matrix if you don't have too many classes.
The scale isn't too extreme for the loss function either. The delta from the lowest to the highest loss is about 0.3. I don't have the experience to know if that is a large difference, but it might be something to consider.