This is my first question here, so forgive me if I'm providing too little or too much detail.
I am training a CNN for image classification of fog conditions on camera images. I have three classes: No fog, light fog and dense fog.
To train this model I use PyTorch, where I finetune a pre-trained Resnet-18 model that was trained on ImageNet data on my data. Specifically, I've removed the last Fully Connected layer and replaced it to train for a 3-class classification problem. Learning rate = 0.001.
The data was randomly split into 80% train data and 20% test data. Furthermore, there is a large class imbalance in my data, meaning that there are 152.333 images with no fog, 1187 images with light fog and 604 images with dense fog. To try and counteract this problem I have used PyTorch's WeightedRandomSampler, which makes sure that there is class balance in every training batch. I've checked this, and an equal amount of the three classes are shown in every batch.
Now comes my actual 'problem': The validation loss is lower than the training loss at the very first epoch and continues to stay lower. Also, this validation loss (and accuracy) seems to randomly fluctuate but always stays high. Here are the loss curves of the model training:
I cannot figure out what might be causing this strange behaviour. I thought that maybe the model was predicting most of the validation images to be the majority class, but this confusion matrix shows that it is actually not doing this:
0 = no fog
1 = light fog
2 = dense fog
The images were taken by 8 cameras that are static and thus shoot images of the same scenery all the time. Randomcrop and RandomHorizontalFlip are used for the training set.
Does anybody have any clue as to what might be causing this strange behaviour of the validation loss, or is this maybe something that is actually O.K. when training a CNN for image classification?
Best Answer
You need to use the same loss for training and validation if you want to compare them. Your training loss is taking into account sample weights (generated by the
WeightedRandomSampler) in order to compensate the class inbalance: misclassification of minority class samples is penalized much more than misclassification of majority class samples.In the validation loss, you are not doing any correction like this: all classes are penalized equally. From the confusion matrix it is obvious that the validation set is very inbalanced, therefore a small change on the performance on the class 0 results in a large change in the validation loss curve (simply because it gets multiplied by the number of samples of class 0).
The solution is to use weighted loss function also for validation and assign a weight $1/N_k$ to each sample, where $N_k$ is the number of samples of class $k$ in the validation set.