Question
What are good reference articles/blogs/tutorials to learn how to intepret learning curves for deep convolutional neural networks?
Background
I am trying to apply convolutional neural networks (CNN) for vessel segmentation (specifically to determine whether or not the center pixel of an image patch is on a vessel) using caffe.
I have about 225000 training images (~50% positive) and 225000 (~50% positive) testing/validation images.
My input images are of size 65 x 65. I have four convolutional layers (48x6x6, 48x5x5, 48x4x4, 48x2x2) each followed by 2×2 max-pooling layers , one fully connected layer of 50 neurons, and a final scoring layer with 2 neurons. My training batch size is 256 and my testing batch size is 100.
I am using a stochastic gradient descent optimizer (SGD) and an inverse decay learning rate policy. Below are my caffe solver parameters:
- type: "SGD"
- base_lr: 0.01
- lr_policy: "inv"
- gamma: 0.1
- power: 0.75
- momentum: 0.9
- weight_decay: 0.0005
Below is the learning curve i am getting:
I am using the cross entropy classification loss or multinomial logistic loss (see here).
I would like to hear how people interpret this learning curve and what parameters they would change to try and improve the test accuracy.
-
The training loss is decreasing but decreasing very slowly. Could it mean that my learning rate is low?
-
On the contrary i see that the test loss decreases quickly at first and then slows down. Could this mean that my learning rate was high and it got stuck in the local minimum?
-
Also the test accuracy has stabilized and stopped increasing too soon. Could this mean that i have to try and increase my model capacity or decrease my regularization?
In general, what would help me and also probably others is if someone could point out a reference article/book/blog-post that delves deeply into the interpretation of such learning curves with many example cases.
I found this blog post which was very helpful but there is not much about the interpretation of learning curves (atleast not to my satisfaction).
Best Answer
2 things:
I hope I was clear enough, good luck in your work :)