I have seen in many machine learning papers and talks, people refer to loss convergence. I assume they refer to loss on development set, but, what I am not sure about is that if they mean when they run the training code, at each epoch of the training procedure the loss drops, Or they mean if they run their training code several times, it always ends up at the same loss value on the development set?
I think given the complexity of the objective functions (in many cases) it is very unlikely to end up with the same loss every time training is done. So perhaps they mean it drops between each consecutive epochs. Is this correct?
Best Answer
Yes you are correct. The usual way to train a neural network is to train the same network on several epochs.
The training stops when a certain number of epochs is attained or when an early stopping criterion is attained: when the loss on the validation set stops decreasing.