MATLAB: Does the neural network training end before reaching the specified maximum number of epochs

Deep Learning Toolboxmaxepochsoverfittingstoptraining

Why does the neural network training end before reaching the specified maximum number of epochs?

This is how I am setting the training option:

options = trainingOptions('sgdm', 'MiniBatchSize',miniBatchSize,'MaxEpochs',4000)

But, it looks like the training ended without reaching the max epoch. Is this normal? And what will actually affect the total epoch number in the training?

Best Answer

There are many parameters that can cause a neural network to stop training.

As you may know, an epoch is the full pass of the training algorithm over the entire training set. In general, the training will stop before reaching the specified maximum number of epochs to avoid overfitting to the data, thus improving the network generalization. That is, the training will stop if the results of the cross validation are not getting any better (within some tolerance).

Please refer to the following link for more information on the early stopping behavior to improving generalization of the network:

https://www.mathworks.com/help/nnet/ug/improve-neural-network-generalization-and-avoid-overfitting.html#bss4gz0-32

Related Solutions

MATLAB: Does train function return neural network with the weights of last epoch or the weights of the model with the smallest loss of all epochs

You can refer to Early Stopping & Improve Shallow Neural Network Generalization and Avoid Overfitting documentation.

According to it "When the validation error increases for a specified number of iterations (net.trainParam.max_fail), the training is stopped, and the weights and biases at the minimum of the validation error are returned."

For more information on the "net.trainParam.max_fail" parameter, enter the following in command window:

help nnparam.max_fail

MATLAB: Pretrained Neural network ALEX-NET training process.

I think the model is not overfitting. The validation loss normally decreases during the initial phase of training, as does the training loss. However, when the network begins to overfit the data, the loss on the validation set typically begins to rise and clearly there is not much difference between the training loss and validation loss. You can refer to Improve Shallow Neural Network Generalization and Avoid Overfitting for more understanding of Overfitting and steps to avoid overfitting.

The Validation accuracy can be higher than the training (mini-batch) accuracy, one possible situtation is when the network has layers that behave differently during prediction than during training for example, dropout layers. It also depends on how the training & validation data are split.

Best Answer

Related Solutions

MATLAB: Does train function return neural network with the weights of last epoch or the weights of the model with the smallest loss of all epochs

MATLAB: Pretrained Neural network ALEX-NET training process.

Related Question