I was researching k-fold cross-validation, and read that one should train on k-1 of the k partitions and test on the remaining partition, and then repeat for each partition, averaging the results to get an estimate of model performance. This I understand; however, if there is no validation dataset, when should I stop the training (since early stopping is not possible, and so the model cannot simply be trained until generalisation performance starts to worsen)? I.e. should training be stopped after a set number of epochs, or when the gradient falls below a certain limit? Are there any tips for what these stopping parameters should be?
Solved – Neural network training without early stopping
cross-validationneural networks
Related Solutions
You can use cross-validation to trigger early stopping. Basically it involves splitting the data in a train an validation set. After a set of iterations using the training data you can check if the extra iterations resulted in better performance on the validation set. If so, continue training, if not, consider stopping.
If you start with very small parameters, this can provide a form of regularization.
In practice this criterion is not always smooth; the performance on the validation set tends to go up and down. For more in depth information see: http://page.mi.fu-berlin.de/prechelt/Biblio/stop_tricks1997.pdf
What would be a good validation frequency? Should I check my model on the validation data at the end of each epoch? (My batch size is 1)
There is no gold rule, computing the validation error after each epoch is quite common. Since your validation set much smaller than your training set, it will not slow down the training much.
Is it the case that the first few epochs might yield worse result before it starts converging to better value?
yes
In that case, should we train our network for several epochs before checking for early stopping?
You could, but then the issue is how many epochs should you skip. So in practice, most of the time people do not skip any epoch.
How to handle the case when the validation loss might go up and down? In that case, early stopping might prevent my model from learning further, right?
People typically define a patience, i.e. the number of epochs to wait before early stop if no progress on the validation set. The patience is often set somewhere between 10 and 100 (10 or 20 is more common), but it really depends on your dataset and network.
Example with patience = 10:
Best Answer
If there is no validation set, make one: from the training fold keep a few samples out and use them for early stopping.
Other options are: