Solved – Method for early stopping in a neural network

machine learningneural networks

What is people's method of early stopping? I run my validation data against the trained network and if the error is > the previous low error then I increment a counter. If the error < the the previous low error then I record the new low and reset the counter. When the counter hits 3 I stop training.

I also continue training if the validation error is better than the training error. Here's my code:

    NetworkTestResult oResult = testANetwork(voNetwork, oTestData, oTestResults);
    System.out.println("Validation Error: " + oResult.getError());

    // is the validation error greater than our lowest result?
    if (oResult.getError() > nLowValidationError) {
        nIncreaseErrorCount++;
    } else {
        nIncreaseErrorCount = 0; // reset
        nLowValidationError = oResult.getError();
    }

    // if validation error is better than training error then carry on training
    if (oResult.getError() < vnTrainError) {
        return true;                
    }

    // if we've seen higher error 3 times then stop training
    if (nIncreaseErrorCount == 3) {
        System.out.println("Early stopping");
        return false;
    }

    return true;

Is this a good method?

Best Answer

There are multiple early stopping strategies. I recommend reading the article Early stopping - but when? by Lutz Prechelt (1997), where many stopping policies are described and evaluated. The one you proposed is often used, even though higher number than three is used.

What is often done in neural network training, however, is that you train for a fixed number of epochs (sufficient for the training error to converge) while recording validation error and saving weight snapshots every few epochs. At the end, you just select the snapshot with lowest validation error.

Also, note that there are many threads on this site describing early stopping.

Related Question