You are putting the emphasis on the wrong thing.
The performance estimate of the training subset is highly biased because you are using the same data for training and performance estimation. Because of this, the training subset performance is not used to rank the goodness of multiple designs.
Because designs usually start with random initial weights, typically, multiple designs are created and their performance is ranked using the nontraining validation subset. Since the validation subset is used to stop training when the ability of the net to obtain good performance on the nontraining validation set deteriorates, it's estimate is also biased.
However, the bias of the val subset estimate is relatively small. Therefore it is used to rank multiple designs.
Finally, to obtain an UNBIASED estimate of net performance the NONTRAINING TEST SUBSET is used in the net chosen via the validation subset.
Sometimes the m best designs are combined to form an ENSEMBLE in which the output is a (weighted or unweighted) average of the m nets.
In summary, the training subset and nontraining validation subset are used to obtain and rank multiple designs. However the unbiased estimate of the net performance is obtained by using the 1st ranked net on the nontraining test subset.
Hope this helps.
Thank you for formally accepting my answer
Greg
Best Answer