MATLAB: Bootstraping for neural network

neural network

hi every body
I am using bootstrap sampling with neural network to select the best network architecture. For instance, if I use 30 bootstraping for each network, I would have 30 error for each training and test data related to them. To select the best model, should I use the variance of these errors and select the model with the lowest variance? and should the training and test error be considered separately for bootstrapping? Any other suggestion is welcomed.

Best Answer

Training errors are biased because the same data is used for creation and evaluation.
Attempts to mitigate the bias involve reducing the number of training equations, Ntrneq, by the number of estimated unknown weights, Nw, to obtain the number of estimation degrees of freedom, Ndof ( see Wikipedia/R-squared), that is used to obtain the adjusted mean-squared-error estimate:
[Ntrn O ] = size(ttrn) % size of the training target matrix
Ntrneq = prod(size(ttrn)) = Ntrn*O % Number of training equations
etrn = ttrn-ytrn; % Training set errors
SSEtrn = ssetrn(etrn) % Sum-squared-errors
MSEtrn = SSEtrn/Ntrneq % Mean-squared-error estimate (biased)
Ndof = Ntrneq-Nw
MSEtrna = SSEtrn/Ndof % "a"djusted mean-squared-error estimate
Although I would only rely on the test set estimates and standard deviations, I would also keep track of the estimates and standard deviations of MSEtrn and MSEtrna.
Hope this helps.
Thank you for formally accepting my answer
Greg
Related Question