MATLAB: Retrain neural networks and different forecasts

neural networkneural networks

Dear all,
Thank you in advance for your help.
I have prepared the following script in order to train a NN and then use it for forecasting.
     trainFcn = 'trainlm';  % Levenberg-Marquardt backpropagation.
      % Create a Fitting Network
    hiddenLayerSize = 3;
    net = fitnet(hiddenLayerSize,trainFcn);
    net.input.processFcns = {'removeconstantrows','mapminmax'};
    net.output.processFcns = {'removeconstantrows','mapminmax'};
    net.divideFcn = 'dividerand';  % Divide data randomly
    net.divideMode = 'sample';  % Divide up every sample
    net.divideParam.trainRatio = 70/100;
    net.divideParam.valRatio = 15/100;
    net.divideParam.testRatio = 15/100;
    net.performFcn = 'mse';  % Mean Squared Error
    net.plotFcns = {'plotperform','plottrainstate','ploterrhist', ...
        'plotregression', 'plotfit'};
    % Train the Network
    [net,tr] = train(net,x,t);
    % Test the Network
    y = net(x);
    e = gsubtract(t,y);
    performance = perform(net,t,y)
    % Recalculate Training, Validation and Test Performance
    trainTargets = t .* tr.trainMask{1};
    valTargets = t .* tr.valMask{1};
    testTargets = t .* tr.testMask{1};
    trainPerformance = perform(net,trainTargets,y)
    valPerformance = perform(net,valTargets,y)
    testPerformance = perform(net,testTargets,y)
    % Forecast
    fv = net(xv);
  1. Each time I run the script I get the NN is retrained and produces different results (similarly to pressing the retrain button in the GUI).
  2. Each time the training is very good with test and validation performance between 0.025-0.035 however when I use the NN to forecast the out-of-sample data I get completely different results. For example the Mean Absolute Error can be between 10%-25% while the times the absolute error is less than 20% is between 60%-86%.
  3. If I clear the memory I get the same results (and if I clear and then keep executing the script I get the same results in the same order).
I can imagine that this is due to local minima and the split of the training/validation and test data. Is that correct?
And the second question is which NN do I keep since all of them in-sample perform very well and similarly (comparing the performance criteria, MSE and R^2) ?
Any suggestion in order to improve the methodology and/or stabilising the results?

Best Answer

I have posted zillions of double loop solutions in both the NEWSGROUP & ANSWERS.
The outer loop is over number of hidden nodes and the inner loop is over Ntrials random weight initializations AND trn/val/tst data divisions.
Typically, assignment statements that assign default values are omitted.
Each loop is over 10 values so that 100 nets are designed.
rng('default')    % for reproducibility
j=0
for h = Hmin:dH:Hmax
   j=j+1
   .....
   for i = 1:Ntrials
    ...
   end
end
I usually print the resulting NMSE or Rsquare values in four (total, training, validation and testing) matrices of size 10 x 10.
To view some of the results, search in both forums using
 Hmin:dH:Hmax Ntrials
Hope this helps.
Thank you for formally accepting my answer
Greg
Related Question