MATLAB: Questions about the regularization (Modified Performance Function) of neural network

validation curve regularization

Hello, everyone. I tried to find out the best regularization ratio for a very simple problem from Matlab, using the function trainbgf for a shallow neural network. Then I plotted a validation curve. The problem is that the curve didn't make any sense. I just followed the contents from the official document as follows:

Here are my codes.

*******************************************

regularization_term = [0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1];

m = size(regularization_term,2);

[x,t] = simplefit_dataset;

x_train = x(1:70);

t_train = t(1:70);

x_test = x(71:94);

t_test = t(71:94);

trainPerformance = zeros(50,11);

testPerformance = zeros(50,11);

for j = 1:50

for i = 1:m

net = feedforwardnet(10,'trainbfg');

net.divideFcn = '';

net.trainParam.epochs = 300;

net.trainParam.goal = 1e-5;

net.performParam.regularization = regularization_term(i);

net = train(net,x_train,t_train);

y_train = net(x_train);

trainPerformance(j,i) = sqrt(perform(net,t_train,y_train));

y_test = net(x_test);

testPerformance(j,i) = sqrt(perform(net,t_test,y_test));

end

plot(regularization_term, mean(trainPerformance),regularization_term,mean(testPerformance))

legend('trainperformance-RMSE','testperformacne-RMSE','best')

xlabel('Regularization Ratio')

ylabel('RMSE')

************************************************

Here is the learning curve I plotted.

I think that the RMSE of the training data should increase as the regularization ratio increases and the RMSE of the test data should decrease at first and at a certain point start to increase as the regularization ratio increases. I'm not sure where I made a mistake, can anyone give me advice? Thank you in advance!

Best Answer

Oh! … O.K.

The simplefit_dataset is smooth with 4 interior local extrema. Therefore, you probably only need H = 4 hidden nodes.

More than H = 4 hidden nodes can be considered overfitting. So, if you use the default H = 10, you will have an overfit net and should implement a mitigation to

 PREVENT OVERTRAINING AN OVERFIT NET.

The most common mitigations are

 1. DO NOT OVERFIT 
 A. No. of unknown weights <= No. of training equations:
       Nw <= Ntrneq
 AND/OR
 B. Minimize weighted sum of SQUARED ERRORS AND SQUARED WEIGHTS 
    MSE + gamma * MSW
 2. DO NOT OVERTRAIN:
 Use a validation subset to implement EARLY STOPPING

Hope this helps.

Greg

Related Solutions

MATLAB: Test, train and validation performance are so different from global performance in the neural network

As per my knowledge, the way you are calculating the trainPerformance, valPerformance & testPerformance may not be correct. Instead I would suggest that you can make use of tr.trainInd, tr.valInd & tr.testInd instead of tr.trainMask, tr.valMask & tr.testMask as follows:

trainPerformance = perform(net,t(tr.trainInd),y(tr.trainInd))
valPerformance = perform(net,t(tr.testInd),y(tr.testInd))
testPerformance = perform(net,t(tr.valInd),y(tr.valInd))

Then w.r.t performance function, according to your above code the performance function you are using is 'crossentropy' and hence perform(net,t,y,ew) returns network performance calculated according to the net.performFcn which is 'crossentropy' & not 'mse'.

Also for your additional part, you may have to use

trp = crossentropy(net,t(tr.trainInd),y(tr.trainInd),{1},'regularization',net.performParam.regularization,'normalization',net.performParam.normalization)

and not

trp = mse(net,trainTargets,y)

Refer to the following resources: net.performFcn, crossentropy, Train Neural Network Using mse Performance Function, Analyze Shallow Neural Network Performance After Training, training record & help(net.performFcn) in command window

MATLAB: I m working with JAFFE dataset.how to divide dataset into training and testing for neural network i m giving LBP feature of JAFFE images as a input.i have a code which divide 70% -30% ratio but how can i know that this 70% is for training,30%testin

As the above code indicates, the default is 70/15/15 . To change it, just change the percentages in the above assignment statements. HOWEVER, I do not recommend changing it !!!

If you use 70/0/30, you are flirting with the potentially disastrous possibility of overtraining an overfit net (i.e., No. of unknown weights, and thresholds, Nw, exceeds the number of training equations Ntrneq)

To mitigate this without a validation set for a regression/curve-fitting net like FITNET, use MSEREG instead of MSE and/or TRAINBR instead of TRAINLM and/or No. of training equations Ntrneq >> No. of unknown weights and biases Nw

HOWEVER, it is not clear to me what to do for a classification/pattern-recognition net like PATTERNNET that uses CROSSENTROPY instead of MSE and TRAINSCG instead of TRAINLM.

   [ I N ] = size(input)
   [ O N ] = size(target)  
 Ntrneq = Ntrn*O 
 Nw      = (I+1)*H+(H+1)*O

Obviously want

 H << (Ntrn*O -O)/ (I+O+1)

Need size(input) to continue for overfitting check

Hope this helps

Thank you for formally accepting my answer

Greg

Best Answer

Related Solutions

MATLAB: Test, train and validation performance are so different from global performance in the neural network

MATLAB: I m working with JAFFE dataset.how to divide dataset into training and testing for neural network i m giving LBP feature of JAFFE images as a input.i have a code which divide 70% -30% ratio but how can i know that this 70% is for training,30%testin

Related Question