MATLAB: NN accuracy on test set low

best netDeep Learning Toolboxneural networkpattern recognition

I have implemented a neural network in Matlab R2013a for character recognition. I have used trainbr function for nn training. 80% samples were used for training and the rest for testing. When i plot the confusion matrix, i get 100% accuracy on the training set. But for the test set the accuracy is very low(around 60%). What could be possibly wrong?

Best Answer

See my example at

http://www.mathworks.com/matlabcentral/newsreader/view_thread/334932#919949

Hope this helps.

Greg

Related Solutions

MATLAB: How to calculate the error in Neural Network Back Propagation Network

Use as many defaults as possible

 MSEgoal = 0.01*mean(var(target',1))
 MinGrad = MSEgoal/100
 net = newff(input,target);
 net.trainParam.goal     = MSEgoal;
 net.trainParam.min_grad = MinGrad;
 [ net tr output error ] = train(net, input, target);

For details, search ANSWERS using the search word msegoal

Hope this helps.

Greg

Thank you for formally accepting my answer

Greg

MATLAB: Nnstart and test ROC AUC

 clear all, close all, clc, plt=0;
 tic
 [ x, t ] = simpleclass_dataset;
 [ I N ] = size(x)        % [ 2 1000 ]
 [ O N ] = size(t)        % [ 4 1000 ]
 Ntst = round(0.15*N)     % 150 default

 Nval = Ntst              % 150 default
 Ntrn = N-Nval-Ntst       % 700 default
 Ntrneq = prod(size(t))   % 4000 No. of training equations
 % H = No. of hidden nodes    ( I-H-O NN )
 % Nw = (I+1)*H+(H+1)*O = No. of unknown weights
 % NO OVERFITTING <==> Nw < Ntrneq 
 % NO OVERFITTING <==>  H <= Hub  (upperbound)
 Hub = -1 + ceil( ( Ntrneq-O) / (I+O+1) )  % 570

% For a robust design desire Nw << Ntrneq. Therefore, if possible, choose H << Hub. The PATTERNNET default is H = 10. However, the smaller H, the more robust the design. Obtaining 110 designs ( 10 trials each for H = 0:10 shows that ANY OF THESE VALUES CAN YIELD EXCELLENT RESULTS PROVIDED THE RANDOM WEIGHT INITIALIZATION IS NOT TOO POOR. The complete result can be duplicated below by choosing Hmax = 10 with Ntrials = 10. However, the result below for Hmax = 0 with Ntrials = 4 is sufficient to illustrate the concept.

 Hmin = 0
 dH = 1
 Hmax = 0
 Ntrials = 4
 rng(0)                           % Initialize RNG
 j=0
 for h = Hmin:dH:Hmax
     j=j+1
     if h==0
         net = patternnet([]);;
         Nw = (I+1)*O
     else
         patternnet(h);
         Nw = (I+1)*h+(h+1)*O
     end
     for i = 1:Ntrials
         s(i,j) = rng              % Store state of RNG
         net = configure(net,x,t);
         [ net tr y ] = train(net,x,t);
         tstind = tr.testInd;
         ttst = t(:,tstind);
         ytst = y(:,tstind);
         plt=plt+1,figure(plt)
         plotconfusion(ttst,ytst)
         title([ ' TEST SET CONFUSION MATRIX. TRIAL =  ', num2str(10*(j-1)+i )] )
         hold off
         plt=plt+1,figure(plt)
         plotroc(ttst,ytst)
         title([ ' TEST SET ROC. TRIAL =  ', num2str(10*(j-1)+i )] )
         hold off
     end
 end
 toc                                   % 7.3 sec

Hope this helps.

Thank you for formally accepting my answer

Greg

Best Answer

Related Solutions

MATLAB: How to calculate the error in Neural Network Back Propagation Network

MATLAB: Nnstart and test ROC AUC

Related Question