MATLAB: Nnstart and test ROC AUC

machine learningneural networknnstartpattern recognition

Hi all,
I'm doing a project now that requires training a large number of pattern recognizing neural nets. It's my first time using them, so I've been using a "simple script" generated from Matlab's neural network toolbox.
during training, I'm able to look at the ROC for the train, test, and validation sets. I'm wondering if it's possible to somehow extract only the test numeric ROC AUC from the neural network after it's final iteration of training.
Any help would be great!
Thanks, J

Best Answer

 clear all, close all, clc, plt=0;
 tic
 [ x, t ] = simpleclass_dataset;
 [ I N ] = size(x)        % [ 2 1000 ]
 [ O N ] = size(t)        % [ 4 1000 ]
 Ntst = round(0.15*N)     % 150 default

 Nval = Ntst              % 150 default
 Ntrn = N-Nval-Ntst       % 700 default
 Ntrneq = prod(size(t))   % 4000 No. of training equations
 % H = No. of hidden nodes    ( I-H-O NN )
 % Nw = (I+1)*H+(H+1)*O = No. of unknown weights
 % NO OVERFITTING <==> Nw < Ntrneq 
 % NO OVERFITTING <==>  H <= Hub  (upperbound)
 Hub = -1 + ceil( ( Ntrneq-O) / (I+O+1) )  % 570
% For a robust design desire Nw << Ntrneq. Therefore, if possible, choose H << Hub. The PATTERNNET default is H = 10. However, the smaller H, the more robust the design. Obtaining 110 designs ( 10 trials each for H = 0:10 shows that ANY OF THESE VALUES CAN YIELD EXCELLENT RESULTS PROVIDED THE RANDOM WEIGHT INITIALIZATION IS NOT TOO POOR. The complete result can be duplicated below by choosing Hmax = 10 with Ntrials = 10. However, the result below for Hmax = 0 with Ntrials = 4 is sufficient to illustrate the concept.
 Hmin = 0
 dH = 1
 Hmax = 0
 Ntrials = 4
 rng(0)                           % Initialize RNG
 j=0
 for h = Hmin:dH:Hmax
     j=j+1
     if h==0
         net = patternnet([]);;
         Nw = (I+1)*O
     else
         patternnet(h);
         Nw = (I+1)*h+(h+1)*O
     end
     for i = 1:Ntrials
         s(i,j) = rng              % Store state of RNG
         net = configure(net,x,t);
         [ net tr y ] = train(net,x,t);
         tstind = tr.testInd;
         ttst = t(:,tstind);
         ytst = y(:,tstind);
         plt=plt+1,figure(plt)
         plotconfusion(ttst,ytst)
         title([ ' TEST SET CONFUSION MATRIX. TRIAL =  ', num2str(10*(j-1)+i )] )
         hold off
         plt=plt+1,figure(plt)
         plotroc(ttst,ytst)
         title([ ' TEST SET ROC. TRIAL =  ', num2str(10*(j-1)+i )] )
         hold off
     end
 end
 toc                                   % 7.3 sec
Hope this helps.
Thank you for formally accepting my answer
Greg
Related Question