MATLAB: How to do cross-validation and calculate the optimum number of hidden neurons in the Neural Networks Toolbox 7.0.2 (R2011b)

Deep Learning Toolbox

How can I do the following in the Neural Networks Toolbox:

1. K-fold cross validation instead of random partition of data. I know that I can do it using specific code, but is this built in to the NNTOOL?

2. Calculate the optimum number of hidden neurons automatically during training of the neural network.

Best Answer

The ability to use k-fold cross validation instead of random partitioning, or to calculate the optimum number of hidden neurons during training, are not available in NNTOOL. As a workaround for cross-validation, you can use CVPARTITION in the Statistics Toolbox.

Related Solutions

MATLAB: How not to normalize the data in patternnet function and how to do K-fold cross validation technique

The optimal approach is to learn the best way to take advantage of existing MATLAB functions and structure. Although it is not necessarily the approach that you proposed, it is no less valid.

For example, by default, MATLAB NN design functions automatically

 1. Normalize input and target data   
 2. Divide the data into training, validation and test subsets.
 3. Initialize default parameter settings including random initial 
    weights
 4. Train the net until one of several conditions occur. For example
    a. Training subset error is reduced to a specified level
    b. Validation subset error increases continually for a 
       specified number (default = 6) of epochs
    c. A maximum number (default = 1000) of epochs is reached.
5. Use the target normalization parameters to 
unnormalize the output data
6. Calculate the performance measure

Simple examples are given in the help and doc documentation examples obtained using the commands

 help patternnet

and

 doc patternnet

Typically, there are two main reasons for unsuccessful efforts:

 1. Inadequate (high or low) specification for the number of hidden nodes
 2. Inappropriate choice of random initial weights.

Consequently, I have devised a double loop approach where

 1. The outer loop is over an interval for the number of hidden nodes
 2. The inner loop is over a number of random initial weight settings

Examples can be found by searching both NEWSGROUP and ANSWERS using search words that include

GREG PATTERNNET

 GREG PATTERNNET TUTORIAL

Hope this helps

Thank you for formally accepting my answer

Greg

MATLAB: Trial-and-error or K-fold cross-validation

If you search in both the NEWSGROUP and ANSWERS you will see zillions of examples of my two loop solution:

%Outer loop over number of hidden nodes, e.g.,

 rng(0), j=0
 for h = Hmin:dH:Hmax
     j = j + 1
     net = fitnet(h);
     etc ...

%Inner loop over Ntrials sets of random initial weights

     for i = 1:Ntrials
         net = configure(net,x,t);
         etc ...

Hope this helps.