MATLAB: How many neurons in the hidden layer??

emotion recognitionhidden layer

Number of neurons in the input layer of my feed-forward network is 77, number of neurons in output layer is 7, I want to use multiple hidden layers, How many neurons, Should I keep in each hidden layer from first to last between input and output layer

Best Answer

For a given performance on training data, the fewer the number of weights, the better. Then the net will better generalize to validation, test and unseen data with the same statistical properties.

Since universal approximators can be made with a single hidden layer. Most default configurations have an I-H-O node topology for I dimensional inputs, O-dimensional outputs and H hidden nodes. The resulting number of unknown weights is

 Nw = (I+1)*H+(H+1)*O

With Ntrn input/target training pairs, the number of training equations is

 Ntrneq = Ntrn*O.

For the number of unknowns to not exceed the number of equations, Nw <= Ntrneq which yields the following upper bound on the number of hidden nodes

 H <= Hub = (Ntrneq-O)/(I+O+1)

If possible, increase stability, robustness and generalization with Nw << Ntrneq which yields.

   H << Hub

It is very doubtful that you need 77 inputs. Using more than necessary will increase the number of weights and, typically, destabilize your design. Therefore, you should first reduce the number of inputs before trying to reduce the number of hidden nodes.

The choice of variables obtained from the linear STEPWISE and STEPWISEFIT model functions are, typically relevant.

Hope this helps.

Thank you for formally accepting my answer

Greg

PS I should have a number of posts in the NEWSGROUP and ANSWERS which are relevant.

Related Solutions

MATLAB: Layer in neural network

Two of the most important uses for Neural Networks are regression(function approximation, curvefitting) and classification/pattern-recognition.

The best way to design NNs is via the Neural Network Toolbox

In general, the best function to use for regression is FITNET. Start with the help and doc documentation using the command line commands

 help fitnet
 doc fitnet

Next check out some of my posts on the NEWSGROUP and ANSWERS. Search using

 greg fitnet

In general, the best function to use for classification/pattern-recognition is PATTERNNET. Start with the help and doc documentation using the command line commands

 help patternnet
 doc patternnet

Next check out some of my posts on the NEWSGROUP and ANSWERS. Search using

 greg patternnet

Finally, use MATLAB data to practice with

 help nndatasets
 doc nndatasets

If you have problems, it is easier for us to help if you use those datasets.

The help and doc examples use as many default settings as possible. Therefore my best advice for beginners is to begin your designs similarly with one exception. Since by default, the trn/val/tst data division and initial weights are chosen randomly, it is important to know the initial state of the random number generator. The best way to do that is for you to set it to your favorite state before using TRAIN. If designing multiple designs in a loop, initialize RNG before the loop.

 help rng
 doc rng

The default topology is I-H-O ( H=10 default) for I dimensional input vectors and corresponding O-dimensional Output target vectors. The one hidden layer contains H=10 hidden nodes containing tansig (tanh) transfer functions.

The one hidden layer net is a universal approximator. Increasing the number of hidden nodes increases the ability to model more complex functions. However using more than needed decreases the ability to operate well on unseen data. The only reason to use more layers is if you want to decrease the total number of hidden nodes or, to try to model subclasses for classification.

 hiddenLayerSize=200

Replaces the 10 tansig hidden units with 200. Typically, this is a ridiculous number.

 net.divideParam.trainRatio=70/100 
 net.divideParam.valRatio=15/100 
 net.divideParam.testRatio=15/100

Divides the data into training, validation and test subsets with the ratio 70/15/15. However, this is the default ratio. Therefore those statements can be deleted.

Hope this helps.

Thank you for formally accepting my answer

Greg

MATLAB: How to know the number of the hidden nodes in ANN

[I N ] = size(input)         % size("I"nput)
[O N ] = size(target)        % size("O"utput)
Ntrn   = N - 2*round(0.15*N) % Default no. of training examples ~ 0.7*N
Ntrneq = Ntrn*O              % No. of training equations 
Nw     = (I+1)*H +(H+1)*O    % No. of unknown weights for H = number of hidden nodes

Overfitting ( Nw > Ntrneq ) allows decreased performance on nontraining (e.g., val, test and unseen) data

Since H > ( Ntrneq -O )/(I+O+1) for overfitting, one training strategy is H <= Hub or preferably, H << Hub where

     Hub = -1 + ceil( (Ntrneq - O) / (I+O+1) ) % integer H

My training strategy:

Minimize the number of hidden nodes subject to the constraint that the mean-square-error is less than 1 percent of the mean target variance

     net.trainPerform.goal = 0.01*mean(var(target',1)))

Very often this is accomplished by trial and error subject to

     0 <= Hmin <= H <= Hmax <= Hub

However, sometimes it is necessary to exceed Hub. The mitigation for this is

     Validation Stopping and/or regularization (e.g., msereg or TRAINBR}

Hope this helps.

Thank you for formally accepting my answer

Greg

Best Answer

Related Solutions

MATLAB: Layer in neural network

MATLAB: How to know the number of the hidden nodes in ANN

Related Question