MATLAB: Pattern recognition neural network : training process performance question

neural networkpattern recognition

I am just starting using neural networks, but I am troubled by something when I use patternnet function. I have a set of about 200 data, each having 5 parameters (input is 5×200). I want the NN to classify those data into 7 classes, so my target data is a 7×200 matrix (eg. [0; 0; 1; 0; 0; 0; 0]). These data are quite "noisy".
net = patternnet(20);
net = train(net,inputs,targets);
My problem is that the results vary each time I restart the training process. What I mean is that if I delete this network and build another one with the same characteristics and the same inputs/targets, the performance of the NN vary greatly.
To give a better idea of my results, I can obtain 91% good calls (with the confusion matrix) one time, and the next time, it is 28%…
Is this normal, or is it a problem with the characteristics of the NN, or could It be my data?
Like I said, I just started using NN, and maybe it's the type of NN that is not correct for my situation, but I thought NN would be more steady in the training process.
Thanks.

Best Answer

Initial weights are chosen randomly. In order to duplicate previous runs you have to reset the RNG to the same initial state. I use a famous birthdate via rng(4151941). However, there are newer (better)ways to intialize the RNG, e.g. help/doc randstream.
As you have experienced, the ability to find a good local minimum is random. This is normal. My approach is to run Ntrials = 10 to 30 trials for every candidate value of H = numhidden (double for loop) and store the results in 3 matrices (trn/val/tst) with size(results) = [ Ntrials H] . My training goal is usually ~mean(var(t'))/100 (R^2 > ~0.99) where t is the training target with unit matrix columns, e.g., [ 0 0 0 1 0 0 ]'.
I then choose a net with the smallest trial value of H that achieves the minimum test set classification rate.
You may want to form an ensemble by combining several good designs. See design examples in the Newsgroup by searching on
heath newff Ntrials
Also try newfit, fitnet, patternnet & feedforwardnet.
Hope this helps.
Greg
Related Question