I am trying to reproduce the results of a neural network created by "patternnet" function, using the "feedforwardnet" function instead. I fix
- the architecture
- the initialization method, and
- the seed in the randStream (for generating random numbers)
to be the same. However I still can't get identical results. Although I use the same initialization function I found that the difference is in the initial weights and biases. why are the weights and biases initialized differently? Debugging the functions I found that the "init" function is called more times than expected using the "patternnet" function. Why is this?
I post some code and results to prove that random number generator is not an issue. (A similar question has been asked here, but the problem has not been resolved).
TrainFeatNN = SelectedFeat40.Train;TrainObsNN = trainToReduce.ObsClasses(:,2:3);testInd = false(size(SelectedFeat40.Train,1),1);testInd2 = false(size(SelectedFeat40.Train,1),1);testInd3 = false(size(SelectedFeat40.Train,1),1);testInd4 = false(size(SelectedFeat40.Train,1),1);posclass = 1;x = TrainFeatNN';t = TrainObsNN'; %
%%%Neural Networks patternnet #2
%%RandStream.setGlobalStream (RandStream ('mrg32k3a','Seed', 1234));trainFcn = 'trainscg';hiddenLayerSize = 10;net = patternnet(hiddenLayerSize, trainFcn);net.input.processFcns = {'removeconstantrows','mapminmax'};net.output.processFcns = {'removeconstantrows','fixunknowns'};net.divideFcn = 'divideblock'; net.divideMode = 'sample'; net.divideParam.trainRatio = 60/100;net.divideParam.valRatio = 20/100;net.divideParam.testRatio = 20/100;net.trainParam.max_fail = 6;net.trainParam.epochs = 1000;net.performFcn = 'crossentropy'; [net,tr] = train(net,x,t);y = sim(net,x);PredProb_NN = transpose(y(2,tr.testInd));testInd(tr.testInd) = true;[XNN,YNN,~,AUC_NN] = perfcurve(TrainObsNN(testInd,end),PredProb_NN, posclass);display(sprintf('Method: patternnet (#1): \nBest Epoch: %d,\nBest performance: %.4f,\nAUC: %.4f\n',tr.best_epoch,tr.best_perf,AUC_NN)%%%%Neural Networks feedforwardnet #1
%%RandStream.setGlobalStream (RandStream ('mrg32k3a','Seed', 1234));trainFcn = 'trainscg';hiddenLayerSize = 10;net2 = feedforwardnet(hiddenLayerSize,trainFcn);net2.performFcn = 'crossentropy';net2.outputs{numel(hiddenLayerSize)+1}.processParams{2}.ymin = 0;net2.layers{net2.numLayers}.transferFcn = 'softmax';net2.plotFcns = {'plotperform','plottrainstate','ploterrhist','plotconfusion','plotroc'};net2.initFcn='initlay';net2.layers{1}.initFcn='initnw';net2.layers{2}.initFcn='initnw';net2.layers{1}.transferFcn = 'tansig';net2 = init(net2);net2.input.processFcns = {'removeconstantrows','mapminmax'};net2.output.processFcns = {'removeconstantrows','fixunknowns'};net2.divideFcn = 'divideblock'; net2.divideMode = 'sample'; net2.divideParam.trainRatio = 60/100;net2.divideParam.valRatio = 20/100;net2.divideParam.testRatio = 20/100;net2.trainParam.max_fail = 6;net2.trainParam.epochs = 1000;[net2,tr2] = train(net2,x,t);y2 = sim(net2,x);PredProb_NN2 = transpose(y2(2,tr2.testInd));testInd2(tr2.testInd) = true;[XNN2,YNN2,~,AUC_NN2] = perfcurve(TrainObsNN(testInd2,end), PredProb_NN2, posclass);display(sprintf('Method: feedforwardnet (#1): \nBest Epoch: %d,\nBest performance: %.4f,\nAUC: %.4f\n',tr2.best_epoch,tr2.best_perf,AUC_NN2))%%%%Neural Networks patternnet #2%%RandStream.setGlobalStream (RandStream ('mrg32k3a','Seed', 1234));trainFcn = 'trainscg';hiddenLayerSize = 10;net3 = patternnet(hiddenLayerSize, trainFcn);net3.input.processFcns = {'removeconstantrows','mapminmax'};net3.output.processFcns = {'removeconstantrows','fixunknowns'};net3.divideFcn = 'divideblock'; net3.divideMode = 'sample'; net3.divideParam.trainRatio = 60/100;net3.divideParam.valRatio = 20/100;net3.divideParam.testRatio = 20/100;net3.trainParam.max_fail = 6;net3.trainParam.epochs = 1000;net3.performFcn = 'crossentropy';[net3,tr3] = train(net3,x,t);y3 = sim(net3,x);PredProb_NN3 = transpose(y3(2,tr3.testInd));testInd3(tr3.testInd) = true;[XNN3,YNN3,~,AUC_NN3] = perfcurve(TrainObsNN(testInd3,end), PredProb_NN3, posclass);display(sprintf('Method: patternnet (#2): \nBest Epoch: %d,\nBest performance: %.4f,\nAUC: %.4f\n',tr3.best_epoch,tr3.best_perf,AUC_NN3))%%%%Neural Networks feedforwardnet #2
%%RandStream.setGlobalStream (RandStream ('mrg32k3a','Seed', 1234));trainFcn = 'trainscg';hiddenLayerSize = 10;net4 = feedforwardnet(hiddenLayerSize,trainFcn);net4.performFcn = 'crossentropy';net4.outputs{numel(hiddenLayerSize)+1}.processParams{2}.ymin = 0;net4.layers{net4.numLayers}.transferFcn = 'softmax';net4.plotFcns = {'plotperform','plottrainstate','ploterrhist','plotconfusion','plotroc'};net4.initFcn='initlay';net4.layers{1}.initFcn='initnw';net4.layers{2}.initFcn='initnw';net4.layers{1}.transferFcn = 'tansig';net4 = init(net4);net4.input.processFcns = {'removeconstantrows','mapminmax'};net4.output.processFcns = {'removeconstantrows','fixunknowns'};net4.divideFcn = 'divideblock'; net4.divideMode = 'sample'; net4.divideParam.trainRatio = 60/100;net4.divideParam.valRatio = 20/100;net4.divideParam.testRatio = 20/100;net4.trainParam.max_fail = 6;net4.trainParam.epochs = 1000;[net4,tr4] = train(net4,x,t);y4 = sim(net4,x);PredProb_NN4 = transpose(y4(2,tr4.testInd));testInd4(tr4.testInd) = true;[XNN4,YNN4,~,AUC_NN4] = perfcurve(TrainObsNN(testInd4,end), PredProb_NN4, posclass);display(sprintf('Method: feedforwardnet (#2): \nBest Epoch: %d,\nBest performance: %.4f,\nAUC: %.4f\n',tr4.best_epoch,tr4.best_perf,AUC_NN4))%%if ~isequal(tr.testInd,tr2.testInd) || ~isequal(testInd,testInd2) || ... ~isequal(tr.testInd,tr3.testInd) || ~isequal(tr.testInd,tr4.testInd) warning('The test sets are not the same!')endif isequal(y,y3) && isequal(y2,y4) display('Random number generation is *not* an issue.')else warning('There is a random number generator issue')endif isequal(y,y2) || isequal(y3,y4) display('The two networks are identical')else warning('The two netowrks are *not* identical!')end
Output:
Method: patternnet (#1): Best Epoch: 122, Best performance: 0.2379, AUC: 0.7348
Method: feedforwardnet (#1): Best Epoch: 136, Best performance: 0.2380, AUC: 0.7331
Method: patternnet (#2): Best Epoch: 122, Best performance: 0.2379, AUC: 0.7348
Method: feedforwardnet (#2): Best Epoch: 136, Best performance: 0.2380, AUC: 0.7331
Random number generation is not an issue. Warning: The two netowrks are not identical!
Best Answer