Good again, I followed by testing and the truth I'm a little confused. By plotting the predicted values in closeloop confuses me a bit and I can not understand the results, When I get low is when R2tst generalizes better because if I get high values both in train, validation and test accuracy is great I just suspect overfitting occurs because it is very similar and it is displaced with respect to the graph of original targets. Practically the resusltados I posted in the previous post. If anyone can explain this I would really appreciate it as I think is the last step and the one that deserves special attention in order to interpret the data correctly. Thank you very much.
MATLAB: New questions and valuation of final results
Deep Learning Toolboxnarx
Related Solutions
newff is obsolete. Use fitnet instead.
I think you wasted too much time on finding parameters. Typically, you should use defaults for everything except
1. Set the initial state of the random generator with your favorite seed. For example
rng('default')2. Search for the smallest acceptable number of hidden nodes using the outer loop search
h = Hmin:dH:Hmax3. Search for a suitable combination of random initial weights and train/val/test data divisions using the inner loop search
i = 1:Ntrials4. Typically, I start with ~ 10 values of h and 10 weight/datadivision trials for each value of h.
5. The documentation example for fitnet (the one for newff is similar)
help fitnet doc fitnet [x,t] = simplefit_dataset; net = fitnet(10); net = train(net,x,t); view(net) y = net(x); perf = perform(net,t,y)6. However, in general, my approach yields fewer failures. I have posted scores of examples in the NEWSGROUP and ANSWERS. Try searching with (or newff)
greg fitnet Ntrials7. Two very important things to understand are that
a. Increasing the number of hidden nodes makes it easier to obtain a solution. However, the smaller the number of hidden nodes, the better the net resists noise, interference, measurement errors and transcription errors. Just as (or more) importantly, the net performs better on nontraining (validation, test and unseen) data. b. Because initial weights and datadivision are random, the design may fail when all of the other parameters are perfect. Thus the double loop search.Hope this helps.
Thank you for formally accepting my answer
Greg
% 1. Selected ending semicolons can be removed to aid debugging
[P, T ] = simplenarx_dataset; whos p= cell2mat(P); t = cell2mat(T); ID = 1:1 FD = 1:1 H = 5 NID= length(ID) NFD=length(FD) Nw = (NID*I+NFD*O+1)*H+(H+1)*O% 2. Use NID and NFD for Nw in case delays are not single
% 3. No need to use tonndata because the simplenarx_data set is instantly ready for preparets.
% 4. No need for (p1,t1) and (p2,t2). Delete both.
% 5. Input delays are suboptimal. Did you try to find the significant lags of the target/input cross-correlation function?
% 6. Feedback delays are suboptimal. Did you try to find the significant lags of the target autocorrelation function?
% 7. H is suboptimal. Was it chosen using the suboptimal delays? If so, please explain how.
rng(0) net = narxnet(ID,FD,H); [inputs,inputStates,layerStates,targets] = preparets(net,P,{},T); whos P T inputs inputStates layerStates targets%8. N=N-1: DELETE. NOT A GOOD IDEA TO USE A VARIABLE OR PARAMETER NAME ON BOTH SIDES OF AN EQUATION. BESIDES, PREPARETS OUTPUTS THE CORRECT DIMENSIONS
[ I N ] = size(inputs) [ O N ] = size(targets)% 9. No need for ttotal it should be the same as targets. % No need for Neq, MSE00,MSE00a and t4. Delete
net.divideFcn='divideblock'; [trainInd,valInd,testInd] = divideblock(N,0.7,0.15,0.15); ttrn = targets(trainInd); tval = targets(valInd); ttest = targets(testInd); Ntst = length(ttrn) Nval = length(valInd) Ntst = length(testInd) Ntrneq = prod(size(ttrn)) % Ntrn*O
Ndof = Ntrneq-Nw%Naive Constant Output Model
ytrn00= mean(ttrn,2); Nw00 = size(ytrn00,2) Ndof00 = Ntrneq-Nw00 MSEtrn00 = sse(ttrn-ytn000)/Ntrneq MSEtrn00=mean(var(ttrn,1)) MSEtrn00a = sse(ttrn-ytrn00)/Ndof00 MSEtrn00a=mean(var(ttrn,0))%9. MSEval00 and MSEtst00 should be obtained from the Naive constant output model output
MSEval00 = mse(tval-ytrn00) MSEtst00 = mse(tttst-ytrn00) net.trainParam.goal = 0.01*Ndof*MSEtrn00a/Ntrneq; % R2trna >= 0.99
rng(0) [net,tr,Ys,Es,Xf,Af] = train(net,inputs,targets,inputStates,layerStates); outputs = net(inputs,inputStates,layerStates); errors = gsubtract(targets,outputs); MSE = perform(net,targets,outputs); MSEa=Neq*MSE/(Neq-Nw) R2=1-MSE/MSE00 R2a=1-MSEa/MSE00a% 10. The DOF "a"djustment is only applied to the training data % 11. Can delete the last 6 equations that refer to all of the data instead of the trn/val/tst division.
MSEtrn=tr.perf(end) MSEtrna = Ntrneq*MSEtrn/Ndof MSEval=tr.vperf(end) MSEtst=tr.tperf(end)% 12.Using "end" is only valid if training converges because of tr.min_grad (not valstop ). Better to use "tr.best_epoch".
R2trn=1-MSEtrn/MSEtrn00 R2trna=1-MSEtrna/MSEtrn00a%13 Original MSEtrna misprint.
R2val=1-MSEval/MSEval00 R2tst=1-MSEtst/MSEtst00and my results are:
% 14. Unable to compare results because you did not intialize the RNG before the first call of the RNG in the net creation command net = ... where H =5 random weights were assigned to the input bias. I will use rng(0) before the net creation.
I will do the closeloop part next.
Thank you for formally accepting my answer
Greg
Related Question
- How can i improve the performance of a closed loop NARX neural network
- MLP Neural network and k-fold cross validation
- How to get better test error/accuracy with neural networks
- R^2 for comparing the results of measured vs. predicted (non-linear model) values
- A DIFFICULT NEURAL NETWORK
- Precisions obtained with network openloop NarX
- Retrain neural networks and different forecasts
- Precision network analysis NarX
Best Answer