MATLAB: Precision network analysis NarX

Deep Learning Toolboxnarx

Well I think I know how to do it: in the matrix of new inputs and new targets I can go poking above the predicted outputs and by looping multiple outputs could get ahead, because as I've seen in a code, to perform 300 iterations of a one time I need to give the 300 inputs (question that is impossible because of the future inputs do not know). Now the dilemma is this: I have done an analysis of details, I've closed the loop and I checked with more data if more or less fulfilled the prediction, but unfortunately I have not been successful. For this I used about 2100 but I think data are insufficient to achieve the desired accuracy. Besides the variables are correlated but do not show much correlation (0.35 or so). In openloop get accuracy R2a = 0.82 with MSE = 0.0016 and performing the analysis of R2 in training / validation / test none of them made me overfitting. You think that you should use more data? thank you very much.

Best Answer

1. Use original trn/val/tst data and 'divideblock' to create and evaluate the open loop net.
2. If val and tst performance is unsatisfactory, try to design better nets via changing delays, numHidden, random initial weights and/or data-split ratios. Oherwise ...
3. Close the loop and evaluate using the ORIGINAL data. Unbiased val and tst performances are much more important than the biased trn performance (even if the latter bias is mitigated by a degree-of-freedom-adjustment).
4. Predict closed loop performance using NEW test input data. If corresponding NEW target data is available, evaluate the performance.
 I think data are insufficient to achieve the desired accuracy. Besides the variables are correlated but do not show much correlation (0.35 or so).
Are you referring to zero lag correlations obtained from corrcoef or dynamic correlations obtained from nncorr, xcorr or crosscorr? Nonzero lag correlations are usually significant above 0.14 or so.
 In openloop get accuracy R2a = 0.82 with MSE = 0.0016 ...
Relatively irrelevant. What are R2val and R2tst?
 and performing the analysis of R2 in training / validation / test none of them made me overfitting.
Overfitting is usually detected by high R2trn but low R2val, R2tst and possibly R2trna.
You think that you should use more data?
If Nw << Ntrneq, you probably have enough training data.
If std(MSEtst) << MSEtst you probably have enough test data and similarly for val.
Look up the expression for std(MSE) assuming MSE is Chi-square distributed in Wikipedia (I only remember it for binomially distributed classifications).
Hope this helps.
Thank you for formally accepting my answer.
Greg
Related Question