MATLAB: Training NARX open loop but testing closed loop

closed loopelectricity loadnarxopen looptestingtime series

Hello everyone,
I have an electricity load time series including trend and two components of seasonality. I want to train my network with an open loop structure but then want to test it with closed loop because I need to forecast next 1,5 years after training the network.
Therefore I updated the narx toolbox script, but I cannot be sure if it is correct. You will see that i defined a new network called netc and recorded its outputs in outputs, so examine the forecast performance with outputsc.
I would be glad if you can take a look to see it is correct.
Thanks in advance..
% Solve an Autoregression Problem with External Input with a NARX Neural Network
% Script generated by NTSTOOL
% Created Sun Feb 03 20:02:41 CET 2013
%

% This script assumes these variables are defined:
%
%   data - input time series.
%   VALUE - feedback time series.
inputSeries = tonndata(data,false,false);
targetSeries = tonndata(VALUE,false,false);
% Create a Nonlinear Autoregressive Network with External Input
inputDelays = 1:1;
feedbackDelays = 1:24;
hiddenLayerSize = [10];
net = narxnet(inputDelays,feedbackDelays,hiddenLayerSize);
% Prepare the Data for Training and Simulation
% The function PREPARETS prepares timeseries data for a particular network,
% shifting time by the minimum amount to fill input states and layer states.
% Using PREPARETS allows you to keep your original time series data unchanged, while
% easily customizing it for networks with differing numbers of delays, with
% open loop or closed loop feedback modes.
[inputs,inputStates,layerStates,targets] = preparets(net,inputSeries,{},targetSeries);
% Setup Division of Data for Training, Validation, Testing
net.divideFcn = 'divideind';
%net.divideMode = 'sample';
net.divideParam.trainInd = 1:35104;
net.divideParam.valInd = 35105:39104;
%net.divideParam.testInd = 39105:43104;
%[inputs,valInd,testInd] = divideind(2000,1:1000,1001:1500,1501:2000);
% Train the Network
[net,tr] = train(net,inputs,targets,inputStates,layerStates);
% Test the Network
netc = closeloop(net);
y1=targetSeries(39105:43104);
u1=inputSeries(39105:43104);
[p1,Pi1,Ai1,t1] = preparets(netc,u1,{},y1);
outputsc = netc(p1,Pi1,Ai1);
errors = gsubtract(t1,outputsc);
performance = perform(netc,t1,outputsc)
% View the Network
view(netc)

Best Answer

% So below you can see how my final code looks like for at least the open loop part still this part has two problems.
% 2.1. It says that valRatio is not a legal parameter, not like an error but appears on command window.
          Apparently, it is a bug; To get around this
          a. COULD TRY Nval = 1 and net.trainParam.max_fail = 1000.
          OR
          b. Treat training with 'dividetrain' and testing as separate problems
% 2.2. despite i put net.performFcn='mae' it doesnt set it and when i enter netperformFcn= it still says MSE.
          Could be another bug. Or maybe NARX only allows MSE . For the time
          being, just use the default  MSE.
% 2.3. and one additional question, do i need to normalize the data before feeding it? In my previous studies with NNs I always did but when I searched for NARX I didn't see anything mentioning it so thought maybe preparets does it for me, does it?
        No.
        TRAIN uses the common default MAPMINMAX. Instead of overriding it 
        with MAPSTD, I, typically,  recommend just using ZSCORE in the pre-        training preparations (plotting, detection of outliers and      
        significant lag values; constant and linear models) before considering 
        a neural model. Target standardization automatically normalizes MSE so 
        that R2a = 1-MSEa.
%Solve an Autoregression Problem with External Input with a NARX Neural Network Script generated by NTSTOOL Created Sun Feb 03 20:02:41 CET 2013 This script assumes these variables are defined: data - input time series. VALUE - feedback time series.
inputSeries = tonndata(data,false,false);
targetSeries = tonndata(VALUE,false,false);
      whos 
      VERY IMPORTANT to verify dimensions (I,O,N) and class(cell or double)
% Create a Nonlinear Autoregressive Network with External Input
inputDelays = 0;
      WHY? IF YOU ARE GOING TO HAVE A BUFFER FOR DELAYS 
      ALWAYS CHECK TO SEE IF THERE ARE SIGNIFICANT I/O CROSS  
      CORRELATIONS THAT CAN BE USED.
feedbackDelays = [1:2 23:25 168:169];
      WHY HAVE A SIZE 169 BUFFER FOR ONLY 7 DELAYS?
      Hub = ?
hiddenLayerSize = [10];
       Ndof = Ntrneq - Nw   % DOF CHECK:
net = narxnet(inputDelays,feedbackDelays,hiddenLayerSize)
net.performFcn = 'mae';
net.trainParam.valRatio = 0;
      USE MSE DEFAULT
      valRatio = 0 NOT ALLOWED (BUG)
[inputs,inputStates,layerStates,targets] = preparets(net,inputSeries,{},targetSeries);
    whos inputSeries targetSeries inputs inputStates layerStates targets
    ALWAYS CHECK SIZE AND CLASS AFTER PREPARETS
%Setup Division of Data for Training, Validation, Testing
---SNIP
net.divideParam.valInd = 20950:24950; % Nval = 4001
      NOT CONSISTENT WITH VAL RATIO = 0
net.divideParam.testInd = 24951:29927; % Ntst = 5027
     ARE YOU SURE ALL OF THIS DATA IS STATIONARY (I.E., CONSTANT 
     SLIDING WINDOW SUMMARY STATISTICS)?
-----SNIP
testY = cell2mat(targetSeries(testInd));
     USE 'Y' FOR OUTPUTS, NOT TARGETS
% Train the Network
-----SNIP
%Collect Statistics
     WHY IGNORE THE INFO IN TR??
-----SNIP ) errpct_train = abs(err_train)./trainY*100;
% WHAT IF TRAINY = 0 SOMEWHERE?
      I IGNORED THE REST
GREG
Related Question