I am trying to implement Neural Fitted Q iteration as described by Riedmiller (2005): http://ml.informatik.uni-freiburg.de/_media/publications/rieecml05.pdf This is a real-time application, where I update the weights of the neural network using the samples stored over a number of iterations (so, a small batch). I have tried to use the Neural Network Toolbox, but I have found out that I cannot update the weights of the neural network when I want to do it.
Trying to be a bit clearer, I use a function to initialize the neural network, which contains the following code:
net = feedforwardnet(5,'trainlm'); % say I use 5 hidden nodes to start with
net.trainParam.showWindow = false;net.numInputs = 2; % say I have 2 inputs
net.inputs{1}.range = [-1 1;-1 1]; % any range in fact
% Network weight and bias initalization:
net = init(net);Now, with the code above I have initialized the weight of the neural network, but if I try to run
y = net([x1;x2]);the network would return an empty matrix. This is a problem for the first iteration of the loop in the Fitted Q-learning program, since I have to call this function before I have any actual data with which to update the neural network. So, a solution could be:
net = configure(net,[0;0],0); % configuring neural network to 2 inputs, 1 output
net = init(net);or otherwise:
net = train(net,[7;0],1);But by doing this, I then observe that the neural network will return 0 from now on indefinitely, i.e. I CANNOT update the weights of the neural network anymore despite training it again multiple times in the main function whenever I have a new batch of data (at first 20 items, then 40, then 60 until I stabilize with 100 values).
So, is there a way I can use the neural network in a pseudo-real-time application? Should I store the weights in external variables, then use them as input before the next training step? Any other solutions?
Thank you in advance for your help!
Best Answer