MATLAB: RL Agent does not learn

deep learningpgagentreinforcement learning

Hello,
I'm up to Reinforcement Learning with the RL Toolbox. After I built a custom environment in Simulink I have problems training the PG Agent. The RL problem is to control a system with diffuse irradiance, direct irradiance, and temperature as states and a mass flow rate as actions which can be 0 or 30. While Simulink generates a cost function it is also the reward signal and the objective is to reduce the cost over one episode of 24 time steps.
The code is the following:
obsInfo = rlNumericSpec([3 1]);
obsInfo.Name = 'Observation';
actInfo = rlFiniteSetSpec([0 30]);
actInfo.Elements = [0 30];
actInfo.Name = 'Action';
env = rlSimulinkEnv(mdl,[mdl '/RL Agent'],obsInfo,actInfo);
% Create deep neural network approximator for the actor
net = [ imageInputLayer([3 1 1], 'Normalization', 'none', 'Name', 'state')
fullyConnectedLayer(32, 'Name', 'fc1')
reluLayer('Name','relu1')
fullyConnectedLayer(32,'Name','fc2')
reluLayer('Name','relu2')
fullyConnectedLayer(32,'Name','fc3')
reluLayer('Name','relu3')
fullyConnectedLayer(2, 'Name', 'fc4')
softmaxLayer('Name', 'actionProb') ];
% Create actor
actorOpts = rlRepresentationOptions('LearnRate',0.01,'GradientThreshold',1);
actor = rlStochasticActorRepresentation(net, obsInfo, actInfo, 'Observation', 'state',actorOpts);
% Create Agent
opt = rlPGAgentOptions('DiscountFactor',0.0001);
agent = rlPGAgent(actor);
When I train the agent for 2000 Episodes with different configurations of the neural net I have the problem that it does not converge to a path at all. At some point the policy finds configurations which result in better reward but afterwards the agent does not converge further and does not follow the improved policy.
It would be great if you could help me solve the problem. Do you think this happens due to a insufficient reward signal or does the structure of my neural net not fit my observation and action signals? I also tried using tanhLayer or different amounts of nodes without any success.
Thank you very much for your help!
Best regards
Janika

Best Answer

Hello,
It is really hard to say just by looking at the training plot. The first thing I would try is 1) a different agent (maybe DQN since you have a discrete action space). If the agent is still not learning, assuming your network structure is roughly similar to what you see in other shipping examples, it probably has something to do with the reward or lack of exploration. It sounds like this is a controls problem, so I would highly recommend to look at some of the shipping examples to get a feeling of how different agents behave.
Related Question