MATLAB: Deep Deterministic Policy Gradient Agents (DDPG at Reinforcement Learning), actor output is oscilating a few times then got stuck on the minimum.

agentddpgdeep learningqlearningreinforcement learning

Hi
I am not experienced on Simulink and RL. I have tried to simulate a very simple scenario to test DDPG before implementing my complex system. The agent is randomly placed around (0,0) and the goal is to move to (500,500) or its nearby.
But it doesn't work for me. The action output (2×1) should be continuous in the range [-2 2]. For the first few episodes, the output oscillates between max and min and then stay on the minimum for the rest of the episodes.
I changed deep network settings as well as RL options but same problem. I have changed the output range and make it (-inf inf) with saturation but still the same. Also, I simulate it for a few thousand episodes but the same problem.
Codes are attached.

Best Answer

A few points I have identified with your original script
  • You should include the action bounds when defining action specification. With this information, the DDPG will not automatically adjust the gain for tanh output. It only saturates the output with this range. 
actionInfo = rlNumericSpec([numAct 1]), 'LowerLimit', -2, 'UpperLimit', 2);
You can refer to DDPG Pendulum example to see how a scalingLayer is included after tanhLayer to scale the action range from -2 to 2 (same with your case)
  • The noise model is unstable. To create a stable Ornstein-Uhlenbeck noise model, ensure that abs(1 - MeanAttractionConstant.*SampleTime) is less than or equal to 1.
Due to the above points (unstable noise model, no action bound), I observe Inf action from your original set up (which later propagate to NaN weights in the network). I change MeanAttractionConstant to 1/30 and it works fine.
Related Question