MATLAB: Reinforcement Learning Toolbox: Not enough Room in buffer

buffersizeddpgReinforcement Learning Toolboxrlsimulink

Problem was a missunderstanding of an example. So this error is caused by an user error that was explained in one of the comments to this issue if that comment does not apply to you, you most likely have a different issue.
Hello,
I am pretty new in the realm of RL and am using the RL Toolbox to controll a Simulink modell with the DDPG Agent.
I have 2 actions and 2 observations
My Problem is that everytime i try to train the agent I get the Error:
An error occurred while running the simulation and the simulation was terminated
Caused by:
MATLAB System block 'rlMockLoop/RL Agent/AgentWrapper' error occurred when invoking 'outputImpl' method of 'AgentWrapper'. The error was thrown from '
'/usr/local/MATLAB/R2019a/toolbox/rl/rl/+rl/+util/ExperienceLogger.m' at line 30
'/usr/local/MATLAB/R2019a/toolbox/rl/rl/+rl/+agent/AbstractPolicy.m' at line 95
'/usr/local/MATLAB/R2019a/toolbox/rl/rl/simulink/libs/AgentWrapper.m' at line 107'.
Not enough room in the buffer to store the new experiences. Make sure the bufferSize argument is big enough.
I tried to increase the agentOption ExperienceBufferLength (even to pretty high values).
Is that even the right Option I should be looking at or am I missing something?
Code snippets:
Ts ~ 0.05
actionInfo = rlNumericSpec([2 1],...
    'LowerLimit',[0 0]',...
    'UpperLimit',[100  100]');
actionInfo.Name = 'StromstaerkeProzent';
actionInfo.Description = 'Aout, Ain';
%% Specify Observations
observationInfo = rlNumericSpec([2 1]);
actionInfo.Name = 'pressure';
actionInfo.Description = 'DruckWasser, Druck';
agentOpts = rlDDPGAgentOptions(...
    'SampleTime',Ts,...
    'TargetSmoothFactor',1e-3,...
    'ExperienceBufferLength',512*((10/Ts)*1000),...
    'DiscountFactor',0.99,...
    'MiniBatchSize',512);
agent = rlDDPGAgent(actor,critic,agentOpts);
trainingOptions = rlTrainingOptions(...
    'MaxEpisodes',1000, ...
    'MaxStepsPerEpisode',10/Ts, ...
    'ScoreAveragingWindowLength',5,...
    'Verbose',false, ...
    'Plots','training-progress',...
    'StopTrainingCriteria','AverageReward',...
    'StopTrainingValue',-1100,...
    'SaveAgentCriteria','EpisodeReward',...
    'SaveAgentValue',-1100);
simOptions = rlSimulationOptions('MaxSteps',10/Ts);
experience = sim(env,agent,simOptions);
Other:
I tried to make the buffer size relative to the episode count and the length of 10s.
I really hope somebody can help me.

Best Answer

Sometimes connecting the output of the agent directly back to the reward will cause this situation, maybe you need a delay module
Related Question