MATLAB: Create policy evaluation function for RL agent

Question

Hello,

I'm using the Reinforcement Learning Toolbox to train an agent for a particular task. Once the training is complete, I want to generate a policy evaluation function so I can use it in a simulink file. I can create the policy function using "generatePolicyFuncion" command. The matlab script that is generated is the following:

function action1 = evaluatePolicy(observation1)
%#codegen
 
% Reinforcement Learning Toolbox
% Generated on: 23-Sep-2019 09:27:39
 
action1 = localEvaluate(observation1);
end
%% Local Functions
function action1 = localEvaluate(observation1)
persistent policy
if isempty(policy)
                policy = coder.loadDeepLearningNetwork('agentData.mat','policy');
end
action1 = predict(policy,observation1);
end

When I run the simulink file, the following errors pop up:

Simulink does not have enough information to determine output sizes for this block. If you think the errors below are inaccurate, try specifying types for the block inputs and/or sizes for the block outputs.

Component:MATLAB Function | Category:Coder error

Dot indexing is not supported for variables of this type.

Function 'MATLAB Function' (#171.286.341), line 13, column 11:

"coder.loadDeepLearningNetwork('agentData.mat','policy')"

Launch diagnostic report.

Component:MATLAB Function | Category:Coder error

Persistent variable 'policy' must be assigned before it is used. The only exception is a check using 'isempty(policy)' that can be performed prior to assignment.

Function 'MATLAB Function' (#171.365.371), line 15, column 19:

"policy"

Launch diagnostic report.

Component:MATLAB Function | Category:Coder error

Function call failed.

Function 'MATLAB Function' (#171.140.167), line 7, column 11:

"localEvaluate(observation1)"

Launch diagnostic report.

Component:MATLAB Function | Category:Coder error

Errors occurred during parsing of MATLAB function 'test_RL/MATLAB Function'

Component:MATLAB Function | Category:Coder error

Simulink cannot determine sizes and/or types of the outputs for block 'test_RL/MATLAB Function' due to errors in the block body, or limitations of the underlying analysis. The errors might be inaccurate. Fix the indicated errors, or explicitly specify sizes and/or types for all block outputs.

Component:MATLAB Function | Category:Coder error

Dot indexing is not supported for variables of this type.

Function 'MATLAB Function' (#171.286.341), line 13, column 11:

"coder.loadDeepLearningNetwork('agentData.mat','policy')"

Launch diagnostic report.

Component:MATLAB Function | Category:Coder error

Persistent variable 'policy' must be assigned before it is used. The only exception is a check using 'isempty(policy)' that can be performed prior to assignment.

Function 'MATLAB Function' (#171.365.371), line 15, column 19:

"policy"

Launch diagnostic report.

Component:MATLAB Function | Category:Coder error

Function call failed.

Function 'MATLAB Function' (#171.140.167), line 7, column 11:

"localEvaluate(observation1)"

Launch diagnostic report.

Component:MATLAB Function | Category:Coder error

Errors occurred during parsing of MATLAB function 'test_RL/MATLAB Function'

How can I solve this issue?

Thanks

Answer 1

Can you try defining the size of inputs and outputs in the MATLAB Function block? This seems to be coming up a lot in the error message