MATLAB: How to test critic network in DDPG agent

ddpg agentReinforcement Learning Toolbox

I've trained a ddpg agent and I am trying to test my critic network with arbitrary observation and action. Is there any way to do this?

Best Answer

You can use getValue for inference on critic networks.

Related Solutions

MATLAB: How to extract a trained RL Agent’s network’s weights and biases

You can get the parameters from the trained's critic representation for DQN agent. In MATLAB R2020a, see getLearnableParameters and getCritic functions (function name changes a bit since R2019b). You can follow similar steps to get the actor's parameters from actor-based agent like DDPG or PPO.

critic = getCritic(agent);
criticParams = getLearnableParameters(critic);

MATLAB: Confusion in Critic network architecture design in DDPG

Hello,

Does this paper use DDPG as well? Any images that show the network architecture? If it's another algorithm, the critic may be implemented with a state value network V(s).

DDPG uses Q-network for the critic which needs to take in state and actions (s,a). Reinforcement Learning Toolbox lets you implement this architecture by providing separate input "channels" or paths for the state and the action. That allows you to use different layers in these two paths to extract features more efficiently. See for example image below:

If you want, you can concatenate the observation and action inputs and use a common feature extraction path as follows:

% create a network to be used as underlying critic approximator
statePath = featureInputLayer(numObservations, 'Normalization', 'none', 'Name', 'state');
actionPath = featureInputLayer(numActions, 'Normalization', 'none', 'Name', 'action');
commonPath = [concatenationLayer(1,2,'Name','concat')
             fullyConnectedLayer(400, 'Name', 'CriticStateFC1')
             reluLayer('Name', 'CriticRelu1')
             fullyConnectedLayer(300, 'Name', 'CriticStateFC2')
             reluLayer('Name','CriticRelu2')
             fullyConnectedLayer(1,'Name','StateValue')];
criticNetwork = layerGraph(statePath);
criticNetwork = addLayers(criticNetwork, actionPath);
criticNetwork = addLayers(criticNetwork, commonPath);
criticNetwork = connectLayers(criticNetwork,'state','concat/in1');
criticNetwork = connectLayers(criticNetwork,'action','concat/in2');
plot(criticNetwork)

Hope that helps

Best Answer

Related Solutions

MATLAB: How to extract a trained RL Agent’s network’s weights and biases

MATLAB: Confusion in Critic network architecture design in DDPG

Related Question