MATLAB: Reinforcement Learning Noise Model Mean Attraction Constant

Question

What does the mean attraction constant do? How can I tune it properly to promote exploration and learning? I can't seem to get the logic behind it.

With a sample time of 2, when I set it to 1 I get very noisy outputs. In the following graphs, rpm and valve%opening are the agents outputs and they are already scaled by a scaling layer.

When I set it to 0.05, then it seems like the noise model is not doing much explorations.

I also noticed that by applying the abs(1 – MeanAttractionConstant.*SampleTime) formula,

When sample time is 2 and the MAC is 1, the formula gives 1.

When sample time is 2 and the MAC is 0.05, the formula gives 0.9.

How does this relate to how fast the noise converge to the mean?

Thank you very much.

Answer 1

Assuming you are using DDPG, there is some information on the noise model here. I wouldn't worry too much about the mean attraction constant. The value of variance, variancedecayrate and variancemin play a much bigger role on 1) how much noise is added to the agent output and 2) for how long. If you want less noise to be added, reduce the variance value. If you want to explore for longer time, reduce the decay rate and set variancemin to a larger value.