I am reading these notes on slide 34 and came across strategies to prevent gradients from becoming too big in Deep Q Learning (DQN). Since, we don't usually use deep architectures in DQN, I don't think it's an exploding gradient problem. My understanding is that it has something to do with the linear regression squared error loss function, since DQN is a regression network. Could someone please explain it to me?
Reinforcement Learning – Handling Large Gradients in DQN Models
gradient descentneural networksregressionreinforcement learning
Related Solutions
- The update rule
$$w_i^{(t+1)} = w_i^{(t)} - \frac{\eta}{\sqrt{\sum_{\tau=1}^t g_{\tau,i}^2}}g_{t,i},$$
is the composite mirror descent variety of ADAGRAD. Take a look at equation 4 (or 23) from the paper. When there's no regularization term, the update can be derived (take derivative and set to 0) as:
$$ w_{t+1} = w_t - \eta \,\text{diag}(G)^{-1/2} g_t.$$
Notice that $\text{diag}(G)^{-1/2}_{ii} = \frac{1}{\sqrt{\sum_{\tau=1}^t g_{\tau,i}^2}}$ and that multiplying a diagonal matrix by a vector amounts to element-wise multiplication.
$\eta$ is a constant step size which determines the size of the step at each update. I would try several choices and compare them.
Changing an algorithm from SGD to ADAGRAD just requires plugging in your gradient values. That is, you need to keep track of the sum of squared gradient elements for the term in the denominator.
In ordinary least squares regression, the variance of the data in the regressors will end up in the denominator for the expression of the error of the parameters
$$\text{Cov} (\hat\beta) = \hat{\sigma}^2 (X^TX)^{-1} $$
If the columns in the regressor matrix $X$ are perpendicular (and they are if you use principle components) then the standard error can be expressed as
$$s.e.(\beta_i) \approx \sqrt{\frac{\sigma^2} {n \text{Var}(X_i)} }$$
So larger variance in the regressor $X_i$ means smaller variance/error in the estimate for the coefficient/slope/gradient.
See also the following image. It gives the same correlated data but with a different variance for the $x$ variable. This changes the slope. If the scale of the $x$ variable is smaller then the slope becomes larger (and also the error of the slope).
Best Answer
It's hard to say for sure, since the slide is so sparse, but I think the implication is that squaring a number with absolute value larger than 1 can become very large very quickly, and likewise the gradients can grow very fast. The author's suggestion of Huber loss makes sense as a remedy to this problem, because you can choose where the loss becomes MAE loss, which does not grow as quickly as a square error loss.
Likewise, this suggests that you're correct that the problem isn't exploding gradients. Exploding gradients refers to multiple layers having gradients larger than 1, so the magnitude of the update compounds as the chain rule is applied.
Even though the gradient is not exploding, it's still true that gradients can cause instability in training -- large steps in nearly-orthogonal directions can prevent the model from improving, or simply slow improvement.