deep learningderivativeloss-functionsneural networks
In Goodfellow's (2016) book on deep learning, he talked about equivalence of early stopping to L2 regularisation (https://www.deeplearningbook.org/contents/regularization.html page 247).
Quadratic approximation of cost function $j$ is given by:
$$\hat{J}(\theta)=J(w^*)+\frac{1}{2}(w-w^*)^TH(w-w^*)$$
where $H$ is the Hessian matrix (Eq. 7.33). Is this missing the middle term? Taylor expansion should be:
$$f(w+\epsilon)=f(w)+f'(w)\cdot\epsilon+\frac{1}{2}f''(w)\cdot\epsilon^2$$
Saturation means it's not updating . Which means it is on some kind of a local minimum rather than in the desired global minima.
So when increasing the learning rate it can jump our from that local minima and move further .
Refer to the image below about local minima and global minima problems .
Here's a good thread - Improving loss with , increasing learning rate
After reviewing the equations a few more times. I think the correct loss is the following:
$$\mathcal{L} = (11.1 - 4.3)^2$$
My reasoning is that the q-learning update rule for the general case is only updating the q-value for a specific $state,action$ pair.
$$Q(s,a) = r + \gamma \max_{a*}Q(s',a*)$$
This equation means that the update happens only for one specific $state,action$ pair and for the neural q-network that means the loss is calculated only for one specific output unit which corresponds to a specific $action$.
In the example provided $Q(s,a) = 4.3$ and the $target$ is $r + \gamma \max_{a*}Q(s',a*) = 11.1$.
Best Answer
They talk about the weights at optimum:
At that point, the first derivative is zero—the middle term is thus left out.