I am following a lecture on logistic regression using gradient descent and I have an issuer understanding a short-path for a derivative :
Let be :
- $z=w_1x_1+w_2x_2+b$
- $a=\sigma(z)$
- and the loss function $L(a,y)=-y(\log(a)+(1-y)\log(1-a))$, which I know have a name but I can't remember it it.
In order to have $\frac{\delta\mathcal L(a,y)}{\delta z}$ I am able to compute $\frac{\delta\mathcal L(a,y)}{\delta a}$. But where do we get $\frac{\delta a}{\delta z}=a(1-a)$ ?
I can only have $\frac{\delta a}{\delta z}=\frac{\delta\sigma(w_1x_1+w_2x_2+b)}{\delta z}$
Best Answer
If $\sigma(x) = \frac{1}{1+\exp{(-x)}}$ then $\sigma^\prime(x) = \frac{-1}{[1+\exp(-x)]^2}\cdot(-\exp(-x)) = \frac{\exp(-x)}{[1+\exp(-x)]^2}$ (by the quotient rule)
And on the other hand, $\sigma(x)\cdot [1-\sigma(x)]$ is
$$\begin{align*} \frac{1}{1+\exp{(-x)}} \cdot \left[1-\frac{1}{1+\exp{(-x)}}\right] &= \frac{1}{1+\exp{(-x)}} \cdot \left[\frac{1+\exp{(-x)}}{1+\exp{(-x)}} - \frac{1}{1+\exp{(-x)}}\right]\\&=\frac{\exp(-x)}{\left[1+\exp(-x)\right]^2}\end{align*}$$
which is the same thing. So, $\sigma^\prime(x) = \sigma(x) \cdot [1-\sigma(x)]$.