I would like to calculate $E(relu'(X))$ with
$X \sim \mathcal{N}(0,\sigma^2)$
$f(x) = \frac{1}{\sigma \sqrt{2\pi}} \cdot e^{-\frac{1}{2} \cdot (\frac{x-\mu}{\sigma})^2}$
$relu(x) = max(0,x)$
$relu'(x) = \begin{cases}1 &\text{, } x>0\\0 &\text{, } x < 0\end{cases}$
\begin{align*}
E(relu'(x)) &= \int_{-\infty}^{\infty} relu'(x) \cdot f(x) dx\\
&= \int_{-\infty}^{\infty} relu'(x) \cdot \frac{1}{\sigma \sqrt{2\pi}} \cdot e^{-\frac{1}{2} \cdot (\frac{x-\mu}{\sigma})^2} dx\\
&\overset{\text{bc of }x > 0 \Rightarrow relu'(x)=1}{=} \lim_{r \to 0+} \int_{r}^{\infty} \frac{1}{\sigma \sqrt{2\pi}} \cdot e^{-\frac{1}{2} \cdot (\frac{x-\mu}{\sigma})^2} dx
\end{align*}
Unfortunately I don't how to continue with this equation
Best Answer
Because we have $$\int_{-\infty}^\infty relu’(x)f(x)dx=\int_{-\infty}^\infty I(x>0)N(0,\sigma^2)dx=\int_0^\infty N(0,\sigma^2)dx=\frac12$$
the answer is $\frac12$.