When you apply a Gaussian probability densityfunction to Rectified Linear/Sigmoid function,
you will get a rectified(*)/distorted Gaussian function.
(*)rectified Gaussian function is a mixture of delta and truncated gaussian as shown belown
https://en.wikipedia.org/wiki/Rectified_Gaussian_distribution
I am wondering, what is the Expectation and Variance of this rectified/distorted Gaussian?
The Expectation of distorted Gaussian from Sigmoid is shown below.
Expected value of applying the sigmoid function to a normal distribution
How about Variance? or Expectation and Variance of rectified Gaussian from Rectified Linear?
Thank you.
Best Answer
Let $X\sim N(\mu, \sigma)$. "Rectified" Gaussian is then $Y = \max(0, X)$. For both the expectation and the variance use the law of total expectation, as suggested by Michael. I'll also make use of the first two moments of the truncated normal distribution, which are readily available from Wikipedia.
\begin{align} \mathbb{E}Y &= \mathbb{E}[X|X>0]\mathbb{P}(X>0) + 0\times\mathbb{P}(X\leq0) \end{align}
Notice that the random variable $X|X>0$ is a truncated normal (with parameters $\mu, \sigma, 0, \infty$) which has a mean $\mu + \sigma \frac{\phi(-\mu/\sigma)}{1-\Phi(-\mu/\sigma)}$. Hence, as already established: $$ \mathbb{E}Y = \mu \left(1-\Phi\left(-\frac{\mu}{\sigma}\right)\right) + \sigma \phi\left(-\frac{\mu}{\sigma}\right) $$
For the variance \begin{align} Var(Y) &= \mathbb{E}[Y^2] - (\mathbb{E}Y)^2 \\ &= \mathbb{E}[X^2|X>0]\mathbb{P}(X>0) + 0\times\mathbb{P}(X\leq0)- (\mathbb{E}Y)^2 \end{align}
The only thing needed to calculate the above which we don't have is the second moment of the truncated normal, $\mathbb{E}[X^2|X>0]$, which can be obtained from its mean and the variance: \begin{align} \mathbb{E}[X^2|X>0] &= \sigma^2\left( 1+ \frac{-\frac{\mu}{\sigma} \phi(-\frac{\mu}{\sigma})}{1-\Phi\left(-\frac{\mu}{\sigma}\right)} - \frac{\phi\left(-\frac{\mu}{\sigma} \right)^2}{\left(1-\Phi\left(-\frac{\mu}{\sigma}\right)\right)^2}\right) + \left(\mu + \sigma \frac{\phi\left(-\frac{\mu}{\sigma}\right)}{1-\Phi\left(-\frac{\mu}{\sigma}\right)} \right)^2 \\ & = \sigma^2 +\mu^2 +\mu\sigma \frac{\phi\left(-\frac{\mu}{\sigma}\right)}{1-\Phi\left(-\frac{\mu}{\sigma}\right)} \end{align}
You can expand $(\mathbb{E}Y)^2$ and try to simplify a bit further: \begin{align} Var(Y) &= (\sigma^2 + \mu^2)\left(1-\Phi\left(-\frac{\mu}{\sigma}\right)\right) + \mu\sigma \phi\left(-\frac{\mu}{\sigma}\right) - \left(\mu \left(1-\Phi\left(-\frac{\mu}{\sigma}\right)\right) + \sigma \phi\left(-\frac{\mu}{\sigma}\right) \right)^2 \\ &= \mu^2\Phi\left(-\frac{\mu}{\sigma}\right)\left(1-\Phi\left(-\frac{\mu}{\sigma}\right)\right) + \mu\sigma\phi\left(-\frac{\mu}{\sigma}\right)\left(2\Phi\left(-\frac{\mu}{\sigma}\right)-1 \right) + \sigma^2\left(1-\Phi\left(-\frac{\mu}{\sigma}\right) - \phi\left(-\frac{\mu}{\sigma}\right)^2\right) \end{align}