This is the first time that I'm asking on this site so apologies in advance if it's not quite the usual standard.
I'm going thorough the paper Auto-Encoding Variational Bayes https://arxiv.org/abs/1312.6114 and in page 10 appendix B there is a simple enough looking equation that I don't fully understand how they got the results they have.
$$\int q_{\phi}(\mathbf{z})log\,p(\mathbf{z})d\mathbf{z} = … $$
when I expand $log\,p(\mathbf{z})$ I get
$$log\,p(\mathbf{z}) = log\, \mathcal{N}(\mathbf{z}; \mathbf{0}, \mathbf{I}) =
log\, det(2\pi I)^{-1/2}\,e^{-1/2 \,\mathbf{z}'I\mathbf{z}} = \frac{-J}{2}\, log \,2\pi \, -\frac{1}{2}\left\lVert \mathbf{z} \right\rVert ^{2}$$
where $J$ is the dimension of $\mathbf{z}$. From there it is easy to see that the term $\int q_{\phi}(\mathbf{z})\frac{-J}{2}\, log \,2\pi\, d\mathbf{z}$ evaluates to $\frac{-J}{2}\, log \,2\pi$ given that the integral over a density function evaluates to 1 which is in agreement with the paper. However I'm not sure how to proceed with the second part $\int q_{\phi}(\mathbf{z})\frac{-J}{2}\, \left\lVert \mathbf{z} \right\rVert ^{2} d\mathbf{z}$.
I'll really appreciate if anyone that knows could please explain what is it that I'm misunderstanding.
Thanks.
Best Answer
It's actually not terribly difficult, the key observation is that the integral actually splits into a bunch of single variable integrals,
\begin{align} \int q_\phi({\bf z})\|{\bf z}\|_2^2d{\bf z}&=\int q_\phi({\bf z})(z_1^2+\ldots+z_n^2)d{\bf z}\notag\\ &=\int q_\phi({\bf z})z_1^2d{\bf z} + \ldots + \int q_\phi({\bf z})z_n^2d{\bf z}\notag\\ &=\int q_\phi(z_1)z_1^2dz_1 + \ldots + \int q_\phi(z_n)z_n^2dz_n\notag. \end{align}
Now since $q_\phi(z_i) = N(\mu, \sigma^2)$, the above expression can be rewritten in expected value form as,
$$\sum_{i=1}^n\text{E}Z_i^2,$$
and using the identity that $\text{Var}X = \text{E}X^2 - (\text{E}X)^2$, we obtain,
$$\sum_{i=1}^n\text{E}Z_i^2=\sum_{i=1}^n(\text{E}Z_i)^2 + \text{Var}Z_i=\sum_{i=1}^n\mu_i^2 + \sigma_i^2.$$