Consider $p(x)$ and $q(x)$ are two pdfs of $X$, where $q(x)$ is a Gaussian distribution with the following form
$$q(x) = \frac{1}{\sqrt{2\pi\sigma_q^2}} \exp\left(-\frac{x^2}{2\sigma_q^2}\right).$$
Additionally, the variance of $x$ under pdf $p(x)$ is given (constrained), say $\sigma_p^2$, where $\sigma_p \neq \sigma_q$. Note that $p$ can be any pdf which is Lebesgue integrable.
Then which distribution $p(x)$ can minimize the KL-divergence $D(p\|q)$? The KL-divergence is defined as follows.
$$D(p\|q) = \int_{-\infty}^\infty p(x) \log\frac{p(x)}{q(x)} \, dx.$$
Thanks!
Note: If there is no variance constraint of $p(x)$, the answer is simply $p = q$ (a.e.). But this question is based on $\sigma_p \neq \sigma_q$. How to find the minimizer $p(x)$? Thanks a lot!
Best Answer
Hint. The optimal distribution is still Gaussian. In this case, the optimal Gaussian parameter is easily calculated. We show this by Lagrange multipliers: $$D^*(p\|q)=\int p(x)\log\frac{p(x)}{q(x)}dx-\lambda_1\left(\int x^2p(x) dx-\mu_p^2-\sigma_p^2\right)\\-\lambda_2\left(\int xp(x)dx-\mu_p\right)-\lambda_3\left(\int p(x)dx-1\right)$$ By calculus of variation we get $$ \log\frac{p(x)}{q(x)}+1-\lambda_1x^2-\lambda_2x-\lambda_3=0 $$ From which we may conclude $p(x)=q(x)e^{\lambda_1x^2+\lambda_2x+\lambda_3-1}$, hence a Gaussian.