It's my understanding that the Kullback-Leibler divergence can never be negative. Denoting the divergence as:
$$D(P||Q) = E_{p}[\log(p(X) / q(X))] = \int p(x) \log(p(x) / q(x)) dx$$
We can then use Jensen's inequality as follows:
$$E_{p}[\log(p(X) / q(X))] = -E_{p}[\log(q(X) / p(X))] \ge \log(E_{p}[q(X) / p(X)]) = \log\left(\int q(x)dx\right) = \log(1) = 0$$
However, I seem to be getting a counterexample in the Kullback-Leibler divergence between a $\text{Laplace}(\mu, b_{1})$ (the $P$ distribution) and $\text{Laplace}(\mu, b_{2})$ (the $Q$ distribution). Knowing that if $X$ has a $\text{Laplace}(\mu, b)$ distribution, $E[|X – \mu|] = b$, my argument was as follows:
$$D(P||Q) = E_{p}[\log(p(X) / q(X))] = E_{p}\left[\left(\log\left(\frac{1}{2b_1}\right) – \frac{1}{b_1}|X – \mu|\right) – \left(\log\left(\frac{1}{2b_2}\right) – \frac{1}{b_2}|X – \mu|\right)\right] = \log\left(\frac{b_2}{b_1}\right) – 1 + \frac{b_1}{b_2}$$
However, this Desmos graph seems to imply that negative KL divergences are possible for some values of $(b_1, b_2)$. Suspecting I made a mistake somewhere, I looked for someone else's derivation. Stumbled across this, where they computed the KL divergence between a $\text{Laplace}(\mu_1, b_1)$ and $\text{Laplace}(\mu_2, b_2)$. My above formula is just a special case of theirs, with $\mu_1 = \mu_2$.
What am I missing here?
Best Answer
The formula you got is correct for $\log$ in base $e$ (natural log). Sadly, Desmos understands that $\log = \log_{10}$. Replace that by $\ln$ and you'll get the right graph.
Besides (using natural log): it's well known that
$$\log x \ge 1-\frac{1}{x} \implies \log x -1 + \frac{1}{x} \ge 0$$
for any $x >0$. Setting $x=b_2/b_1$ you get the desired property.