Let $p,q \in [0,1]$. Then the following bound seems to hold:
$$ p \log \frac{p}{q} + (1-p) \log \frac{1-p}{1-q} \leq
\left( \log \frac{p}{q} – \log \frac{1-p}{1-q} \right)^2$$ The left-hand side is the KL divergence between two Bernoulli random variables, so the logarithm is to base $2$. I've tested the inequality in MATLAB on a million equispaced points $(p,q)$ in $[0,1]^2$ and it holds on all of them.
My only thought was that this looks vaguely like the inequality
$f(\alpha x + (1-\alpha) y) \leq \alpha f(x) + (1-\alpha) f(y) – (\mu/2) \alpha (1-\alpha) (x-y)^2$ for a $\mu$-strongly convex function, but the direction of this one seems to go the other way.
Best Answer
In terms of the natural logarithm $\ln$ your inequality becomes $$ \tag{*} \ln 2 \cdot \left( p \ln \frac{p}{q} + (1-p) \ln \frac{1-p}{1-q} \right) \leq \left( \ln \frac{p}{q} - \ln \frac{1-p}{1-q} \right)^2 \, . $$ Let us fix $q \in (0, 1)$ and consider the function $$ f(x) = \ln 2 \cdot \left( x \ln \frac{x}{q} + (1-x) \ln \frac{1-x}{1-q} \right) - \left( \ln \frac{x}{q} - \ln \frac{1-x}{1-q} \right)^2 $$ for $x \in (0, 1)$. Then $f(q)=0$, and the task is to show that this is the maximum of $f$ in the interval.
An elementary calculation gives $$ f'(x) = \left( \ln \frac{x}{q} - \ln \frac{1-x}{1-q} \right) \left( \ln 2 - \frac{2}{x(1-x)} \right) \, . $$ Since $x(1-x) \le \frac 14$, the second factor is $\le \ln 2 - 8 < 0$, so that the sign of $f'(x)$ is determined by the sign of the first factor only. It follows that $$ f'(x) \begin{cases} > 0 & \text{for } 0 < x < q \, , \\ < 0 & \text{for } q < x < 1 \, , \\ \end{cases} $$ so that indeed $f(x) \le f(q) = 0$.
Actually the proof shows that $(*)$ holds with $\ln 2$ on the left-hand side replaced by $8$.