$\int f \;d\mu=\int g \;d\mu=1$ and $f \log g \in L^1(\mu)$ implies $\int f \log g \; d\mu\leq \int f \log f \; d\mu$

convex-analysisjensen-inequalitylebesgue-integralprobability theoryreal-analysis

I'm trying to show the following inequality. Let $f,g:X\to (0,\infty)$ be measurable functions with $\int f\; d\mu=\int g \;d\mu=1$. Show that if $f \log g \in L^1(\mu)$, then

$$\int f \log g \;d\mu\leq \int f \log f \;d\mu$$

This seems like an application of Jensen's inequality but I'm having difficulty applying it correctly. I defined a measure $$d\nu=f\;d\mu$$ and so I need to show that

$$\int \log g \;d\nu\leq \int \log f \;d\nu$$

Note that $(X,\nu$) is a probability space. I need help recognizing the convex function $\phi$ and the $\nu-$integrable function $F$ onto which I can apply Jensen's inequality:

$$\phi(\int F\;d\nu)\leq \int \phi(F)\;d\nu$$

Any help with this is appreciated. Thanks for your time.

Best Answer

Special Case. The claim easily follows from Jensen's inequality if we also assume that $f\log f$ is in $L^1(\mu)$:

$$ \int \log\left(\frac{g}{f}\right) \, f \, \mathrm{d}\mu \overset{\text{(Jensen)}}{\leq} \log \left( \int \frac{g}{f} \, f \, \mathrm{d}\mu \right) = \log \left( \int g \, \mathrm{d}\mu \right) = 0. $$

General Case. Now we turn to the general case. If we are only given that $\int f \, \mathrm{d}\mu = 1$, then it is possible to have both $\int \left(f\log f\right)_+ \, \mathrm{d}\mu = \infty$ and $\int \left(f\log f\right)_- \, \mathrm{d}\mu = \infty$:

Example. Let $f \in \mathbb{R} \to (0, \infty)$ be defined by $$ f(x) = \begin{cases} \frac{2}{x (\log x)^2}, & \text{if } x \in (0, e^{-4}) \cup (e^4, \infty), \\ 0, & \text{elsewhere}. \end{cases} $$ Then $f$ is a valid PDF such that $f > 1$ on $(0, e^{-4})$ and $0 < f < 1$ on $(e^4, \infty)$. However, it is easy to check that $\int \left(f\log f\right)_+ \, \mathrm{d}\mu = \infty$ and $\int \left(f\log f\right)_- \, \mathrm{d}\mu = \infty$.

So, OP's assumption $f \log g \in L^1(\mu)$ will play a crucial role in ensuring that $\int f \log f \, \mathrm{d}\mu$ is well-defined. More generally, we will prove:

Theorem. Let $f, g : X \to (0, \infty)$ be measurable on the measure space $(X, \mathcal{F}, \mu)$. Suppose

  1. $\int f \, \mathrm{d}\mu = 1$ and $\int g \, \mathrm{d}\mu = 1$, and
  2. either $\int (f \log g)_- \, \mathrm{d}\mu < \infty$ or $\int (f \log f)_+ \, \mathrm{d}\mu < \infty$.

Then both $\int f \log g \, \mathrm{d}\mu$ and $\int f \log f \, \mathrm{d}\mu$ exist, and we have $$ \int f \log g \, \mathrm{d}\mu \leq \int f \log f \, \mathrm{d}\mu. $$

Proof. To begin with, we fix a compact subset $K$ of $(0, \infty)$ so that $\mu(\{f, g \in K\}) > 0$. Then, for any compact intervals $I = [a, b]$ and $J = [c, d]$ of $(0, \infty)$ containing $K$, we define the measure $\nu_{I, J}$ on $X$ by

$$ \nu_{I, J}(\mathrm{d}x) = \frac{f(x) \mathbf{1}_{\{ f \in I, g \in J \}}(x) \, \mu(\mathrm{d}x)}{\int f \mathbf{1}_{\{ f \in I, g \in J \}} \, \mathrm{d}\mu}. $$

Since $ \int f \mathbf{1}_{\{ f \in I, g \in J \}} \, \mathrm{d}\mu \geq \int f \mathbf{1}_{\{ f, g \in K \}} \, \mathrm{d}\mu > 0 $, this is indeed a well-defined probability measure. Moreover, the choice of truncation $\mathbf{1}_{\{ f \in I, g \in J \}}$ guarantees that $\log f$ and $\log g$ are in $L^1(\nu_{I, J})$. So by the Jensen's inequality,

\begin{align*} \int \log\left(\frac{g}{f}\right) \, \mathrm{d}\nu_{I, J} \leq \log\left( \int \frac{g}{f} \, \mathrm{d}\nu_{I, J} \right) = \log\left( \frac{\int g \mathbf{1}_{\{ f \in I, g \in J \}} \, \mathrm{d}\mu}{\int f \mathbf{1}_{\{ f \in I, g \in J \}} \, \mathrm{d}\mu} \right). \tag{1} \end{align*}

Rearranging $\text{(1)}$, we get

\begin{align*} \int \left( f \log g \right) \mathbf{1}_{\{ f \in I, g \in J \}} \, \mathrm{d}\mu &\leq \int \left( f \log f \right) \mathbf{1}_{\{ f \in I, g \in J \}} \, \mathrm{d}\mu + E_{I, J}, \tag{2} \end{align*}

where $E_{I,J}$ is defined by

$$ E_{I,J} = \left( \int f \mathbf{1}_{\{ f \in I, g \in J \}} \, \mathrm{d}\mu \right) \log\left( \frac{\int g \mathbf{1}_{\{ f \in I, g \in J \}} \, \mathrm{d}\mu}{\int f \mathbf{1}_{\{ f \in I, g \in J \}} \, \mathrm{d}\mu} \right). $$

By our choice of $K$, we can check that $E_{I, J}$ is bounded. Let $M$ be a bound of $E_{I, J}$. Now we example two cases:

  • Case 1. Suppose $\int \left(f \log g\right)_- \, \mathrm{d}\mu < \infty$. Then rearranging $\text{(2)}$, we get \begin{align*} \int \left( - f \log f \right) \mathbf{1}_{\{ f \in I, g \in J \}} \, \mathrm{d}\mu &\leq \int \left( - f \log g \right) \mathbf{1}_{\{ f \in I, g \in J \}} \, \mathrm{d}\mu + E_{I, J} \\ &\leq \int (f \log g)_- \, \mathrm{d}\mu + M, \end{align*} and so, \begin{align*} \int \left(f \log f \right)_- \mathbf{1}_{\{ f \in I, g \in J \}} \, \mathrm{d}\mu &\leq \int \left(f \log f \right)_+ \mathbf{1}_{\{ f \in I \}} \, \mathrm{d}\mu + \int (f \log g)_- \, \mathrm{d}\mu + M. \end{align*} Recall that $I = [a, b]$ and $J = [c, d]$. Assuming WLOG $b > 1$ and then letting $a, c \to 0^+$ and $d \to \infty$, MCT shows that \begin{align*} \int \left(f \log f \right)_- \, \mathrm{d}\mu &\leq \int \left(f \log f \right)_+ \mathbf{1}_{\{ f \leq b \}} \, \mathrm{d}\mu + \int (f \log g)_- \, \mathrm{d}\mu + M < \infty. \end{align*}

  • Case 2. Suppose $\int \left(f \log f\right)_+ \, \mathrm{d}\mu < \infty$. Then arguing similarly as in Case 1, we can show that $\int \left(f \log g\right)_+ \, \mathrm{d}\mu < \infty$.

Hence, the assumption implies that both $\int \left(f \log f\right) \, \mathrm{d}\mu $ and $\int \left(f \log g\right) \, \mathrm{d}\mu $ exist. Finally, by letting $a, c \to 0^+$ and $b, d \to \infty$ in $\text{(2)}$,

\begin{align*} \int f \log g \, \mathrm{d}\mu \leq \int f \log f \, \mathrm{d}\mu + \underbrace{ \left( \int f \, \mathrm{d}\mu \right) \log\left( \frac{\int g \, \mathrm{d}\mu}{\int f \, \mathrm{d}\mu} \right) }_{=0} \end{align*}

and therefore the desired claim follows.

Related Question