The differential entropy of the sum of independent random variables

information theoryprobability theory

Let $X$ be a random variable with density $f$. The differential entropy is defined by
$$h(X):=-\int_{\mathbb R^d} f(x)log(f(x))dx.$$

The conditional entropy is defined by replacing the density with conditional density.

Let $X$ and $Y$ be two independent random variables with densities. I want to show
$$h(X+Y)> h(X).$$

I feel like we probably need the formula for the chain rule of conditional entropy somewhere, like $h(X+Y)=h(X,X+Y)-h(X|X+Y)$

Best Answer

You can do it directly like this \begin{align*} h(X)&=h(X|Y)\\ &=h(X+Y|Y)\\ &\leq h(X+Y) \end{align*}

We can show the inequality is strict by observing that if $X$ and $Y$ are independent then $\text{Cov}(X,Y)=0$ which means that \begin{align*} \text{Cov}(X+Y,Y) &= \mathbb E[(X+Y-\mathbb E[X+Y]))(Y-\mathbb E[Y])]\\ &=\text{Cov}(X,Y)+\text{Var}(Y)\\ &=\text{Var}(Y)\\ &>0 \end{align*} So $X+Y$ and $Y$ are not independent hence $0<I(X+Y;Y)=h(X+Y)-h(X+Y|Y)$ which proves your statement.

$\text{Var}(Y)>0$ is a sufficient condition but may not be necessary.


Let's prove that $h(X|Y)=h(X+Y|Y)$. We can write \begin{align*} h(X|Y)&=-\int p_Y(y) \int p_{X|Y}(x|y) \log p_{X|Y}(x|y) dy\\ &=-\int p_Y(y) \int p_{X+Y|Y}(x+y|y) \log p_{X+Y|Y}(x+y|y) dy\\ &=h(X+Y|Y) \end{align*} By a change of variable $p_{X+Y|Y}(x+y|y)=p_{X|Y}(x|y)$.

Related Question