Can the mutual information be larger than the joint entropy

Question

For a Gaussian random variable of unitary variance, the entropy is given by
$H = 0.5\log(2 \pi e)$. Also, the mutual information (MI) between correlated Gaussian random variables is given by $MI = -0.5\log(1 – \rho^2)$, where $\rho$ denotes correlation. With these equations in mind, I can think of values for $\rho$ with which the MI is larger than the joint entropy between the two random variables. Am I missing something here or the MI may indeed be larger than the joint entropy?

Answer 1

For a Gaussian random variable of unitary variance, the entropy is given by $H=0.5 \log(2 \pi e)$.

Not quite. That's the differential entropy, which is a very different thing from the Shannon entropy. You cannot compare in a meaningful mutual information with differential entropies.

In particular, calling $h()$ the differential entropies, it's true that

$$I(X;Y) = h(X) - h(X|Y)$$

but it's not true that $h(X|Y) \ge 0$. Hence, the mutual information can be larger that the (differential) entropy of any variable (or the joint two variables).

See also here and here and here...