The Shannon entropy of a discrete random variable ${\textstyle X}$ with possible values ${\textstyle \left\{x_{1},\ldots ,x_{n}\right\}}$ and probability mass function ${\textstyle \mathrm {P} (X)}$ is defined:
$${\displaystyle \mathrm {H} (X)=-\sum _{i=1}^{n}{\mathrm {P} (x_{i})\log _{2}\mathrm {P} (x_{i})}}.$$
The measure should be maximal if all the outcomes are equally likely (uncertainty is highest when all possible events are equiprobable); in this case $H\leq\log _{2}(n)$.
Let us consider two discrete random variables ${\displaystyle X}$ and ${\displaystyle Y}$. Mutual information can be equivalently expressed as
$$\operatorname {I} (X;Y)=\mathrm {H} (X)+\mathrm {H} (Y)-\mathrm {H} (X,Y),$$
where ${\displaystyle \mathrm {H} (X)}$ and ${\displaystyle \mathrm {H} (Y)}$ are the marginal entropies, and ${\displaystyle \mathrm {H} (X,Y)}$ is the joint entropy of ${\displaystyle X}$ and ${\displaystyle Y}$.
I wonder if there exists an upper bound for $\operatorname {I} $ like the $H\leq\log _{2}(n)$ and if it involves the equiprobability of events.
Best Answer
First :
should be
Regarding upper bounds for $I(X;Y)$, you should write
$$I(X;Y)=H(X) -H(X|Y) \implies I(X;Y) \le H(X) \le \log(|\mathcal X|)$$
where $|\mathcal X|$ is the cardinality of the alphabet for $X$ ($n$ above). Because you can also write $I(X;Y)=H(Y) -H(Y|X)$ you get
$$I(X;Y) \le \log( \min(|\mathcal X|,|\mathcal Y|))$$