Solved – conditional mutual information and how to deal with zero probabilities

conditional probabilityinformation theorymutual informationprobability

the conditional mutual between three sets of mutually exclusive variables, X, Y, and Z, is defined as follows.

$I(X,Y|Z) = \sum_{xyz} P(x,y,z) \log \frac{P(z)P(x,y,z)}{P(x,z)P(y,z)}$

my questions concern the $\log$ of the ratio of the probability products.

  1. if $P(z)$ or $P(x,y,z)$ is 0, then $\log(0)$ is undefined.
  2. if $P(x,z)$ or $P(y,z)$ is 0, then $\log(\infty)$ is undefined.

how do i deal with these 2 situations? the approach can be very flexible. for example, i thought about ignoring the inner sums where such conditions occur, but is this correct or reasonable?

any help is appreciated.

Best Answer

Ignoring the terms where this happens is the correct thing to do. You can justify this by noting that in each case you've outlined, no matter what happens inside the $\log$ you will have $P(x,y,z) = 0$. You can see this by applying the Frechet inequalities, namely that $P(A,B) \le \min\{P(A), P(B)\}$.

Related Question