Solved – Conditional Mutual Information, Chain Rule

information theory

Consider three discrete binary random variables — A, B, C.

I have calculated I(A;B) and I(A;C), but I want to calculate I(A; BC)

I'm having trouble implementing this.

From reading this paper I have an idea, but can't get it to work out in python.

I have these functions (python 3)

def entropy(X, Y):
probs = []
for c1 in set(X):
for c2 in set(Y):
probs.append(np.mean(np.logical_and(X == c1, Y == c2)))
return np.sum(-p * np.log2(p) for p in probs if p > 0)

and

def calc_MI(x, y, bins):
c_xy = np.histogram2d(x, y, bins)[0]
mi = mutual_info_score(None, None, contingency=c_xy)
return mi

But I'm not sure how to move forward, thanks in advance.

Best Answer

If you want to compute I(A;BC) you do not necessarily have to use the conditional mutual information. I(A;BC) is the mutual information between A and the joint variable BC.

In python you might encode BC with 4 values: 0,1,2,3 for all the combinations of values for B and C. You might do this using: 2*b + c where b can be either 0 or 1 and c the same. Then you can compute the mutual information between A and BC.

Otherwise as you point out, you can use the formula from the paper: I(A;BC) = I(A;B) + I(A;C|B). In this case you have to use conditional probabilities.

Related Question