[Math] when it is conditional entropy minimized

conditional probabilityentropy

for example let us consider following table

enter image description here

know that entropy of variable is maximum when it is equally distributed,all of it's variable has equal probability,but what about joint entropy or conditional entropy?we know that channel capacity is equal

$I(X,Y)=H(X)-H(X|Y)=H(Y)-H(Y|X)$

it is equal maximum when $H(X)$ is maximum and $H(X|Y)$ is minimum,but when it happens this?for first we can say that all marginal probability of $X$ should be equal to each other,but what about second?is it same also?clearly if $H(X|Y)$ is zero in addition of $H(X)$ be maximum,then we will have maximum mutual information,but when it happens?thanks in advance

Best Answer

$$H(X|Y)=\sum_{i,j}p(x,y)\log(p(x|y))$$

So to make this zero we want $p(x|y)=1$ whenever $p(x,y)\neq 0$. This means that $x$ is a function of $y$. So to maximise $I(X,Y)$, take both variables to be equal and with uniform marginals.

Related Question