beforehand,i want to congrat coming new year guys,wish all you everything best in your life,now i have a little problem and please help me,i know definition of entropy which has a formula
$$\sum\limits_{i=1}^{n} -p(m_{i})\log_{2}(p(m_{i}))$$
now suppose we have following table for joint distribution
we know that marginal distribution of X is (1/2,1/4,1/8,1/8) and for y (1/4,1/4,1/4,1/4)
i know how is calculating marginal distribution,we can calculate that Entropy $H(x)=7/4$ bits and $H(y)=2$ bits;
but i have a question with following formula,there is given that
$$H(X\mid Y)= \sum_{i=1}^4 P(Y=i) H(X\mid Y=i)$$
and here is calculation process finally $H(X\mid Y)$ according to above formula
$$H(X\mid Y)=\frac{1}{4}\left( H(1/2,1/4,1/8,1/8)+H(1/4,1/2,1/8,1/8)+H(1/4,1/4,1/4,1/4)+H(1,0,0,0)\right)$$
so my question is how we have got last one?please help me
Best Answer
I think your question is where $$ H(X|Y)=(1/4)H(1/2,1/4,1/8,1/8)+(1/4)H(1/4,1/2,1/8,1/8) $$ $$+(1/4)H(1/4,1/4,1/4,1/4)+(1/4)H(1,0,0,0)$$ comes from given $$ H(X|Y)=\sum_i P(Y=i)H(X|Y=i). $$ First note that $P(Y=i)$ is $(1/4)$ for every $i$ (just sum across the row $Y=i$ in your picture above); this is where the $1/4$ in front of each $H(X|Y=i)$ comes from. Also note that $X|Y=i$ is a probability distribution, with $P(X=j|Y=i)=P(X=j,Y=i)/P(Y=i)$. So, for instance, $X|(Y=1)$ is given by $4(1/8,1/16,1/32,1/32)=(1/2,1/4,1/8,1/8)$ and $H(X|Y=1)$ is $H(1/2,1/4,1/8,1/8)$. Hence the first term in the sum for $H(X|Y)$ is what is written above. I hope this helps, as I'm not sure exactly what you were asking.