Can I determine the Kullback-Leibler distance
$$
D_{\mathrm{KL}}(P\parallel Q)=\sum_{i}\ln\left(\frac{P(i)}{Q(i)}\right) P(i)
$$
between the following probability distributions?
P(X) = 1 2 3 4 5 6 7 8 Q(X) = 1 2 3 4 5 6 7 8
2/8 1/8 1/8 0 1/8 2/8 0 1/8 0 1/8 1/8 2/8 1/8 0 2/8 1/8
These are 2 different probability distributions and sometimes in the sum $\ln(P(i)/Q(i)) = \infty$ and $P(i)=0$. Because somethimes the $P(i)$ or $Q(i)$ is $0$. How I can handle it?
Best Answer
As the others already mentioned for the given data there exist no exact calculation of the KL divergence. However if you want to have something in hand anyway, then you can use deliver an $\epsilon$ amount of probability from the non zero probabilites to the zero probabilites, say $1/100$ from each. Then you wil have $3/100$ for the zero probabilities. Now you can calculate the KL-divergence.
In the next step deliver even less amount of probability say $1/1000$ and in the same way calculate the KL- divergence. This procedure will eventually give some ideas. It is also okay to remove a very tiny amount of probability from non zero elements because in the divisions or multiplications it will have a little effect while it will make your problem well defined.