[Math] How to ensure the non-negativity of Kullback-Leibler Divergence KLD Metric (Relative Entropy)

entropyprobability distributionsst.statistics

I’m having some problems in ensuring the non-negativity of KLD!

I know that KLD is always positive and I went over the proof. However, it doesn’t seem to work for me. In some cases I’m getting negative results. Here is how I’m using KLD:

$${\rm KLD}( P(x) || Q(x) ) = \sum P(x) \log \left( \frac{P(x)}{Q(x)} \right) \, ,$$
where the Log is in base 2, and $P(x)$ and $Q(x)$ are two different distributions for all $x \in X$.

For example, $P(x) = {\rm Frequency}(x)/{\text Total Size}$; just a normal PMF! The same thing for $Q(x)$.
Note that the Total_Size of $P$ might be different from that of the $Q$ distribution.

Could you please let me know if I’m missing something? Are there any special conditions that I have to take into consideration to avoid having negative results?

Best Answer

"The K-L divergence is only defined if P and Q both sum to 1 and if Q(i) > 0 for any i such that P(i) > 0."

I suspect that the second condition is your problem. Say that you have x which appears in P but not Q -- in this case you're probably adding zero contribution to the sum in your code so that you don't have to divide by zero or take the logarithm of zero, but this is effectively throwing out mass from P and you get a negative number for the divergence.

http://en.wikipedia.org/wiki/Kullback%E2%80%93Leibler_divergence