I need to find two non-trivial examples for when the KL-Divergence happens to be symmetric for two distributions $P$ and $Q$, i.e.:
$$-\sum_{x\in\mathcal{X}} P(x) \log\left(\frac{Q(x)}{P(x)}\right)
=
-\sum_{x\in\mathcal{X}} Q(x) \log\left(\frac{P(x)}{Q(x)}\right).
$$
I have already found the following example: Let $P$ and $Q$ be two Bernoulli distributed RVs. Then it must hold:
$$\sum_{x\in\mathcal{X}}-p_x \log\left(\frac{q_x}{p_x}\right)
-(1-p_x) \log\left(\frac{1-q_x}{1-p_x}\right)
=
\sum_{x\in\mathcal{X}}-q_x \log\left(\frac{p_x}{q_x}\right)
-(1-q_x) \log\left(\frac{1-p_x}{1-q_x}\right)
$$
This is true when $p_x = 1-q_x$ for all $x\in\mathcal{X}$.
However, I am having a hard time to come up with a second example. Could you give me some hints?
Best Answer
Consider any distribution (here, I am identifying a probability distribution with its probability mass function) $P$ supported on $[k]=\{1,2,3\dots,k\}$, and let $\pi\colon[k]\to[k]$ be an involution (i.e., a permutation such that $\pi=\pi^{-1}$). Define $Q=P\circ\pi$; that is, for every $x\in [k]$, $Q(x) = P(\pi(x))$.
Then, since $\pi$ is a permutation, it is easy to check that $$ \sum_{x\in[k]} P(x) \log P(x) = \sum_{x\in[k]} Q(x) \log Q(x) \tag{1} $$ so that it suffices to show that $$ \sum_{x\in[k]} P(x) \log Q(x) = \sum_{x\in[k]} Q(x) \log P(x) \tag{2} $$ But this is the case, as $$\begin{align} \sum_{x\in[k]} Q(x) \log P(x) &= \sum_{x\in[k]} P(\pi(x)) \log P(x) = \sum_{y\in[k]} P(y) \log P(\pi^{-1}(y))\\ &= \sum_{y\in[k]} P(y) \log P(\pi(y)) = \sum_{y\in[k]} P(y) \log Q(y) \end{align}$$ where in the third equality we used $\pi^{-1}=\pi$.
Note that your example is a special case of the above, with $k=2$ and $\pi$ being the permutation which swaps the two elements of the domain.