I am currently studying a proof for the NMF multiplicative update rule based on generalized Kullback-Leibler divergence in the paper Algorithms for Non-negative Matrix
Factorization, and for some reason I just can't seem to figure out how to derive equation 29 (in proof for Lemma 3).
I was thinking about using the classic version of Jensen's inequality, stating that $\varphi\Big(\frac{\sum_{a} \alpha_a x_a}{\sum_{a} \alpha_a} \Big) \leq \frac{\sum_{a} \alpha_a \varphi(x_a)}{\sum_{a} \alpha_a}$ for positive $\alpha_a$'s and $\varphi$ convex. Since the $\alpha_a$'s sum to unity, I guess the inequality can be reduced to $\varphi(\sum_{a} \alpha_a x_a) \leq \sum_{a} \alpha_a \varphi(x_a)$, and hence by letting $x_a = \frac{W_{ia} h_a}{\alpha_a}$, we obtain the inequality
\begin{align*}
\log \sum_a \alpha_a \frac{W_{ia} h_a}{\alpha_a} = \log \sum_a W_{ia} h_a \leq \sum_a \alpha_a \log \frac{W_{ia} h_a}{\alpha_a},
\end{align*}
but then multiplying with -1 the inequlity flips, which is not the case in the paper.
Any suggestions to where I'm wrong and how to fix it?
Best Answer
You should apply the Jensen inequality (as stated) to the function $-\log$ (which is convex) instead of $\log$ (which is concave).