Derivative of Renyi entropy

calculusderivativesentropy

Let $\log$ denote the logarithm with base $2$. It is claimed that for $0\leq p_i\leq 1$, $\sum_i p_i = 1$ and for any $0\leq \alpha \leq \infty, \alpha\neq 1$

$$\frac{d}{d\alpha}\left(\frac{1}{1-\alpha}\log\sum_i p_i^\alpha\right) = \frac{1}{(1-\alpha)^2}\sum_i\frac{p_i^\alpha}{\sum_jp_j^\alpha}\log\frac{p_i^{\alpha-1}}{\sum_k p_k^\alpha}$$

I am not able to show this result. Taking the derivative with respect to $\alpha$, using the product rule and noting that $\frac{d}{dx}a^x = \ln(a) a^x$, I get

$$\frac{d}{d\alpha}\left(\frac{1}{1-\alpha}\log\sum_i p_i^\alpha\right) = \frac{1}{(1-\alpha)^2}\log\sum_i p_i^\alpha + \frac{1}{1-\alpha}\frac{1}{\sum_j p_j^\alpha}\sum_ip_i^\alpha \ln(p_i)$$

How does one proceed to get the desired result?

Best Answer

Your derivative is not correct, in order to avoid mistake with $\log_2$ when differentiating I always transform it to $\ln$ : \begin{align*} &\frac{d}{d\alpha}\left(\frac{1}{1-\alpha}\log\sum_i p_i^\alpha\right) \\ =& \log(e)\frac{d}{d\alpha}\left(\frac{1}{1-\alpha}\ln\sum_i p_i^\alpha\right)\\ =&\log(e)\left(\frac{1}{(1-\alpha)^2}\ln\sum_i p_i^\alpha + \frac{1}{1-\alpha}\frac{1}{\sum_j p_j^\alpha}\sum_ip_i^\alpha \ln(p_i)\right)\\ =&\frac{1}{(1-\alpha)^2}\log\sum_i p_i^\alpha + \frac{1}{1-\alpha}\frac{1}{\sum_j p_j^\alpha}\sum_ip_i^\alpha \log(p_i)\\ =& \frac{1}{(1-\alpha)^2}\left[ \frac{\sum_{i} p_i^\alpha}{\sum_{i} p_i^\alpha} \log\sum_{k} p_k^\alpha + \frac{1}{\sum_j p_j^\alpha}\sum_ip_i^\alpha (1-\alpha)\log(p_i) \right]\\ =&\frac{1}{(1-\alpha)^2\sum_{i} p_i^\alpha} \sum_{i} p_i^\alpha \left(\log\sum_{k} p_k^\alpha + (1-\alpha)\log(p_i)\right) \\ =&\frac{1}{(1-\alpha)^2\sum_{i} p_i^\alpha} \sum_{i} p_i^\alpha \log\left(p_i^{1-\alpha}\cdot \sum_{k} p_k^\alpha\right)\\ =&\frac{1}{(1-\alpha)^2\sum_{i} p_i^\alpha} \sum_{i} p_i^\alpha \left(\log\sum_{k} p_k^\alpha + (1-\alpha)\log(p_i)\right) \\ =&\frac{1}{(1-\alpha)^2} \sum_{i} \frac{p_i^\alpha}{\sum_{j} p_j^\alpha} \log\left(\frac{\sum_{k} p_k^\alpha}{p_i^{\alpha-1}}\right) \end{align*}

I think you are missing a minus sign from the wikipedia page, that will flip the $\log$ and give you what you want.