I'm trying to understand the derivation of the Kozachenko-Leonenko entropy estimator which uses a k-nearest neighbor approach to estimate a probability density function. The details of the derivation are on page 4 here. There is an integral that I'm not sure how it's being evaluated. It's equation 5 and it looks like:
$\mathbb{E}[\log p_i(\epsilon)]=\int\limits_{0}^{\infty}\log p_i(\epsilon)P_k(\epsilon)d\epsilon=\psi(k)-\psi(N)$
From the derivation, the integral simplifies to:
$\frac{(N-1)!}{1!(k-1)!(N-k-1)!}\times \int\limits_{0}^{1} p^{k-1}(1-p)^{N-k-1}\log (p) dp =\psi(k)-\psi(N)$.
Where $\psi$ is the digamma function
I'm not understanding how to evaluate the integral to the difference of digamma functions. The text provides no proof. Can anyone help here?
Thanks in advance for the help!
Best Answer
Take the Beta Function: $$B(m,n) = \int_{0}^{1}t^{m-1}(1-t)^{n-1}dt$$ Take its partial derivative with respect to $m$: $$\frac{\partial B(m,n)}{\partial m}=\int_{0}^{1}t^{m-1}(1-t)^{n-1}\log(t)dt$$ Also note that: $$B(m,n) =\frac{\Gamma(m)\Gamma(n)}{\Gamma(m+n)}$$ Now partial differentiate with respect to m: $$\frac{\partial B(m,n)}{\partial m}=\Gamma(n)\frac{\partial }{\partial m}\left(\frac{\Gamma(m)}{\Gamma(m+n)}\right)$$ Using the Quotient Rule: $$\Gamma(n)\left(\frac{\Gamma(m+n)(\frac{\partial\ }{\partial m}(\Gamma(m))-\Gamma(m)(\frac{\partial\ }{\partial m}(\Gamma(m+n))}{\Gamma(m+n)^2}\right)$$ Note the definition of $\psi(x)$, the Digamma Function: $$\psi(x)=\frac{d }{dx}\ln\Gamma(x)=\frac{\Gamma'(x)}{\Gamma(x)}$$ Therefore, $$\Gamma'(x)=\psi(x)\Gamma(x)$$ Back to the original: $$\Gamma(n)\left(\frac{\Gamma(m+n)\Gamma(m)\psi(m)-\Gamma(m)(\frac{\partial\ }{\partial m}(\Gamma(m+n))}{\Gamma(m+n)^2}\right)$$
$$\frac{\partial\ }{\partial m}\Gamma(m+n)=\psi(m+n)\Gamma(m+n)$$
Therefore, $$\Gamma(n)\left(\frac{\Gamma(m+n)\Gamma(m)\psi(m)-\Gamma(m)\psi(m+n)\Gamma(m+n)}{\Gamma(m+n)^2}\right)$$
Simplify the Expression: $$\frac{\Gamma(m)\Gamma(n)}{\Gamma(m+n)}(\psi(m)-\psi(m+n)) $$ The Integral is Hence: $$\int_{0}^{1}t^{m-1}(1-t)^{n-1}\log(t)dt=\frac{\Gamma(m)\Gamma(n)}{\Gamma(m+n)}(\psi(m)-\psi(m+n))$$ Directly using this the integral asked above is resolved.