I'm trying to obtain the Jeffreys' prior for a negative binomial distribution. I can't see where I go wrong, so if someone could help point that out that would be appreciated.
Okay, so the situation is this: I am to compare the prior distributions obtained using a binomial and a negative binomial, where (in both cases) there are $n$ trials and $m$ successes. I get the right answer for the binomial case, but not for the negative binomial.
Let's call the Jeffreys' prior $\pi_J(\theta)$. Then,
$$
\pi_J(\theta)\propto [I(\theta)]^{1/2}.
$$
Under the regularity conditions (fulfilled as we are dealing with the exponential family),
$$
I(\theta)=-E\left(\frac{\partial^2 \log L(\theta|x)}{\partial \theta^2}\right)
$$
where for the negative binomial $n$ is $x$ in the above expression (the total number of successes $m$ is fixed, $n$ is not). The distribution –I think– is
$$
p(m|\theta)\propto\theta^m(1-\theta)^{n-m}
$$
since $\theta$ is defined as the probability of success and $m$ is the number of successes. This is also the likelihood, since $m$ is a scalar and not a vector. Hence,
$$
L(\theta|n)\propto\theta^m(1-\theta)^{n-m}\\
\log L(\theta|n)=m\log\theta +(n-m)\log (1-\theta)\\
\frac{\partial\log L(\theta|n)}{\partial \theta}=\frac{m}{\theta}-\frac{n-m}{1-\theta}\\
\frac{\partial^2\log L(\theta|n)}{\partial \theta^2}=-\frac{m}{\theta^2}-\frac{n-m}{(1-\theta)^2}
$$
so the Fisher information is
$$
I(\theta)=-E\left(\frac{\partial^2\log L(\theta|n)}{\partial \theta^2}\right)=\frac{m}{\theta^2}+\frac{E(n)-m}{(1-\theta)^2}=\frac{m}{\theta^2}+\frac{\frac{m\theta}{1-\theta}-m}{(1-\theta)^2}\\
=\frac{m(1-\theta)^2+\frac{m\theta^3}{(1-\theta)}-m\theta^2}{\theta^2(1-\theta)^2}=\frac{m(1-2\theta)+\frac{m\theta^3}{(1-\theta)}}{\theta^2(1-\theta)^2}\\
=\frac{m(1-2\theta)(1-\theta)+m\theta^3}{\theta^2(1-\theta)^3}=\frac{m(1-3\theta+2\theta^2+\theta^3)}{\theta^2(1-\theta)^3}\\
\propto\frac{1-3\theta+2\theta^2+\theta^3}{\theta^2(1-\theta)^3}
$$
This, however, does not give me the correct answer. The correct answer is
$$
\pi_J(\theta)\propto \frac{1}{\theta(1-\theta)^{1/2}}
$$
which means that the Information I get should be
$$
I(\theta)=\frac{1}{\theta^2(1-\theta)}
$$
since the prior should be proportional to the square root of the information.
Can anyone find any mistakes? I wouldn't be surprised if I screwed something up with the set up of the distribution (successes vs failures with their respective probabilities, etc).
I used the expected value from Wikipedia and I know the correct answer from here (page 3).
Best Answer
The problem arises because the negative binomial distribution can be formulated differently. As a consequence, the expectation differs for different formulations. The way you have specified the negative binomial distribution, the expectation of $n$ is $E(n) = m/\theta$ (e.g. see here on page 3). With that, the Fisher information simplifies to $$I(\theta) = m\left(\frac{1}{\theta^2(1-\theta)}\right)$$
Thus the Jeffreys' prior is $$ \pi_{J}(\theta) = |I(\theta)|^{1/2}\propto \theta^{-1}(1-\theta)^{-1/2} $$
as you already noted.