Given you have an independent random sample $X_1, X_2,…, X_n$ of a Bernoulli random variable with parameter $p$, estimate the variance of the maximum likelihood estimator of $p$ using the Cramer-Rao lower bound for the variance
So, with large enough sample size, I know the population mean of the estimator $\hat{P}$ will be $p$, and the variance will be:
$$Var[\hat{P}] = \frac{1}{nE{[((\partial/\partial p)\;\ln\:{f_x(X)})^2]}}$$
Now I'm having some trouble calculating the variance of $\hat{P}$, this is what I have so far:
since the probability function of $\bar{X}$ is binomial, we have:
$$f_x(\bar{X}) = \binom{n}{\sum\limits_{i=1}^n{X_i}}*p^{\sum\limits_{i=1}^n{X_i}}*(1-p)^{n-\sum\limits_{i=1}^n{X_i}}$$
so:
$$\ln\:{f_X(X)} = \ln(\binom{n}{\sum\limits_{i=1}^n{X_i}})+ \sum\limits_{i=1}^n{X_i} ln({p})+\:(n-\sum\limits_{i=1}^n{X_i})\ln(1-p)$$
and:
$$\frac{\partial \ln\:{f_X(X)}}{\partial p} =
\frac{\sum\limits_{i=1}^n{X_i}}{p} – \frac{(n-\sum\limits_{i=1}^n{X_i})}{(1-p)} =
\frac{n\bar{X}}{p} – \frac{(n-n\bar{X})}{(1-p)}$$
and:
$$\left(\frac{\partial ln\:{f_X(X)}}{\partial p}\right)^2 =
\left(\frac{n\bar{X}}{p} – \frac{(n-n\bar{X})}{(1-p)}\right)^2 =
\frac{n^2 p^2 – 2 n^2 p \bar{X} + n^2 \bar{X}^2}{p^2(1-p)^2}$$
since $E[\bar{X}^2] = \mu^2 + \frac{\sigma^2}{n}$, and for a Bernoulli random variable $E[X] = \mu = p = E[\bar{X}]$ and $Var[X] = \sigma^2 = p(1-p)$:
$$E\left[\left(\frac{\partial \ln\:{f_X(X)}}{\partial p}\right)^2\right] =
\frac{n^2 p^2 – 2 n^2 p E[\bar{X}] + n^2 E[\bar{X}^2]}{p^2(1-p)^2} =
\frac{n^2 p^2 – 2 n^2 p^2 + n^2 \left(p^2 + \frac{p(1-p)}{n} \right)}{p^2(1-p)^2} = \frac{n p (1-p)}{p^2(1-p)^2} =
\frac{n}{p(1-p)}$$
therefore:
$$Var[\hat{P}] = \frac{1}{nE{[((\partial/\partial p)\;\ln\:{f_x(X)})^2]}} =
\frac{1}{n\frac{n}{p(1-p)}} =
\frac{p(1-p)}{n^2}$$
However, I believe the true value I should have come up with is $\frac{p(1-p)}{n}$
Any help would be greatly appreciated!
Best Answer
You seem to have confused the formulas for the lower bound in the case where the sample is IID, and the more general case of not necessarily identically distributed variables.
The correct formula should be (in the unbiased case) $$\operatorname{Var}[\hat p] \ge \operatorname{E}\left[\left(\frac{\partial}{\partial p} \left[\log f_{\boldsymbol X}(\boldsymbol x \mid p)\right]\right)^2 \right]^{-1},$$ when the joint density is over the sample $\boldsymbol x = (x_1, \ldots, x_n)$. Note there is no factor of $n$ in front. But when the sample is IID, this can be written $$\operatorname{Var}[\hat p] \ge n \operatorname{E}\left[\left(\frac{\partial}{\partial p}\left[ \log f_X(x \mid p) \right]\right)^2 \right]^{-1},$$ where the density (notice no boldface!) is the univariate density for a single observation from the sample.