MLE + Cramér-Rao bound of a discrete random variable

fisher informationlog likelihoodmaximum likelihoodparameter estimationstatistics

Can someone explain how to compute the maximum likelihood estimator, the Fisher information and the Cramér-Rao bound of a discrete random variable please?

I came across this exercise while reviewing for my exam but can't solve it, I had only been dealing with continuous distributions so far.

Let $(X_1,…,X_n)$ be an i.i.d. random sample of size $n$ and distribution $\mathbb{P}_\theta$.

$$\mathbb{P}_\theta(\{0\})=1/4,\hspace{20pt}\mathbb{P}_\theta(\{1\})=3\theta/4,\hspace{20pt}\mathbb{P}_\theta(\{2\})=3(1-\theta)/4$$

$\theta\in[0,1]$

Here's what I've done:

$\mathcal{L}(x_1,…,x_n;\theta)=\left\{\begin{array}{cl}
\dfrac{1}{4}&\text{if } x=0\\
\dfrac{3\theta}{4}&\text{if } x=1\\
\dfrac{3(1-\theta)}{4}&\text{if } x=2\\
\end{array}\right.$

$\ln(\mathcal{L}(x_1,…,x_n;\theta))=\left\{\begin{array}{cl}
\ln(1/4)&\text{if } x=0\\
\ln(3/4)+\ln(\theta)&\text{if } x=1\\
\ln(3/4)+\ln(1-\theta)&\text{if } x=2\\
\end{array}\right.$

$\frac{\partial \ln(\mathcal{L}(x_1,…,x_n;\theta))}{\partial{\theta}}=\left\{\begin{array}{cl}
0&\text{if } x=0\\
1/\theta&\text{if } x=1\\
-1/(1-\theta)&\text{if } x=2\\
\end{array}\right.$

But then I can't figure out the MLE, the Fisher information and the Cramér-Rao bound… Thanks.

Best Answer

As I mentioned in a comment, the likelihood can be written in the following convenient way:

\begin{align} L(\theta\mid \boldsymbol x)&=\left(\prod_{i: x_i=0}\frac14\right)\left(\prod_{i:x_i=1}\frac{3\theta}4\right)\left(\prod_{i:x_i=2}\frac{3(1-\theta)}4\right)\quad,\,\theta \in [0,1] \\\\&=\left(\frac14\right)^{n_1}\left(\frac{3\theta}4\right)^{n_2}\left(\frac{3(1-\theta)}4\right)^{n_3}\,, \end{align}

where $n_1,n_2$ and $n_3$ are respectively the number of observations equal to $0,1$ and $2$ in the sample, such that $n_1+n_2+n_3=n$. These $n_i$'s are of course based on the sample observations, which can be explicitly seen by writing them as sums of indicator variables:

$$n_1(\boldsymbol x)=\sum_{i=1}^n \mathbf1(x_i=0)\,,\,n_2(\boldsymbol x)=\sum_{i=1}^n \mathbf1(x_i=1)\,,\,\text{etc.}$$

In fact, it is clear from the likelihood that $(n_1(\boldsymbol X),n_2(\boldsymbol X),n_3(\boldsymbol X))$ has a multinomial distribution.

This should help you in finding the Fisher information.