UMVUE – How to Find UMVUE of Bernoulli Random Variables

rao-blackwellself-studyumvue

Given i.i.d. Bernoulli$\left(\theta\right)$ r.v.s $X_1, X_2, …,X_n$, I'm asked to solve for the UMVUE of $\left(1−\theta\right)^2$ for the case $n=4$.

I think that $\sum X_i$ is my sufficient statistic (and I think completely sufficient) from using the Factorization Theorem, but I'm having trouble proceeding from there.

I know that I have to use the Rao-Blackwell theorem, but I'm not exactly sure how that follows. I would appreciate any tips!

Best Answer

The idea is that you can start with any estimator of $(1-\theta)^2$, no matter how awful, provided it is unbiased. The Rao-Blackwell process will almost magically turn it into a uniformly minimum-variance unbiased estimator (UMVUE).

There are many ways to proceed. One fruitful idea is systematically to remove the complications in the expression "$(1-\theta)^2$". This leads to a sequence of questions:

  1. How to find an unbiased estimator of $(1-\theta)^2$?

  2. How to find an unbiased estimator of $1-\theta$?

  3. How to find an unbiased estimator of $\theta$?

The answer to (3), at least, should be obvious: any of the $X_i$ will be an unbiased estimator because

$$\mathbb{E}_\theta(X_i) = \theta.$$

(It doesn't matter how you come up with this estimator: by guessing and checking (which often works), Maximum Likelihood, or whatever. ML, incidentally, often is not helpful because it tends to produce biased estimators. What is helpful is the extreme simplicity of working with a single observation rather than a whole bunch of them.)

Linearity of expectation tells us an answer to (2) would be any of the $1 - X_i$, because

$$\mathbb{E}_\theta(1-X_i) = 1 - \mathbb{E}_\theta(X_i) = 1-\theta.$$

Getting from this to an answer to (1) is the crux of the matter. At some point you will need to exploit the fact you have more than one independent realization of this Bernoulli variable, because it quickly becomes obvious that a single $0-1$ observation just can't tell you much. For instance, the square of $1-X_1$ won't work, because (since $(1-X_1)^2 = (1-X_1)$)

$$\mathbb{E}_\theta((1-X_1)^2) = 1-\theta.$$

What could be done with two of the observations, such as $X_1$ and $X_2$? A little thought might eventually suggest considering their product. Sure enough, because $X_1$ and $X_2$ are independent, their expectations multiply:

$$\mathbb{E}_\theta((1-X_1)(1-X_2)) = \mathbb{E}_\theta((1-X_1))\mathbb{E}_\theta((1-X_2)) = (1-\theta)(1-\theta)=(1-\theta)^2.$$

You're now good to go: apply the Rao-Blackwell process to the unbiased estimator $T=(1-X_1)(1-X_2)$ to obtain an UMVUE. (That is, find its expectation conditional on $\sum X_i$.) I'll stop here so that you can have the fun of discovering the answer for yourself: it's marvelous to see what kinds of formulas can emerge from this process.


To illustrate the calculation let's take the simpler case of three, rather than four, $X_i$. The sum $S=X_1+X_2+X_3$ counts how many of the $X_i$ equal $1$. Look at the four possibilities:

  1. When $S=0$, all the $X_i=0$ and $T = 1$ constantly, whence $\mathbb{E}(T\,|\,S=0)=1$.

  2. When $S=1$, there are three possible configurations of the $X_i$: $(1,0,0)$, $(0,1,0)$, and $(0,0,1)$. All are equally likely, giving each a chance of $1/3$. The value of $T$ is $0$ for the first two and $1$ for the last. Therefore

    $$\mathbb{E}(T\,|\,S=1) = \left(\frac{1}{3}\right)\left(0\right)+\left(\frac{1}{3}\right)\left(0\right)+\left(\frac{1}{3}\right)\left(1\right) = \frac{1}{3}.$$

  3. When $S=2$ or $S=3$, $T=0$ no matter what order the $X_i$ appear in, giving $0$ for the conditional expectation.

The Rao-Blackwellized version of $T$, then, is the estimator that associates with the sum $S$ the following guesses for $\theta$:

$$\tilde T(0)=1,\ \tilde T(1)=1/3,\ \tilde T(2)=\tilde T(3)=0.$$

As a check, the expectation of $\tilde T$ can be computed as

$$\eqalign{ \mathbb{E}(\tilde T) &= \Pr(S=0)\tilde{T}(0) + \Pr(S=1)\tilde{T}(1) + \Pr(S=2)\tilde{T}(2) + \Pr(S=3)\tilde{T}(3) \\ &= (1-\theta)^3 + \binom{3}{1}\theta(1-\theta)^2\left(1/3\right) + 0 + 0 \\ &= 1 - 3 \theta + 3 \theta^2 - \theta^3 + 3(1/3)(\theta - 2\theta^2 + \theta^2) \\ &= 1 - 2\theta + \theta^2 \\ &=(1-\theta)^2, }$$

showing it is unbiased. A similar calculation will obtain its variance (which is useful to know, since it is supposed to be the smallest possible variance among unbiased estimators).

Note that these calculations required little more than applying the definition of expectation and computing binomial probabilities.