If $\hat{\theta}_n$ is an estimator for the parameter $\theta$, then the two sufficient conditions to ensure consistency of $\hat{\theta}_n$ are:
Bias($\hat{\theta}_n)\to 0$ and Var$(\hat{\theta}_n)\to 0$,
then we will have $\lim_{n\to\infty}Pr(|\hat{\theta}_n-\theta|>\varepsilon)=0, \forall\varepsilon>0$.
Now suppose $X_1,\ldots X_n$ be iid samples drawn from the Bernoulli distribution with parameter $p$. Let $\hat{p}=\bar{X}$ be the estimator for $p$, where $\bar{X}$ is the sample mean. It is clear that $Bias(\hat p)=E(\hat{p})-p=0$ and $Var(\hat{p})=\frac{Var(X_1)}{n}\to 0$, as $n\to\infty$. We should expect that $\hat{p}$ is a consistent estimator.
However, from the definition of consistency, if I choose $\varepsilon=0.25, p=0.5$ and $X_i=1$ for all i. Then $\hat{p}= 1$ and $$Pr(|\hat{p}-p|>\varepsilon) = Pr(1-0.5>0.25) = 1.$$
This looks like a contradiction to the sufficient conditions above. Can anyone let me know about the mistakes in my argument? Thank you!
Best Answer
As pointed out in the comments, $\hat p$ in this context is not a constant---it is a random variable. Before you observe the data, you have no idea what $\hat p$ is. In fact, you treated it correctly as a random variable when you computed the bias and variance. Instead, by Chebyshev's inequality,
$$P(|\hat p-p|>\epsilon)=P\left(\left|\frac{\hat p-p}{\sqrt{p(1-p)/n}}\right|>\frac{\epsilon}{\sqrt{p(1-p)/n}}\right)\leq \frac{p(1-p)}{n\epsilon^2}\to 0$$
as $n\to \infty$. Alternatively, consistency follows directly from the weak law of large numbers.