Suppose $\hat{\theta}$ is an unbiased estimator for $\theta$. Then of course, $\mathbb{E}[\hat{\theta} \mid \theta] = \theta$.
How does one explain this to a layperson? In the past, what I have said is if you average a bunch of values of $\hat{\theta}$, as the sample size gets larger, you get a better approximation of $\theta$.
To me, this is problematic. I think what I'm actually describing here is this phenomenon of being asymptotically unbiased, rather than solely being unbiased, i.e.,
$$\lim_{n \to \infty}\mathbb{E}[\hat{\theta} \mid \theta] = \theta\text{,}$$
where $\hat{\theta}$ is likely dependent on $n$.
So, how does one explain what an unbiased estimator is to a layperson?
Best Answer
Technically what you're describing when you say that your estimator gets closer to the true value as the sample size grows is (as others have mentioned) consistency, or convergence of statistical estimators. This convergence can either be convergence in probability, which says that $\lim_{n \to \infty} P(|\hat{\theta}_n - \theta| > \epsilon) = 0$ for every $\epsilon > 0$, or almost sure convergence which says that $P(\lim_{n \to \infty} |\hat{\theta}_n - \theta| > \epsilon) = 0$. Notice how the limit is actually inside the probability in the second case. It turns out this latter form of convergence is stronger than the other, but both of them mean essentially the same thing, which is that the estimate tends to get closer and closer to the thing we're estimating as we gather more samples.
A subtle point here is that even when $\hat{\theta}_n \to \theta$ either in probability or almost surely, it is not true in general that $\lim_{n \to \infty} \text{E}(\hat{\theta}_n) = \theta$, so consistency does not imply asymptotic unbiasedness as you're suggesting. You have to be careful when moving between sequences of random variables (which are functions) to sequences of expectations (which are integrals).
All the technical stuff aside, unbiased only means that $\text{E}(\hat{\theta}_n) = \theta$. So when you explain it to someone just say that if the experiment were repeated under identical conditions many times that the average value of the estimate would be close to the true value.