Can anyone give an explanation in laymen's terms of what the relationship is, if any, between a sufficient statistic and an unbiased estimator?
Is it not true that both allow us to know something about an unknown parameter $\theta$? I think a sufficient statistic allows us to learn about a sample and the value of an unknown parameter about which we want to make some sort of inference.
And I think that an unbiased estimator gives us the most accurate estimation possible for some unknown population parameter, given information from a sample.
Best Answer
Sufficiency is an essential if rare property: if $S(X)$ is sufficient for model $f_\theta$, considering $S(X)$ for estimation of $\theta$ is sufficient, meaning you need nothing else from $X$. In other words, the collection of estimators based on $S(X)$ cannot be improved by other estimators [in the sense of any convex loss function].
Unfortunately, sufficiency only exists in exponential families. That is, if the density is of the form$$f_\theta(x)=\exp\{ \sigma(x)\cdot \Phi(\theta) - \psi(\theta)\}h(x)$$ then for a sample $(x_1,\ldots,x_n)$, $$S_n(x_1,\ldots,x_n)=\sum_{i=1}^n \sigma(x_i)$$is sufficient. Outside exponential families there is no sufficient statistic $S_n(X_1,\ldots,X_n)\in\mathbb{R}^d$ with fixed dimension $d$ [fixed in $n$] (this is the Pitman-Koopman-Darmois theorem).
On the other hand, unbiasedness is a local property that does not lead to optimality per se. Optimality does depend on the loss function used for comparing estimators and restricting the class of estimators to unbiased estimators can only increase the optimal error.
For instance, in the estimation of a multivariate normal mean vector, when $x\sim\mathcal{N}_d(\mu,I_d)$, under squared error loss, admissible estimators, i.e. estimators that cannot be beaten uniformly over $\mathbb{R}^d$, are all biased. In other words, an unbiased estimator of $\mu$ $\delta_0$ is inadmissible: there always exists a biased estimator $\delta_1$ (and in fact an infinite number of biased estimators) such that$$\mathbb{E}[||\mu-\delta_1(X)||^2]\stackrel{\ne}{\le}\mathbb{E}[||\mu-\delta_0(X)||^2]$$
For further information, I suggest reading Lehmann and Casella on this topic.