In several situations, I have two unbiased estimators, and I know one of them is better (lower variance) than the other. However, I would like to get as much information as possible, and I would like to do better than throwing out the weaker estimator.
$$\newcommand{\Outcome}{\text{Outcome}}\newcommand{\Skill}{\text{Skill}}\newcommand{\Luck}{\text{Luck}}\Outcome = \Skill + \Luck$$
$\Outcome$ is observed. $\Skill$ is what I would like to determine. $\Luck$ is known to have the average value $0$. From other observables, I can estimate $\Luck$ by $L$ so that $\mathbb E(L) = 0$ and $\mathrm{Var}(\Luck-L) \lt \mathrm{Var}(\Luck)$.
$\Outcome$ is an unbiased estimator for $\Skill$. A better estimate from variance reduction is $\Outcome – L$, which is also unbiased. For example, in one situation $\Outcome$ is the average of repeated trials, and I might produce a $95\%$ confidence interval of $[-5.0,13.0]$ without using variance reduction. Using variance reduction, I might get a confidence interval of $[-2.0,4.0]$.
The typical practice is for people to use $\Outcome-L$ instead of $\Outcome$. However, this is unsatisfactory to me because in my experience, there is more information in the pair $(\Outcome, \Outcome-L)$ than in just $\Outcome-L$. Specifically, in some situations I know that if $\Outcome$ is low, then $\Outcome-L$ tends to be an underestimate for $\Skill$, and if $\Outcome$ is high, then $\Outcome-L$ tends to be an overestimate for $\Skill$.
What's a good way to take advantage of the extra information from knowing both estimators?
Best Answer
If you have two unbiased estimators $L_1$ and $L_2$ then accuracy is measured by variance (for a mean square error loss function).
Take $L=aL_1+(1-a)L_2$, a weighted average of the two with $0 < a < 1$.
$$\newcommand{\Var}{\mathrm{Var}}\newcommand{\Cov}{\mathrm{Cov}} \Var(L) =a^2\Var(L_1) +(1-a)^2\Var(L_2) +2a(1-a) \Cov(L_1, L_2) \>. $$
When $\Cov(L_1,L_2) \leq 0$, $$ \Var(L) \leq a^2 \Var(L_1) +(1-a)^2 \Var(L_2) < a \Var(L_1) +(1-a)\Var(L_2) \leq \max(\Var(L_1), \Var(L_2)) \>. $$
If $\Cov(L_1, L_2)$ is sufficiently negative then $\Var(L)$ will also be less than $\min(\Var(L_1), \Var(L_2))$.
If $a$ is not restricted to $[0,1]$ and $1-a$ is replaced by $b$ it is possible to find constants $a$ and $b$ such that $\Var(L) = \Var(a L_1) + \Var(b L_2)$ is less than $\min (\Var(L_1), \Var(L_2))$. For $\Cov(L_1, L_2)>0$ a particular choice of $a>0$ and $b<0$ will work. This shows that there is always a way to improve the accuracy of unbiased estimators if you can determine a linear combination of the two that lowers the variance. One will always exist. Clearly this is a good way to take advantage of the knowledge of both estimators.