Solved – Posterior variance vs variance of the posterior mean

bayesiancoverage-probabilitycredible-intervalfrequentistpoint-estimation

This question is about the frequentist properties of Bayesian methods.

Suppose we have data ${\bf y}$ generated from a distribution with a single parameter $\theta$, equipped with a prior $\pi(\theta)$. This leads to a posterior distribution $\pi(\theta|{\bf y})$. A Bayesian might characterize the uncertainty surrounding $\theta$ using the posterior variance

\begin{align}
\mathbb{V}(\theta|{\bf y}) = \int_\Theta \left(\theta-\hat\theta\right)^2\pi(\theta|{\bf y}) \ d\theta &&&&&& (1)
\end{align}

where $\hat\theta$ is the posterior mean. A frequentist wearing his Bayesian hat (or perhaps the other way around) might choose to focus on the posterior mean (i.e. an estimate of $\theta$) more directly. The variance of the posterior mean is given by

\begin{align}
\mathbb{V}(\hat\theta | \theta) = \int_\Theta\left(\hat\theta – E(\hat\theta)\right)^2 f(\hat\theta|\theta) \ d\hat\theta &&&&&& (2)
\end{align}

If the posterior mean is unbiased for $\theta$, and the posterior distribution $\pi(\theta|{\bf y})$ "looks like" the sampling distribution $f(\hat\theta|\theta)$, then equations (1) and (2) start to look alike.

Why might we care: Thinking about the frequentist coverage of credible intervals (mixing Bayesian and frequentist worlds again), it seems that the ratio of these two quantities should provide valuable information. For example, if the posterior variance is much smaller than the variance of the posterior mean, coverage of the credible intervals may be poor (less than nominal).

  1. Does it even make sense to compare these quantities? What kind of information can be gained from comparing them?
  2. Can anybody point me to some references dealing with this specific issue? (I've tried searching this, but can't find anything on this site or otherwise dealing with this specific issue).

Best Answer

There is no particular reason that $\pi(\theta|\mathbf{y})$ should look anything like $f(\hat{\theta}|\theta)$ as functions of $\theta$. The latter is a sampling distribution for the parameter estimator $\hat{\theta}$, which may have a substantially different form to the likelihood function for the data $\mathbf{y}$. Moreover, the parameter estimator in Bayesian analysis is hardly ever unbiased, since it incorporates a prior distribution. For those reasons I think it is extremely unlikely that the integrands in these variance equations are going to "look like" each other, except perhaps in unusual special cases.

  1. Does it even make sense to compare these quantities? What kind of information can be gained from comparing them?

Since these are variances for entirely different quantities, conditional on different things, any useful comparison is probably going to come through some corresponding interval estimator for $\theta$. The posterior standard deviation $\mathbb{S}(\theta|\mathbf{y})$ ought to give you a rough idea of the width of a credible interval for $\theta$, whereas the standard error $\mathbb{S}(\hat{\theta}|\theta)$ ought to give you a rough idea of the width of a confidence interval for $\theta$.

Consequently, if you were willing to discard "shape information" for the distributions then you could reasonably say that the comparison of the two variances will give you an idea of the relative accuracy of the credible interval versus the confidence interval. This would be a bit tentative in my view, but it might be possible under some simplifying assumptions.

2 . Can anybody point me to some references dealing with this specific issue? (I've tried searching this, but can't find anything on this site or otherwise dealing with this specific issue).

I'm not familiar with any literature on this topic, but perhaps you could search for comparisons of accuracy/width of credible intervals and confidence intervals. If there is any literature on that subject then I imagine it will involve these two variance quantities.

Related Question