Solved – Combining posterior distributions

bayesianbugsdistributionsjagsr

I have 5 different posterior distributions (mcmc samples) which all estimate the same parameter beta. The 5 models are all obtained from 5 independent standardized datasets but estimate the same parameter. What I want to obtain is a single posterior distribution, combining information of my five original distribution. By instinct, following the rules of probability, I would say I can obtain a joint posterior distribution simply as the product of my 5 different posterior distributions. Is this correct, and if yes, how to do this in practice? Is there any BUGS/JAGS or R code anybody is willing to share?

Best Answer

Unfortunately, you cannot combine posterior chains in that way. From your description, what you have are independent draws from MCMC chains for the following posterior distributions:

$$p(\beta|\boldsymbol{x}_i, \pi_i),$$

where $\boldsymbol{x}_i$ is the data-set and $\pi_i$ is a prior in model $i = 1, ..., n$. You want a single posterior:

$$p(\beta|\boldsymbol{x}_1, ..., \boldsymbol{x}_n, \pi),$$

taken using the data from all $n$ models, where $\pi$ is the chosen prior for your analysis. Assuming that all models used the same prior that you want to use (i.e., assuming that $\pi = \pi_1 = ... = \pi_n$) gives some simplification to this problem, but it is still not enough. Unfortunately, even with this simplifying assumption, there is no function that transforms the original posteriors to the posterior you want, and similarly, no way to map the set of original MCMC chains to an appropriate MCMC chain for the posterior distribution you want.

Instead, you will need to go back to first-principles with the data sets from those models. You can either take each of the $n$ data-sets jointly, and build a single model to give an MCMC chain for the posterior from all this data, or you could do this iteratively, by updating one data-set at a time. When updating evidence in Bayesian statistics, the following recursive relationship holds:

$$p(\beta|\boldsymbol{x}_1, ..., \boldsymbol{x}_n, \pi) \propto p(\boldsymbol{x}_n|\beta) \cdot p(\beta|\boldsymbol{x}_1, ..., \boldsymbol{x}_{n-1}, \pi).$$

What this says is that the posterior given $\boldsymbol{x}_1, ..., \boldsymbol{x}_{n}$ can be obtained by using the posterior given $\boldsymbol{x}_1, ..., \boldsymbol{x}_{n-1}$ as your prior, and then updating this with the newly observed data $\boldsymbol{x}_{n}$. When using MCMC chains, this means that you would use the following looping procedure.

For $i = 1, ..., n-1$ do the following:

  1. Take the posterior MCMC chain from the distribution $p(\beta|\boldsymbol{x}_1, ..., \boldsymbol{x}_i, \pi)$.

  2. Use this as a chain of prior values for $\beta$. Input this into your MCMC calculations updating for the new data-set $\boldsymbol{x}_{i+1}$.

  3. This gives you a posterior MCMC chain from $p(\beta|\boldsymbol{x}_1, ..., \boldsymbol{x}_{i+1}, \pi)$. Generate the new posterior MCMC chain.

Running through this iterative process gives you a posterior MCMC chain for the distribution $p(\beta|\boldsymbol{x}_1, ..., \boldsymbol{x}_n, \pi)$, which is of interest.

Related Question