Bayesian Inference – Numerically Sampling from the Posterior Predictive in Bayesian Inference

bayesianmarkov-chain-montecarloposteriorpredictive-models

In the Bayesian approach the posterior distribution $P(\theta|x)$ of some hyper-parameter $\theta$ is given by:

$$P(\theta|x) = \frac{P(x|\theta)P(\theta)}{P(x)}$$

Where $x$ are samples of a random variable $X$. In many cases the posterior $P(\theta|x)$ can not be analytically computed and is instead calculated numerically using Markov Chain Monte Carlo. A popular approach is the Metropolis-Hastings algorithm which draws samples from:

$$P(\theta|x) \propto P(x|\theta)P(\theta)$$

The posterior predictive which describes the distribution of a new i.i.d. data point $x_\text{new}$ is given by:

$$P(x_\text{new}) = \int_\theta P(x_\text{new}|\theta)P(\theta|x)\, d\theta$$

My question is what are the popular methods to numerically sample from the posterior predictive $P(x_\text{new})$ if the posterior $P(\theta|x)$ is computed using Markov chain Monte Carlo?

Best Answer

If you can simulate values from $P(x_{\text{new}}|\theta)$, you can simply use your $N$ samples from posterior predictive and generate $x_{\text{new},i}$ for each posterior sample from this model to get a sample from the posterior predictive $\{x_{\text{new}}\}_{i=1}^N$.

This amounts to obtaining a collection $\{x_{\text{new},i}, \theta_i\}$ and discarding the value $\theta_i$, thus marginalizing over the vector of model parameters.