Bayesian – Using MCMC When Likelihood Function Cannot Be Written

bayesianestimationlikelihoodmarkov-chain-montecarlo

Consider Markov Chain Monte Carlo (MCMC) to sample from the posterior distribution of some unknown parameter $\theta$:

$$P(\theta|X) \propto P(X|\theta)P(\theta)$$

Where $X$ is the observed data, $P(X|\theta)$ is the likelihood function and $P(\theta)$ is the prior distribution.

My question concerns the case where an expression for $P(X|\theta)$ can not be written however for a given value of $\theta$ it is possible to numerically draw samples from $P(X|\theta)$.

How can we draw samples from $P(\theta|X)$? Links to literature would be appreciated.

My idiotic and computationally inefficient approach would be: for the current value of $\theta$ in the markov chain draw an arbitrarily large number of samples from $P(X|\theta)$ and fit an empirical likelihood function. However my intuition is that since the number of draws from $P(X|\theta)$ is arbitrary only one draw needs to be taken but it is not clear to me how this would work.

Best Answer

The situation you describe where $p(x|\theta)$ cannot be computed but simulations from $p(\cdot|\theta)$ can be produced is call a generative model. It leads to likelihood-free resolutions like

  1. ABC (Approximate Bayesian computation), which is indeed properly introduced in the Wikipedia page: Approximate Bayesian computation;
  2. synthetic likelihood, as in Wood (2010), where the unknown model $p(\cdot|\theta)$ is approximated by a Normal $\text{N}(\mu(\theta),\sigma(\theta)^2)$, where $\mu(\theta),\sigma(\theta)$ are estimated by simulation;
  3. Bayesian solutions derived from indirect inference, as in Drovandi et al. (2015)
Related Question