Solved – How do ABC and MCMC differ in their applications

bayesiancomputational-statisticsmarkov-chain-montecarlo

To my understanding Approximate Bayesian Computation (ABC) and Markov Chain Monte Carlo (MCMC) have very similar aims. Below I describe my understanding of these methods and how I perceive the differences in their application to real life data.

Approximate Bayesian Computation

ABC consists at sampling a parameter $\theta$ from a prior, through numerical simulation compute a statistic $x_i$ which is compared to some observed $x_{obs}$. Based on a rejection algorithm, $x_i$ is either retained or rejected. The list of retained $x_i$s made the posterior distribution.

Markov Chain Monte Carlo

MCMC consists at sampling a prior distribution of the parameter $\theta$. It takes a first sample $\theta_1$, compute $P(x_{obs} | \theta_1)P(\theta_1)$ and then jump (according to some rule) to a new value $\theta_2$ for which $P(x_{obs} | \theta_2)P(\theta_2)$ is computed again. The ratio $\frac{P(x_{obs} | \theta_2)P(\theta_2)}{P(x_{obs} | \theta_1)P(\theta_1)}$ is calculated and depending on some threshold value, the next jump will occur from the first or the second position. The exploration of $\theta$ values goes one and one and by the end, the distribution of retained $\theta$ values is the posterior distribution $P(\theta | x)$ (for a reason that is still unknown to me).

I realize that my explanations miss to represent the variety of methods that exists under each of these terms (especially for MCMC).

ABC vs MCMC (pros and cons)

ABC has the advantage that one does not need to be able to analytically solve $P(x | \theta)P(\theta)$. As such ABC is convenient for complex model where MCMC would not make it.

MCMC allows to make statistical tests (likelihood ratio test, G-test, …) while I don't think this is feasible with ABC.

Am I right so far?

Question

  • How do ABC and MCMC differ in their applications? How does one decide to make use of one or another method?

Best Answer

Some additional comments on top of Björn's answer:

  1. ABC was first introduced by Rubin (1984) as an explanation of the nature of Bayesian inference, rather than for computational purposes. In this paper he explained how the sampling distribution and the prior distribution interact to produce the posterior distribution.

  2. ABC is however primarily exploited for computational reasons. Population geneticists came up with the method on tree-based models where the likelihood of the observed sample was intractable. The MCMC (Data Augmentation) schemes that were available in such settings were awfully inefficient and so was importance sampling, even with a parameter of a single dimension... At its core, ABC is a substitute to Monte Carlo methods like MCMC or PMC when those are not available for all practical purposes. When they are available, ABC appears as a proxy that may be used to calibrate them if it runs faster.

  3. In a more modern perspective, I personally consider ABC as an approximate inference method rather than a computational technique. By building an approximate model, one can draw inference on the parameter of interest without necessarily relying on a precise model. While some degree of validation is necessary in this setting, it is not less valid than doing model averaging or non-parametrics. In fact, ABC can be seen as a special type of non-parametric Bayesian statistics.

  4. It can also be shown that (noisy) ABC is a perfectly well-defined Bayesian approach if one replaces the original model and data with a noisy one. As such it allows for all Bayesian inferences one can think of. Including testing. Our input to the debate about ABC and hypothesis testing is that the approximate model underlying ABC may end up as poorly equipped to assess the relevance of an hypothesis given the data, but not necessarily, which is just as well since most applications of ABC in population genetics are concerned with model choice.

  5. In an even more recent perspective, we can see ABC as a Bayesian version of indirect inference where the parameters of a statistical model are related with the moments of a pre-determined statistic. If this statistic is enough (or sufficient in the vernacular sense) to identify these parameters, ABC can be shown to converge to the true value of the parameters with the number of observations.

Related Question