Solved – ABC model selection

Question

It has been shown that ABC model choice using Bayes factors is not to be recommended due to the presence of an error coming from the use of summary statistics. The conclusion in this paper relies on the study of the behaviour of a popular method for approximating the Bayes factor (Algorithm 2).

It is well known that Bayes factors is not the only way for conducting model choice. There are other features, such as predictive performance of a model, that might be of interest (e.g. scoring rules).

My question is: is there a method, analogous to Algorithm 2, for approximating some scoring rule(s) or other quantities that can be used for conducting model choice in terms of the predictive performance in contexts with complicated likelihoods?

Answer 1

Nice question building on our work! Are you aware of the follow-up paper where we derive conditions on the summary statistic to achieve consistency in the Bayes factor? This may sound too theoretical but the consequence of the asymptotic results is quite straightforward:

Given a summary statistic $T$,

1. run an ABC algorithm based on $T$ for each model under evaluation ($i=1,..,I$) and estimate the parameters $\theta_i$ of those models by the ABC estimate $\hat\theta_i(T)$;
2. simulate the distribution of the statistic $T$ for each model and each estimated parameter, by a Monte Carlo experiment;
3. check whether the means $\mathbb{E}_{\hat\theta_i(T)}[T(X)]$ are all different by using step 2 with a sufficiently large number of iterations and, e.g., a t-test.

This procedure is not in the first version of the paper but should soon appear in the revised version