Questions
My questions are:
- Is the following slice-sampling-within-Gibbs approach valid?
- Is there a good reference out there that uses, or better yet, justifies it?
Context
I'm trying to sample from a high-dimensional joint distribution, which I'll simplify to $p(x, y)$. Let's say $x$ and $y$ are scalars for now. I would like to use a Gibbs sampler to draw samples. I.e., I would like to iterate:
- Sample $x$ from its full conditional, $p(x | y)$.
- Sample $y$ from its full conditional, $p(y | x)$.
Unfortunately, I can't sample from $p(x | y)$ or $p(y | x)$ directly. In my case, I have also tried replacing these Gibbs steps with Metropolis-Hastings and rejection sampling, but results were poor. I have had much better results replacing each Gibbs step with a univariate slice sampler. (See Radford Neal's 2003 paper for an explanation of slice sampling.) Now, I do a slice-sampling-within-Gibbs algorithm.
Slice-sampling-within-Gibbs
- Use univariate slice sampling to draw $x$ from $p(x | y)$.
- Use a separate and independent univariate slice sampler to draw $y$ from $p(y | x)$.
In case there is any confusion with multivariate slice sampling, what I specifically mean is:
- Sample $u$ from Uniform(0, $p(x|y)$).
- Sample $x$ from Uniform$\{x : u < p(x | y)\}$.
- Sample $v$ from Uniform(0, $p(y|x)$).
- Sample $y$ from Uniform$\{y : v < p(y | x)\}$.
where the uniforms in steps 2 and 4 are not known exactly. (Also, I don't need the samples of $u$ or $v$, so I discard those.)
Best Answer
I found two references. This one details the algorithm, but the publicly-available pages that I could see on Google Books don't prove that it works.
Another one, also partially available on Google Books for free, seems to allude to slice-sampling-within-Gibbs.
I agree, it would be nice to find solid proof of validity, preferably in a good journal.
EDIT: Even Gelman's famous "Bayesian Data Analysis" (3rd ed) mentions the idea. In Section 12.3: Further extensions to Gibbs and Metropolis, under the "Slice sampling" heading, the end of the first paragraph says
Neal's famous 2003 slice sampling paper is where I think it was first suggested. The first paragraph of Section 4 says
Yet I still can find no proof of correctness.