Solved – Sampling from the joint distribution p(x,y) when y = f(x)

gibbsjoint distributionsampling

Suppose I want to sample from the joint distribution $p(X, Y)$, where $X$ is a random variable and $Y = f(X)$ where $f$ is a known function of $X$. However, sampling from $p(X,Y)$ directly is hard. Could I use Gibbs sampling and sample from the conditionals $x^{(k)} \sim p(X\mid Y=y^{(k-1)})$ with $y^{(k-1)}=f(x^{(k-1)})$ and $y^{(k)} \sim p(Y\mid X=x^{(k)}) = \delta(y-f(x^{(k)}))$ for $k = 1, 2, \dots$? Would this Gibbs sampler converge?

Best Answer

(This answer does not really help with the Gibbs sampler much, but points at something else you could do. I do think the Gibbs sampler would converge).

Recall that, $$p(X,Y) = p(Y | X) p(X)\, . $$

Now if you can sample from the marginal of $X$, then $X$ is the linchpin variable. You can use exact sampling methods if it is a known distribution, or you could use MCMC. This sampler should intuitively converge faster than the Gibbs sampler because the Markov chain is only present in $X$ here, whereas in the Gibbs sampler, the Markov chain samples both $(X, Y)$.

Related Question