I was looking at the Gibbs Sampler when I stumbled upon the following example:
Suppose $y = (y_{1}, y_{2}, \ldots, y_{n})$ are iid observations from an $N(\mu, \tau^{-1})$
Furthermore, suppose there exists prior information on $\mu$ and $\tau$ indicating that they are independent and that $\mu$ follows an $N(\mu_{0}, \sigma_{0}^{2})$ distribution while $\tau$ follows a $\text{Ga}(\alpha_{0}, \beta_{0})$ distribution.
For this example,
$$ p(\mu, \tau|y_{1}, y_{2}, \ldots, y_{n}) \propto f(y|\mu, \tau)p(\mu, \tau)$$
Also,
$$ p(\mu|\tau, y) \sim N(\frac{n\bar{y}\tau+\mu_{0}\tau_{0}}{n\tau+\tau_{0}}, (n\tau+\tau_{0})^{-1}) $$
where $\tau_{0} = (\sigma_{0}^{2})^{-1}$
and
$$ p(\tau|\mu, y) \sim \text{Ga}(\alpha_{0}+\frac{n}{2}, \beta_{0}+\frac{S_{n}}{2}) $$
where $S_{n} = \sum_{i=1}^{n}(y_{i}-\mu)^{2}$
Because it is not easy to compute directly from this distribution, the Gibbs sampler can be used provided the conditional distributions are known.
Could anybody demonstrate how (provide the derivations for) the conditional distributions ($p(\mu|\tau, y)$ and $p(\tau|\mu, y)$) given above?
EDIT:
For example, to achieve $p(\mu|\tau, y)$, is the following valid?
$$ p(\mu|\tau, y) \propto p(\tau, y|\mu)p(\mu)$$
If so, what form will $p(\tau, y|\mu)$ have?
Best Answer
Ok, what you need to do is compute the joint posterior up to a constant, i.e. $f(y_1,...,y_n|\mu,\tau)p(\mu,\tau)$. Then to compute the conditional posterior $\pi(\mu|\mathbf{y},\tau)$ you just treat the $\tau$ terms as fixed and known, so that some of them can be cancelled out. Then you do the same thing with $\mu$ to get $\pi(\tau|\mathbf{y},\mu)$.
So what I mean is, the joint posterior is given by: \begin{equation} \pi(\mu,\tau|\mathbf{y}) \propto \tau^{\frac{n}{2}+\alpha_0-1} \exp\{-\frac{\tau}{2} \sum_{i=1}^{n}(y_i-\mu)^2-\frac{\tau_0}{2}(\mu-\mu_0)^2-\beta_0\tau\}. \end{equation}
Now, to get the conditional posterior $\pi(\tau|\mathbf{y},\mu)$ you just remove the $\mu$ terms that just multiply the expression. So we could re-write the joint posterior as: \begin{equation} \pi(\mu,\tau|\mathbf{y}) \propto \tau^{\frac{n}{2}+\alpha_0-1} \exp\{-\frac{\tau_0}{2}(\mu-\mu_0)^2\}\exp\{-\frac{\tau}{2} \sum_{i=1}^{n}(y_i-\mu)^2-\beta_0\tau\}. \end{equation}
Now, since we are treating $\mu$ as fixed and known, for the conditonal posterior $\pi(\tau|\mu,\mathbf{y})$ we can remove the first exponential term from the equation and it will still be true (owing to the $\propto$ sign rather than the equals sign). It's important that you get this: the conditional posterior says given that we know $\mu$ what is $\tau$, so $\mu$ is known (and hence fixed). So the kernel for the conditional posterior for $\tau$ becomes (after some re-arranging): \begin{equation} \pi(\tau|\mu,\mathbf{y}) \propto \tau^{\frac{n}{2}+\alpha_0-1}\exp\{-\tau(\frac{1}{2} \sum_{i=1}^{n}(y_i-\mu)^2+\beta_0)\}, \end{equation} which is the kernel for the Gamma distribution with the parameters you state.
Now, for $\pi(\mu|\tau,\mathbf{y})$ you do the same thing, but it's a bit trickier because you have to complete the square. After cancelling the relevant terms not involving $\mu$ you get: \begin{equation} \pi(\mu|\tau,\mathbf{y}) \propto \exp\{ -\frac{\tau}{2} \sum_{i=1}^{n} (y_i-\mu)^2 - \frac{\tau_0}{2}(\mu-\mu_0)^2\}. \end{equation} If you multiply out the squared brackets and take the relevant summations, then remove all terms not involving $\mu$ (as they are all just multiplying the kernel by a constant) this becomes: \begin{equation} \pi(\mu|\tau,\mathbf{y}) \propto \exp\{ -\frac{1}{2}[\mu^2(n\tau+\tau_0) - 2\mu(\tau n\bar{y} - \tau_0\mu_0)]\}. \end{equation} If you complete the square here you get the kernel for a Gaussian with the mean and variance you state.
Hope that helps.