We are given the following information:
$\Theta = \mathbb{R}, Y \in \mathbb{R}, p_\theta=N(\theta, 1), \pi = N(0, \tau^2)$. I am asked to compute the posterior. So I know this can be computed with the following 'adaptation' of Bayes's Rule: $\pi(\theta \mid Y) \propto p_\theta(Y)\pi(\theta)$. Also, I've used that we have a normal distribution for the likelihood and a normal distribution for the prior, and we know that the posterior is going to be a normal distribution again due to conjugacy. But I don't really know how to derive the parameters of the posterior normal distribution. So far, I have computed the following:
$\pi(\theta \mid Y) \propto p_\theta(Y)\pi(\theta)$
$= \exp(\theta/\tau^2+(Y-\theta)^2/2)$
$=\exp((2\theta^2+\tau^2(Y-2\theta Y+\theta^2))/\tau^2)$
$\propto \exp(\theta^2/\tau^2-\theta/Y+\theta^2/2)$
But now I'm stuck. I know we need to work towards a form $\exp((\theta-a)^2/b^2)$, but I don't know how to get there. Any tips or a derivation of the parameters would be appreciated.
(this question was also asked here, Gaussian product – posterior probability distribution but it was never answered and less detailed.)
Best Answer
Your notation is confusing because it is not clear how the hierarchical model is specified. Based on the expression $\pi(\theta \mid Y) \propto p_\theta(Y)\pi(\theta)$, I can only assume that the model should be $$Y \mid \Theta \sim \operatorname{Normal}(\mu = \Theta, \sigma^2 = 1), \\ \Theta \sim \operatorname{Normal}(\mu = 0, \sigma^2 = \tau^2).$$
As the posterior for $\Theta$ is proportional to the joint density, we write $$\begin{align*} \pi_{\Theta \mid Y}(\theta \mid y) &\propto f_{Y \mid \Theta}(y_i \mid \theta)\pi_\Theta(\theta) \\ &\propto \exp\left(-\frac{(y - \theta)^2}{2}\right)\exp\left(-\frac{\theta^2}{2\tau^2}\right)\\ &= \exp\left(-\frac{\tau^2 (y-\theta)^2 + \theta^2}{2\tau^2}\right). \end{align*}$$ This motivates us to complete the square with respect to $\theta$:
$$\begin{align*} \theta^2 + \tau^2(y-\theta)^2 &= (1+\tau^2)\theta^2 - 2\tau^2 y \theta + \tau^2 y^2 \\ &= (1+\tau^2) \left(\theta^2 - \frac{2\tau^2 y}{1+\tau^2} \theta\right) + (\tau y)^2 \\ &= (1+\tau^2) \left(\theta - \frac{\tau^2 y}{1+\tau^2}\right)^{\!2} - \frac{(\tau^2 y)^2}{1+\tau^2} + (\tau y)^2 \\ &= (1 + \tau^2)\left(\theta - \frac{\tau^2 y}{1+\tau^2} \right)^{\!2} + \frac{(\tau y)^2}{1+\tau^2}. \end{align*}$$ Consequently, $$\pi_{\Theta \mid Y}(\theta \mid y) \propto \exp \left( - \frac{(\theta - \kappa)^2}{2\varsigma^2} \right) \exp\left(-\frac{y^2}{2(1+\tau^2)}\right) \propto \exp \left( - \frac{(\theta - \kappa)^2}{2\varsigma^2} \right) $$ where $$\kappa = \frac{\tau^2 y}{1+\tau^2}, \quad \varsigma^2 = \frac{\tau^2}{1+\tau^2}$$ represent the posterior mean and variance of $\Theta$. The constant factor that was dropped in the last proportionality is independent of $\theta$. Therefore, our posterior is $$\Theta \mid Y \sim \operatorname{Normal}(\mu = \kappa, \sigma^2 = \varsigma^2).$$ Since $\tau$ is typically used to denote precision rather than variance, i.e. $\tau \sigma^2 = 1$, if this is the intended parametrization, the posterior mean and variance would be written as $$\kappa = \frac{y}{1+\tau}, \quad \varsigma^2 = \frac{1}{1+\tau}.$$