my first question so apologies in advance.
I'm currently studying posterior distributions and for practice sake I'm trying to directly compute the posterior distribution for the mean of a normal random variable (unit variance) with standard normal prior. I'm trying to validate if this is correct (note that this is for observing ONE new data point).
The posterior distribution is defined as
$$p(\theta\mid x)=\frac{p(x\mid\theta)p(\theta)}{\int_\theta p(x\mid\theta) p(\theta) \, d\theta}$$
and I am getting
$$p(\theta\mid x)=\frac{1}{\sqrt{\pi}}e^{-\frac{1}{4}x^2+x\theta-\theta^2}$$
I did a couple sanity checks,
$$\int_{-\infty}^\infty p(\theta\mid X=0)\,d\theta=\int_{-\infty}^\infty \frac{1}{\sqrt{\pi}} e^{-\theta^2} \, d\theta=1$$
$$\int_{-\infty}^\infty p(\theta\mid X=2) \, d\theta=\int_{-\infty}^\infty \frac{1}{\sqrt{\pi}} e^{-1+2\theta-\theta^2} \, d\theta=1$$
Is this the correct analytical solution?
My derivation is as follows:
$$\text{Let } X \sim N(\theta,1)$$
$$\text{Let } \theta \sim N(0,1)$$
The likelihood (dependent on theta):
$$p(x\mid\theta)=\frac{1}{\sqrt{2\pi}}e^{-\frac{1}{2}(x-\theta)^2}$$
The prior:
$$p(\theta)=\frac{1}{\sqrt{2\pi}}e^{-\frac{1}{2}\theta^2}$$
The joint density:
$$p(x\mid\theta)p(\theta)=\frac{1}{2\pi}e^{-\frac{1}{2}(x-\theta)^2}e^{-\frac{1}{2}\theta^2}$$
$$=\frac{1}{2\pi}e^{-\frac{1}{2}x^2+x\theta-\frac{1}{2}\theta^2}e^{-\frac{1}{2}\theta^2}$$
$$=\frac{1}{2\pi}e^{-\frac{1}{2}x^2+x\theta-\theta^2}$$
sanity check:
$$\int_{-\infty}^\infty \int_{-\infty}^\infty \frac{1}{2\pi}e^{-\frac{1}{2} x^2 + x\theta-\theta^2} \, dx \, d\theta=1$$
The marginal:
$$p(x)=\int_\theta p(x\mid\theta)p(\theta) \, d\theta$$
$$=\frac{1}{2\pi}e^{-\frac{1}{2}x^2} \int_{-\infty}^\infty e^{-\theta^2+\theta x} \, d\theta$$
$$=\frac{1}{2\pi}e^{-\frac{1}{2}x^2}\Biggl[\sqrt{\pi}e^{\frac{1}{4}x^2}\Biggl] \text{by Gaussian integral}$$
$$=\frac{1}{2\sqrt{\pi}}e^{-\frac{1}{4}x^2}$$
sanity check:
$$\int_{-\infty}^\infty \frac{1}{2\sqrt{\pi}}e^{-\frac{1}{4}x^2} \, dx=1$$
The posterior:
$$p(\theta\mid x)=\frac{p(x\mid\theta)p(\theta)}{\int_\theta p(x\mid \theta) p(\theta) \, d\theta} = \frac{\frac{1}{2\pi}e^{-\frac{1}{2}x^2+x\theta-\theta^2}}{\frac{1}{2\sqrt{\pi}}e^{-\frac{1}{4}x^2}}$$
$$=\frac{\sqrt{\pi}}{\pi}e^{-\frac{1}{2}x^2+x\theta-\theta^2+\frac{1}{4}x^2}$$
$$=\frac{1}{\sqrt{\pi}}e^{-\frac{1}{4}x^2+x\theta-\theta^2}$$
Best Answer
\begin{align} & (x-\theta)^2 + \theta^2 = \big(x^2-2x\theta+\theta^2\big) + \theta^2 \\[8pt] = {} & 2\theta^2 - 2x\theta + x^2 \\[6pt] = {} & 2(\theta^2 - x\theta) + x^2 \\[6pt] = {} & 2\left( \theta^2 - x\theta + \tfrac 1 4 x^2 \right) + x^2 -\tfrac 1 2 x^2 \quad\text{(completing the square)} \\[6pt] = {} & 2\left( \theta - \tfrac 1 2 x \right)^2 + \tfrac 1 2 x^2 \\[8pt] \text{So } & e^{-\frac 1 2(x-\theta)^2} e^{-\frac 1 2 \theta^2} = \exp\left(-\frac 1 {2\sigma^2} \left( \theta-\frac 1 2 x \right)^2 \right) \times \text{constant} \\[6pt] \text{where } & \sigma^2 = 1/2 \text{ and “constant” means not depending on } \theta. \end{align} Thus the posterior mean is $x/2$ and the posterior variance is $1/2.$