I came across some strange finding that currently puzzles me. So I thought I'd share it here hoping that somebody can enlighten me. The setting is the following: Suppose you search for a posterior distribution of a random variable $\tau \in \mathbb{R}_{>0}$
$$p(\tau \vert x, \mu) = \frac{p(x \vert \tau, \mu)p(\tau \vert \mu)}{\int_{\mathbb{R}_{>0}} p(x \vert \tau, \mu)p(\tau \vert \mu) d\tau}$$
Now in the specific case that
$$ p(x \vert \tau, \mu) = \mathcal{N}(x \vert \mu, \tau) = \frac{1}{\sqrt{2\pi\tau}} \exp\left( -\frac{(x – \mu)^2}{2\tau}\right) $$
and
$$p(\tau \vert \mu) = \textit{Expo}\left(\tau\ \middle|\ \frac{1}{2 \lambda^2}\right) = \frac{1}{2 \lambda^2} \exp \left(- \frac{\tau}{2 \lambda^2} \right)$$
above posterior reduces to
$$p(\tau \vert x, \mu) = \frac{\mathcal{N}(x \vert \mu, \tau) \textit{Expo}\left(\tau\ \middle|\ \frac{1}{2 \lambda^2}\right)}{\mathcal{L}(x \vert \mu, \lambda)}$$,
where the term in denominator is the Laplacian Distribution
$$
\mathcal{L}(x \vert \mu, \lambda) = \frac{1}{2\lambda} \exp\left(-\frac{\vert x – \mu \vert}{\lambda}\right).
$$
This result, i.e. $\mathcal{L}(x \vert \mu, \lambda) = \int_{\mathbb{R}_{>0}} \mathcal{N}(x \vert \mu, \tau) \textit{Expo}\left(\tau\ \middle|\ \frac{1}{2 \lambda^2}\right) d\tau$, is e.g. shown here.
So far so good! Now what I intend to show is that the posterior follows an Inverse Gaussian distribution
$$
p(\tau \vert x, \mu) = \mathcal{IG}\left(\tau\ \middle|\ \vert x – \mu\vert \lambda, \vert x – \mu \vert^2 \right) = \sqrt{\frac{\vert x – \mu \vert^2}{2 \pi \tau^3}} \exp\left( – \frac{\vert x – \mu \vert ^2 \left(\tau – \vert x – \mu \vert \lambda\right)^2}{2 (\vert x – \mu \vert \lambda)^2 \tau}\right).
$$
The problem now is that when I reformulate the expression of the posterior I only get the following results
\begin{align*}
&\frac{\mathcal{N}(x \vert \mu, \tau) \textit{Expon}\left(\tau \middle| \frac{1}{2\lambda^2}\right)}{\mathcal{L}(x \vert \mu, \lambda)} \\
=\ &\frac{\frac{1}{\sqrt{2 \pi \tau}} \exp\left( – \frac{(x – \mu)^2}{2 \tau} \right) \frac{1}{2 \lambda^2} \exp\left(-\frac{\tau}{2 \lambda^2} \right)}{\frac{1}{2\lambda} \exp\left(-\frac{\vert x – \mu \vert}{\lambda}\right)} \\
=\ &\frac{\frac{1}{\lambda \sqrt{2 \pi \tau}} \exp\left( – \frac{(x – \mu)^2}{2 \tau} \right) \exp\left(-\frac{\tau}{2 \lambda^2} \right)}{\exp\left(-\frac{\vert x – \mu \vert}{\lambda}\right)} \\
=\ &\frac{\frac{1}{\lambda \sqrt{2 \pi \tau}} \exp\left( – \frac{\tau^2 + (x – \mu)^2 \lambda^2}{2 \lambda^2 \tau} \right)}{\exp\left(-\frac{\vert x – \mu \vert}{\lambda}\right)} \\
=\ &\frac{\frac{1}{\lambda \sqrt{2 \pi \tau}} \exp\left( – \frac{\left(\tau – \vert x – \mu \vert \lambda\right)^2}{2 \lambda^2 \tau} – \frac{2 \tau \vert x – \mu \vert \lambda}{2 \lambda^2 \tau}\right)}{\exp\left(-\frac{\vert x – \mu \vert}{\lambda}\right)} \\
=\ &\frac{\frac{1}{\lambda \sqrt{2 \pi \tau}} \exp\left( – \frac{\left(\tau – \vert x – \mu \vert \lambda\right)^2}{2 \lambda^2 \tau}\right) \exp \left( – \frac{\vert x – \mu \vert}{\lambda}\right)}{\exp\left(-\frac{\vert x – \mu \vert}{\lambda}\right)} \\
=\ &\frac{1}{\lambda \sqrt{2 \pi \tau}} \exp\left( – \frac{\vert x – \mu \vert ^2 \left(\tau – \vert x – \mu \vert \lambda\right)^2}{2 (\vert x – \mu \vert \lambda)^2 \tau}\right).
\end{align*}
Now from the last line it becomes obvious that I would only obtain the desired results if
$$
\lambda = \frac{\tau}{\vert x – \mu \vert}.
$$
However, I do not see why I would need this assumption, since I know that the posterior is properly normalized in any case and the exponential term indicates that I am indeed dealing with an Inverse Gaussian distribution.
Can somebody help me out here?
Best Answer
I have now checked my proof again. Interestingly, the expression for the posterior that I derived is correct and indeed integrates to one, which is easily proved in the following way
\begin{align*} &\int_{\mathbb{R}_{>0}} \frac{1}{\lambda \sqrt{2 \pi \tau}} \exp\left( - \frac{\vert x - \mu \vert ^2 \left(\tau - \vert x - \mu \vert \lambda\right)^2}{2 (\vert x - \mu \vert \lambda)^2 \tau}\right) d\tau\\ =& \frac{1}{\vert x - \mu \vert \lambda} \int_{\mathbb{R}_{>0}} \tau \sqrt{\frac{\vert x - \mu \vert^2}{2 \pi \tau^3}} \exp\left( - \frac{\vert x - \mu \vert ^2 \left(\tau - \vert x - \mu \vert \lambda\right)^2}{2 (\vert x - \mu \vert \lambda)^2 \tau}\right) d\tau \\ =& \frac{1}{\vert x - \mu \vert \lambda} \vert x - \mu \vert \lambda = 1. \end{align*}
The second reformulation is possible because the integral can be identified as the mean of an Inverse Gaussian with mean $\vert x - \mu \vert \lambda$.
The new question that this yields is: Does anyone know this probability distribution? Or is it something new.
Even more exciting: It is also easy to show that if $\tau$ is distributed according to the derived probability distribution, I will refer to it from now as
$$ \mathcal{IG}^{-1}\left(\tau\ \middle|\ \vert x - \mu \vert \lambda, \vert x - \mu \vert^2\right) = \int_{\mathbb{R}_{>0}} \frac{1}{\lambda \sqrt{2 \pi \tau}} \exp\left( - \frac{\vert x - \mu \vert ^2 \left(\tau - \vert x - \mu \vert \lambda\right)^2}{2 (\vert x - \mu \vert \lambda)^2 \tau}\right) d\tau $$
then $\tau^{-1}$ is distributed according to
$$\mathcal{IG}\left(\tau^{-1}\ \middle|\ \frac{1}{\vert x - \mu \vert \lambda}, \frac{1}{\lambda^2}\right)$$.
The mean of $\mathcal{IG}^{-1}\left(\tau\ \middle|\ \vert x - \mu \vert \lambda, \vert x - \mu \vert^2\right)$ seems to be given by
$$ \lambda^2 + \vert x - \mu \vert \lambda $$
if I have not screwed up with the calculations.