I have been racking my brains trying to understand Bayes theorem. So, the way I have understood is that the likelihood is the probability of observing the particular outcome given a set of parameter values. The prior describes our belief probabilities over the different parameter values. The denominator acts like a normaliser ensuring that the posterior is a valid distribution.
I have a couple of things I am not clear about:
1: The likelihood is a probability and the prior is a probability as well. So, why is the numerator already not normalised between 0 and 1? Assume I have not ignored any multiplicative constants when computing the likelihood.
2: One way I think of this denominator now is that if we have a bunch of numbers and we would like to normalize them between 0 and 1, one thing we can do is divide each of these numbers by their sum. Is this what the denominator in the Bayes theorem is affectively doing?
Best Answer
From a technical point of view, here is the argument:
For densities (but the argument is analogous in the discrete case), we write $$ \pi \left( \theta |y\right) =\frac{f\left( y|\theta \right) \pi \left(\theta \right) }{f(y)} $$ The norming constant can be obtained as, by writing a marginal density as a joint density and then writing the joint as conditional times marginal, with the other parameter integrated out, \begin{align*} f(y)&=\int f\left( y,\theta \right) d\theta\\ &=\int f\left( y|\theta \right) \pi \left(\theta \right)d\theta \end{align*} It ensures integration to 1 because \begin{align*} \int \pi \left( \theta |y\right) d\theta&=\int\frac{f\left( y|\theta \right) \pi \left(\theta \right) }{\int f\left( y|\theta \right) \pi \left(\theta \right)d\theta}d\theta\\ &=\frac{\int f\left( y|\theta \right) \pi \left(\theta \right) d\theta}{\int f\left( y|\theta \right) \pi \left(\theta \right)d\theta}\\ &=1, \end{align*} where we can "take out" the integral in the denominator because $\theta$ had already been integrated out there.