Bayesian – Bayesian Updating with Discrete Support Explained

bayesianbinomial distributionconvergenceposterior

Suppose I have a coin that is either fair ($Pr(H)=0.5$) or biased ($Pr(H)=0.2$). I have a prior probability $Pr(fair)=\tau_{0}$. I observe a sequence of tosses with outcome $X_{1},X_{2},…,X_{n}$, where $X_{i} \in \{H,T\}$.
Does the posterior $\tau_{n}=Pr(fair|X_{1},…,X_{n})$ follow any well known/well behaved probability distribution?
Is there a way to show that, for $n$ sufficiently big, the posterior converges to the truth?

Best Answer

Because the support is discrete (assuming someone tells you something like "I've used either a fair coin or a coin with a known bias of 0.2), this is really just a straight forward application of Bayes' Rule.

Recall

$$ Pr(\theta=0.5 \vert X_1, \dots, X_n) = \dfrac{Pr(X_1, \dots, X_n \vert \theta=0.5)\tau_0}{Pr(X_1, \dots, X_n \vert \theta=0.5)\tau_0 + Pr(X_1, \dots, X_n \vert \theta=0.2)(1-\tau_0)}$$

Because you're talking about coinflips, the likelihood is binomial

$$ Pr(X_1, \dots, X_n \vert \theta) = \binom{n}{k}\theta^k(1-\theta)^{n-k} $$

where $k$ is the number of heads in the sample. Because your space is only two possible values, you need only compute the first quantity and then subtract it from 1 to find the other.

So far as converging to the truth, there will come a point in time when the likelihood dominates the prior. This fact, along with the law of large numbers, should be enough to justify your point, assuming these are the only 2 biases possible.