Bayesian Statistics – Why and How to Update Prior in Bayesian Analysis

bayesbayesianlikelihoodposteriorprior

I'm still trying to understand prior and posterior distributions in Bayesian inference.

In this question, one flips a coin. Priors:

unfair is 0.1, and being fair is 0.9

Coin is flipped 10x and is all heads. Based on this evidence, P(unfair | evidence) ~ 0.653 and P(fair | evidence) ~ 1-0.653.

Apparently:

Both priors (originally 0.1 and 0.9) must be updated. – user3697176
Apr 27 at 4:38

  1. What exactly does it mean to update a prior? I'm not sure I understand Wiki. Like we replace 0.1 and 0.9 with 0.653 and 1-0.653 and then run another experiment and continue to do so until the priors stabilize?

Question 2 might just follow from 1 but anyway:

  1. Why must they be updated? I'm guessing because 0.1 and 0.9 are sufficiently far from 0.653 and 1-0.653.

If that is right, how far must they be from 0.1 and 0.9 to say that priors must be updated? What if evidence gave us 0.11 and 0.89?

If that is wrong, why then?

Best Answer

In plain english, update a prior in bayesian inference means that you start with some guesses about the probability of an event occuring (prior probability), then you observe what happens (likelihood), and depending on what happened you update your initial guess. Once updated, your prior probability is called posterior probability.

Of course, now you can:

  • stop with your posterior probability;
  • use you posterior probability as a new prior, and update such a probability to obtain a new posterior by observing more evidence (i.e. data).

Essentially, updating a prior means that you start with a (informed) guess and you use evidence to update your initial guess. Recall that

$$ p(\theta | x) = \frac{p(x|\theta)p(\theta)}{p(x)},$$

where $p(\theta)$ is your prior, $p(x|\theta)$ is the likelihood (i.e. the evidence that you use to update the prior), and $p(\theta|x)$ is the posterior probability. Notice that the posterior probability is a probability given the evidence.

Example of coins: You start with the guess the probability of the coin being fair is $p = 0.1$. Then, you toss 10 times the coin, and you obtain a posterior probability $p = 0.3$. At this point, you can decide to be satisfied with $p = 0.3$ or toss the coin again (say 90 times): in this case, your prior will be $p = 0.3$ -- i.e. the posterior becomes the new prior -- and you will obtain a new posterior probability depending on new evidence.

Suppose that after 1000 tosses your posterior probability is $p = 0.9$. At the beginning you prior was $p = 0.1$, so you were supposing that the coin was unfair. Now, based on the evidence of 1000 tosses, you see that the probability of the coin being fair is high.

Notice that the fact that you can easily update a probability as you have new evidences is a strength of the bayesian framework. The point here is not that the prior must be updated, but to use all the available evidence to update your guess about a certain probability.

Related Question