Solved – Bayesian update for Beta distribution

bayesianbeta distributiondirichlet distributioninference

I'm wondering how to find a posterior of a beta distribution when the "new information" is not an outcome of a binomial trial.

Let $p$ be the probability of Head of a (biased) coin toss. As usual in Bayesian inference, let $$p\sim Beta(a,b).$$

When the "new information" is Head or Tail, we can simply update $p$ by adding number of heads or tails to the shape parameters.

However, suppose that the new information I have is $$p\geq \frac{1}{2}.$$

If this is the case, how should I update the posterior in a Bayesian way?

In relation to the above question, and possibly more interestingly, for a Dirichlet distribution, if $$(p_1,p_2,p_3,p_4)\sim Dir(a,b,c,d)$$, what kind of Bayesian inference can be made out of the new information?: $$p_1+p_2\geq p_3+p_3$$

Best Answer

As already noticed by @whuber in a comment to answer by @BruceET, this is not really a Bayesian scenario, since you don't seem to mention any data (nor any likelihood).

From what you are saying, you know that $p \sim \mathsf{Beta}(a, b)$, you also know that $p \ge 1/2$, what translates to knowing that $p$ is distributed according to beta distribution with parameters $a,b$ left truncated at $1/2$.

Same with the Dirichlet distribution, your knowledge that $p_1+p_2\geq p_3+p_3$ is a constraint about the distribution, not an "update" of the prior. Moreover, notice that this constraint leads to situation that may not be possible under Dirichlet distribution, so in fact the statements may be contradictory. The statement is in fact, that the $p_1, p_2, p_3, p_4$ are distributed according to distribution similar to Dirichlet, but constrained.

So...

If you are saying that for $p$ you assume truncated beta distribution as a prior, and want to use it together with some likelihood function and data, it is no more conjugate to binomial distribution, so you would need to use Markov Chain Monte Carlo for estimation. Defining truncated distribution can be done in any probabilistic programming framework, e.g. Stan, PyMC3, JAGS etc.
Same as above applies to the "Dirichlet"-like distribution, but since this is a custom distribution, it would be much more complicated (I have no easy solution for you).
If you are saying that the facts mentioned by you are the only information that you have and will have, and given this information you want to learn something about the distribution (e.g. expected value, quantiles), then this is a typical case of standard Monte Carlo simulation. For truncated beta, you could simply use inverse transform sampling, that is a simple and efficient way of sampling. For the "Dirichlet"-like distribution, it would again, be more complicated, but there are many possible approaches, starting from simple accept-reject sampling, ending at some more sophisticated solutions.

Related Solutions

Probability Bayesian Beta Distribution – Beta Distribution on Flipping a Coin Explained

The quotation is a "logical sleight-of-hand" (great expression!), as noted by @whuber in comments to the OP. The only thing we can really say after seeing that the coin has an head and a tail, is that both the events "head" and "tail" are not impossible. Thus we could discard a discrete prior which puts all of the probability mass on "head" or on "tail". But this doesn't lead, by itself, to the uniform prior: the question is much more subtle. Let's first of all summarize a bit of background. We're considering the Beta-Binominal conjugate model for Bayesian inference of the probability $\theta$ of heads of a coin, given $n$ independent and identically distributed (conditionally on $\theta$) coin tosses. From the expression of $p(\theta|x)$ when we observe $x$ heads in $n$ tosses:

$$ p(\theta|x) = Beta(x+\alpha, n-x+\beta)$$

we can say that $\alpha$ and $\beta$ play the roles of a "prior number of heads" and "prior number of tails" (pseudotrials), and $\alpha+\beta$ can be interpreted as an effective sample size. We could also arrive at this interpretation using the well-known expression for the posterior mean as a weighted average of the prior mean $\frac{\alpha}{\alpha+\beta}$ and the sample mean $\frac{x}{n}$.

Looking at $p(\theta|x)$, we can make two considerations:

since we have no prior knowledge about $\theta$ (maximum ignorance), we intuitively expect the effective sample size $\alpha+\beta$ to be "small". If it were large, then the prior would be incorporating quite a lot of knowledge. Another way of seeing this is noting that if $\alpha$ and $\beta$ are "small" with respect to $x$ and $n-x$, the posterior probability won't depend a lot on our prior, because $x+\alpha\approx x$ and $n-x+\beta\approx n-x$. We would expect that a prior which doesn't incorporate a lot of knowledge must quickly become irrelevant in light of some data.
Also, since $\mu_{prior}=\frac{\alpha}{\alpha+\beta}$ is the prior mean, and we have no prior knowledge about the distribution of $\theta$, we would expect $\mu_{prior}=0.5$. This is an argument of symmetry - if we don't know any better, we wouldn't expect a priori that the distribution is skewed towards 0 or towards 1. The Beta distribution is

$$f(\theta|\alpha,\beta)=\frac{\Gamma(\alpha + \beta)}{\Gamma(\alpha) +\Gamma(\beta)}\theta^{\alpha-1}(1-\theta)^{\beta-1}$$

This expression is only symmetric around $\theta=0.5$ if $\alpha=\beta$.

For these two reasons, whatever prior (belonging to the Beta family - remember, conjugate model!) we choose to use, we intuitively expect that $\alpha=\beta=c$ and $c$ is "small". We can see that all the three commonly used non-informative priors for the Beta-Binomial model share these traits, but other than that, they are quite different. And this is obvious: no prior knowledge, or "maximum ignorance", is not a scientific definition, so what kind of prior expresses "maximum ignorance", i.e., what's a non-informative prior, depends on what you actually mean as "maximum ignorance".

we could choose a prior which says that all values for $\theta$ are equiprobable, since we don't know any better. Again, a symmetry argument. This corresponds to $\alpha=\beta=1$:

$$f(\theta|1,1)=\frac{\Gamma(2)}{2\Gamma(1)}\theta^{0}(1-\theta)^{0}=1$$

for $\theta\in[0,1]$, i.e., the uniform prior used by Kruschke. More formally, by writing out the expression for the differential entropy of the Beta distribution, you can see that it is maximized when $\alpha=\beta=1$. Now, entropy is often interpreted as a measure of "the amount of information" carried by a distribution: higher entropy corresponds to less information. Thus, you could use this maximum entropy principle to say that, inside the Beta family, the prior which contains less information (maximum ignorance) is this uniform prior.
You could choose another point of view, the one used by the OP, and say that no information corresponds to having seen no heads and no tail, i.e.,

$$\alpha=\beta=0 \Rightarrow \pi(\theta) \propto \theta^{-1}(1-\theta)^{-1}$$

The prior we obtain this way is called the Haldane prior. The function $\theta^{-1}(1-\theta)^{-1}$ has a little problem - the integral over $I=[0, 1]$ is infinite, i.e., no matter what the normalizing constant, it cannot be transformed into a proper pdf. Actually, the Haldane prior is a proper pmf, which puts probability 0.5 on $\theta=0$, 0.5 on $\theta=1$ and 0 probability on all other values for $\theta$. However, let's not get carried away - for a continuous parameter $\theta$, priors which don't correspond to a proper pdf are called improper priors. Since, as noted before, all that matters for Bayesian inference is the posterior distribution, improper priors are admissible, as long as the posterior distribution is proper. In the case of the Haldane prior, we can prove that the posterior pdf is proper if our sample contains at least one success and one failure. Thus we can only use the Haldane prior when we observe at least one head and one tail.

There's another sense in which the Haldane prior can be considered non-informative: the mean of the posterior distribution is now $\frac{\alpha + x}{\alpha + \beta + n}=\frac{x}{n}$, i.e., the sample frequency of heads, which is the frequentist MLE estimate of $\theta$ for the Binomial model of the coin flip problem. Also, the credible intervals for $\theta$ correspond to the Wald confidence intervals. Since frequentist methods don't specify a prior, one could say that the Haldane prior is noninformative, or corresponds to zero prior knowledge, because it leads to the "same" inference a frequentist would make.
Finally, you could use a prior which doesn't depend on the parametrization of the problem, i.e., the Jeffreys prior, which for the Beta-Binomial model corresponds to

$$\alpha=\beta=\frac{1}{2} \Rightarrow \pi(\theta) \propto \theta^{-\frac{1}{2}}(1-\theta)^{-\frac{1}{2}}$$

thus with an effective sample size of 1. The Jeffreys prior has the advantage that it's invariant under reparametrization of the parameter space. For example, the uniform prior assigns equal probability to all values of $\theta$, the probability of the event "head". However, you could decide to parametrize this model in terms of log-odds $\lambda=log(\frac{\theta}{1-\theta})$ of event "head", instead than $\theta$. What's the prior which expresses "maximum ignorance" in terms of log-odds, i.e., which says that all possible log-odds for event "head" are equiprobable? It's the Haldane prior, as shown in this (slightly cryptic) answer. Instead, the Jeffreys is invariant under all changes of metric. Jeffreys stated that a prior which doesn't have this property, is in some way informative because it contains information on the metric you used to parametrize the problem. His prior doesn't.

To summarize, there's not just one unequivocal choice for a noninformative prior in the Beta-Binomial model. What you choose depends on what you mean as zero prior knowledge, and on the goals of your analysis.

Solved – Understanding convergence in Bayesian inference of coin tossing

Yes, it does converge to the "true distribution" (suitably defined)

First of all, it is worth noting that it is a little strange to refer to the "true distribution" of the parameter as something aside from the prior and posterior. If you proceed under the operational Bayesian approach then the parameter has an operational definition as a function of the sequence of observable values, and so it is legitimate to refer to a "true value" of the parameter (see esp. Bernardo and Smith 2000). The standard convergence theorems in Bayesian statistics show that the posterior converges weakly to the true parameter, defined operationally through the law-of-large numbers. It is less common to refer to a "true distribution" of the parameter, as something apart from the prior or posterior. If I understand your intention correctly, that would essentially just be a point-mass distribution on the true value. If that is what you mean, then yes, the posterior will converge (weakly) to this. That is really just another restatement of the standard convergence theorems in Bayesian statistics.

Taking $\mathbf{X} = (X_1, X_2, X_3, ...)$ to be the sequence of observable coin-toss outcomes, we define $p_H$ operationally as a function of $\mathbf{X}$. In Bayesian analysis with IID data the most operational definition of the parameters is an index to the limiting empirical distribution of the observable sequence (see O'Neill 2009 for discussion and details). Now, the beta posterior you are referring to arises in the IID model:

$$X_1,X_2,X_3,... | p_H \sim \text{IID Bern}(p_H).$$

The parameter $p_H$ can be given an operational definition as the limit of the sample mean of the observable coin-tosses (see O'Neill 2009 again). To facilitate our analysis, we will use the notation $\hat{p}_H \equiv \lim N_H/N$ to denote this limit, so the "true value" of the parameter $p_H$ is the point $\hat{p}_H$. (In other words $p_H$ is $\hat{p}_H$, but we will use two separate referents to elucidate the convergence.)

Your posterior distribution does indeed converge to the "true distribution" of $p_H$, which is a point-mass distribution on $\hat{p}_H$. To see this, we first derive the asymptotic mean and variance$^\dagger$ of the posterior:

$$\begin{equation} \begin{aligned} \lim_{N \rightarrow \infty} \mathbb{E}(p_H| \mathbf{X}_N) &= \lim_{N \rightarrow \infty} \frac{a_0+N_H}{a_0 + b_0 + N} \\[6pt] &= \lim_{N \rightarrow \infty} \frac{N_H}{N} \cdot \frac{a_0+N_H}{N_H} \Bigg/ \frac{a_0+b_0+N}{N} \\[6pt] &\overset{a.s}{=} \lim_{N \rightarrow \infty} \frac{N_H}{N} = \hat{p}_H. \\[6pt] \lim_{N \rightarrow \infty} \mathbb{V}(p_H| \mathbf{X}_N) &= \lim_{N \rightarrow \infty} \frac{(a_0+N_H)(b_0+N_T)}{(a_0 + b_0 + N)^2(a_0 + b_0 + N+1)} \\[6pt] &= \lim_{N \rightarrow \infty} \frac{a_0+N_H}{a_0 + b_0 + N} \cdot \frac{b_0+N_T}{a_0 + b_0 + N} \cdot \frac{1}{a_0 + b_0 + N + 1} \\[6pt] &\leqslant \lim_{N \rightarrow \infty} \frac{1}{a_0 + b_0 + N + 1} \\[6pt] &= 0. \\[6pt] \end{aligned} \end{equation}$$

So, we have $\mathbb{E}(p_H| \mathbf{X}_N) \overset{a.s}{\rightarrow} \hat{p}_H$ and $\mathbb{V}(p_H| \mathbf{X}_N) \rightarrow 0$, which gives convergence in mean-square to the true parameter value $\hat{p}_H$. Using Markov's inequality this implies convergence in probability to $\hat{p}_H$, which further implies convergence in probability of the posterior to the point-mass distribution on $\hat{p}_H$. This means that we have weak convergence to the "true distribution" of the parameter.

$^\dagger$ Your stated posterior variance is incorrect - I have used the correct posterior variance in my working.

Best Answer

Related Solutions

Probability Bayesian Beta Distribution – Beta Distribution on Flipping a Coin Explained

Solved – Understanding convergence in Bayesian inference of coin tossing

Yes, it does converge to the "true distribution" (suitably defined)

Related Question