Solved – How do Bayesians compare distributions

bayesiandistributions

So, I think that I have a decent grasp of the basics of frequentist probability and statistical analysis (and how badly it can be used). In a frequentist world, it makes sense to ask such a question as "is this distribution different from that distribution", because distributions are assumed to be real, objective and unchanging (for a given situation, at least), and so we can figure out how likely it is that one sample is drawn from a distribution shaped like another sample.

In the Bayesian world view, we only care about what we expect to see, given our past experiences (I'm still a bit vague on this part, but I understand the concept of Bayesian updating). If that is so, how can a Bayesian say "this set of data is different from that set of data"?

For the purposes of this question, I don't care about statistical significance, or similar, just how to quantify difference. I'm equally interested in parametric and non-parametric distributions.

Best Answer

Think your statement through as a Frequentist and make it more specific first. A Frequentist could not say that "data set A is different from data set B", without any further clarification.

First, you'd have to state what you mean by "different". Perhaps you mean "have different mean values". Then again, you might mean "have different variances". Or perhaps something else?

Then, you'd have to state what kind of test you would use, which depends on what you believe are valid assumptions about the data. Do you assume that the data sets are both normally-distributed about some means? Or do you believe that they are both Beta-distributed? Or something else?

Now can you see that the second decision is much like the priors in Bayesian statistics? It's not just "my past experience", but is rather what I believe, and what I believe my peers will believe, are reasonable assumptions about my data. (And Bayesians can use uniform priors, which pushes things towards Frequentist calculations.)

EDIT: In response to your comment: the next step is contained in the first decision I mentioned. If you want to decide whether the means of two groups are different, you would look at the distribution of the difference of the means of the two groups to see if this distribution does or does not contain zero, at some level of confidence. Exactly how close to zero you count as zero and exactly which portion of the (posterior) distribution you use are determined by you and the level of confidence you desire.

A discussion of these ideas can be found in a paper by Kruschke, who also wrote a very readable book Doing Bayesian Data Analysis, which covers an example on pages 307-309, "Are Different Groups Equal?". (Second edition: p. 468-472.) He also has a blog posting on the subject, with some Q&A.

FURTHER EDIT: Your description of the Bayesian process is also not quite correct. Bayesians only care about what the data tells us, in light of what we knew independent of the data. (As Kruschke points out, the prior does not necessarily occur before the data. That's what the phrase implies, but it's really just our knowledge excluding some of the data.) What we knew independently of a particular set of data may be vague or specific and may be based on consensus, a model of the underlying data generation process, or may just be the results of another (not necessarily prior) experiment.

Related Question