Part of a series of trying to understand Bayesian vs frequentist: 1 2 3 4 5 6 7
I think I get the difference of how Bayesians and frequentists approach choosing between hypotheses, but I'm not quite sure if or how that is supposed to explain to me how they view probability.
From what I understand, according to Wiki, a frequentist "defines" probability as follows:
Given probability space $(\Omega, \mathscr{F}, \mathbb{P})$, $\forall A \in \mathscr{F}$, $\mathbb{P}(A) \approx \frac{n_A}{n_t}$, where $n_t$ is the number of trials conducted and $n_A$ is the number of times A has occurred in those trials.
Furthermore, $\mathbb{P}(A) = \lim_{n_t \to \infty} \frac{n_A}{n_t}$.
Okay, so how do Bayesians define probability? The above seems to be one approach to computing probability of an event in addition to defining a probability.
Bayesians seems to assume a prior probability, conduct some trials and then update their probability, but that doesn't really seem to explain how they define what probability is.
Wiki says 'Bayesian probability is a quantity that we assign for the purpose of representing a state of knowledge, or a state of belief.'
What exactly does that mean? Is state synonymous to degree? For example, Walter's state of belief that a particular coin is fair is represented with the number 0.1 while Jesse's state of belief that the same coin is fair is represented with the number 0.2. Given new information, Walter's state of belief could become 0.96 while Jesse's state of belief could become 0.03. So, initially, Walter was less inclined to believe the coin is fair, but later on Jesse was more inclined to believe the coin is fair?
I'm hoping for something in terms of symbols like the frequentist one above.
Same Wiki page says 'The Bayesian interpretation of probability can be seen as an extension of propositional logic that enables reasoning with hypotheses, i.e., the propositions whose truth or falsity is uncertain.', it seems like Bayesian and frequentist probability are analogous to fuzzy and Boolean logic, respectively.
Best Answer
I believe that most 'frequentists' and 'Bayesians' would rigorously define probability in the same way: via Kolmogorov's axioms and measure theory, modulo some issues about finite vs countable additivity, depending on who you're talking to. So in terms of 'symbols' I reckon you'll likely find more or less the same definition across the board. Everyone agrees on how probabilities behave.
I would say the primary difference is in the interpretation of what probabilities are. My (tongue-in-cheek militant Bayesian) preferred interpretation is that probabilities are coherent representations of information about events.
'Coherent' here has a technical meaning: it means that if I represent my information about the world in terms of probabilities and then use those probabilities to size my bets on the occurrence or nonoccurrence of any given event, I am assured that I can not be made a sure loser by agents betting against me.
Note that this involves no notion of 'long-run relative frequency'; indeed, I can coherently represent my information about a one-off event - like the sun exploding tomorrow - via the language of probability. On the other hand, it seems more difficult (or arguably less natural) to talk about the event "the sun will explode tomorrow" in terms of long-run relative frequency.
For a deep dive on this question I'd refer you to the first chapter of Jay Kadane's excellent (and free) Principles of Uncertainty.
UPDATE: I wrote a relatively informal blog post that illustrates coherence.