Can someone give a good rundown of the differences between the Bayesian and the frequentist approach to probability?
From what I understand:
The frequentists view is that the data is a repeatable random sample (random variable) with a specific frequency/probability (which is defined as the relative frequency of an event as the number of trials approaches infinity). The underlying parameters and probabilities remain constant during this repeatable process and that the variation is due to variability in $X_n$ and not the probability distribution (which is fixed for a certain event/process).
The bayesian view is that the data is fixed while the frequency/probability for a certain event can change meaning that the parameters of the distribution changes. In effect, the data that you get changes the prior distribution of a parameter which gets updated for each set of data.
To me it seems that the frequentist approach is more practical/logical since it seems reasonable that events have a specific probability and that the variation is in our sampling.
Furthermore, most data analysis from studies is usually done using the frequentist approach (i.e. confidence intervals, hypothesis testing with p-values etc) since it is easily understandable.
I was just wondering whether anyone could give me a quick summary of their interpretation of bayesian vs frequentist approach including bayesian statistical equivalents of the frequentist p-value and confidence interval. In addition, specific examples of where 1 method would be preferable to the other is appreciated.
Best Answer
In the frequentist approach, it is asserted that the only sense in which probabilities have meaning is as the limiting value of the number of successes in a sequence of trials, i.e. as
$$p = \lim_{n\to\infty} \frac{k}{n}$$
where $k$ is the number of successes and $n$ is the number of trials. In particular, it doesn't make any sense to associate a probability distribution with a parameter.
For example, consider samples $X_1, \dots, X_n$ from the Bernoulli distribution with parameter $p$ (i.e. they have value 1 with probability $p$ and 0 with probability $1-p$). We can define the sample success rate to be
$$\hat{p} = \frac{X_1+\cdots +X_n}{n}$$
and talk about the distribution of $\hat{p}$ conditional on the value of $p$, but it doesn't make sense to invert the question and start talking about the probability distribution of $p$ conditional on the observed value of $\hat{p}$. In particular, this means that when we compute a confidence interval, we interpret the ends of the confidence interval as random variables, and we talk about "the probability that the interval includes the true parameter", rather than "the probability that the parameter is inside the confidence interval".
In the Bayesian approach, we interpret probability distributions as quantifying our uncertainty about the world. In particular, this means that we can now meaningfully talk about probability distributions of parameters, since even though the parameter is fixed, our knowledge of its true value may be limited. In the example above, we can invert the probability distribution $f(\hat{p}\mid p)$ using Bayes' law, to give
$$\overbrace{f(p\mid \hat{p})}^\text{posterior} = \underbrace{\frac{f(\hat{p}\mid p)}{f(\hat{p})}}_\text{likelihood ratio} \overbrace{f(p)}^\text{prior}$$
The snag is that we have to introduce the prior distribution into our analysis - this reflects our belief about the value of $p$ before seeing the actual values of the $X_i$. The role of the prior is often criticised in the frequentist approach, as it is argued that it introduces subjectivity into the otherwise austere and object world of probability.
In the Bayesian approach one no longer talks of confidence intervals, but instead of credible intervals, which have a more natural interpretation - given a 95% credible interval, we can assign a 95% probability that the parameter is inside the interval.