Solved – Subjectivity in Frequentist Statistics

bayesianfrequentistinterpretationphilosophical

I often hear the claim that Bayesian statistics can be highly subjective. The main argument being that inference depends on the choice of a prior (even though one could use the principle of indifference o maximum entropy to choose a prior). In comparison, the claim goes, frequentist statistics is in general more objective. How much truth is there in this statement?

Also, this makes me wonder:

  1. What are concrete elements of frequentist statistics (if any) that can be particularly subjective and that are not present or are less important in Bayesian statistics?
  2. Is subjectivity more prevalent in Bayesian than in frequentist statistics?

Best Answer

I often hear the claim that Bayesian statistics can be highly subjective.

So do I. But notice that there's a major ambiguity in calling something subjective.

Subjectivity (both senses)

Subjective can mean (at least) one of

  1. depends on the idiosyncracies of the researcher
  2. explicitly concerned with the state of knowledge of an individual

Bayesianism is subjective in the second sense because it is always offering a way to update beliefs represented by probability distributions by conditioning on information. (Note that whether those beliefs are beliefs that some subject actually has or just beliefs that a subject could have is irrelevant to deciding whether it is 'subjective'.)

The main argument being that inference depends on the choice of a prior

Actually, if a prior represents your personal belief about something then you almost certainly didn't choose it at any more than you chose most of your beliefs. And if it represents somebody's beliefs then it can be a more or less accurate representation of those beliefs, so ironically there will be a rather 'objective' fact about how well it represents them.

(even though one could use the principle of indifference o maximum entropy to choose a prior).

One could, though this doesn't tend to generalize very smoothly to continuous domains. Also, arguably it's impossible to be flat or 'indifferent' in all parameterisations at once (though I've never been quite sure why you'd want to be).

In comparison, the claim goes, frequentist statistics is in general more objective. How much truth is there in this statement?

So how might we evaluate this claim?

I suggest that in the second second sense of subjective: it's mostly correct. And in the first sense of subjective: it's probably false.

Frequentism as subjective (second sense)

Some historical detail is helpful to map the issues

For Neyman and Pearson there is only inductive behaviour not inductive inference and all statistical evaluation works with long run sampling properties of estimators. (Hence alpha and power analysis, but not p values). That's pretty unsubjective in both senses.

Indeed it's possible, and I think quite reasonable, to argue along these lines that Frequentism is actually not an inference framework at all but rather a collection of evaluation criteria for all possible inference procedures that emphasises their behaviour in repeated application. Simple examples would be consistency, unbiasedness, etc. This makes it obviously unsubjective in sense 2. However, it also risks being subjective in sense 1 when we have to decide what to do when those crteria do not apply (e.g. when there isn't an unbiased estimator to be had) or when they apply but contradict.

Fisher offered a less unsubjective Frequentism that is interesting. For Fisher, there is such a thing as inductive inference, in the sense that a subject, the scientist, makes inferences on the basis of a data analysis, done by the statistician. (Hence p-values but not alpha and power analysis). However, the decisions about how to behave, whether to carry on with research etc. are made by the scientist on the basis of her understanding of domain theory, not by the statistician applying the inference paradigm. Because of this Fisherian division of labour, both the subjectiveness (sense 2) and the individual subject (sense 1) sit on the science side, not the statistical side.

Legalistically speaking, the Fisher's Frequentism is subjective. It's just that the subject who is subjective is not the statistician.

There are various syntheses of these available, both the barely coherent mix of these two you find in applied statistics textbooks and more nuanced versions, e.g. the 'Error Statistics' pushed by Deborah Mayo. This latter is pretty unsubjective in sense 2, but highly subjective in sense 1, because the researcher has to use scientific judgement - Fisher style - to figure out what error probabilities matter and shoudl be tested.

Frequentism as subjective (first sense)

So is Frequentism less subjective in the first sense? It depends. Any inference procedure can be riddled with idiosyncracies as actually applied. So perhaps it's more useful to ask whether Frequentism encourages a less subjective (first sense) approach? I doubt it - I think the self conscious application of subjective (second sense) methods leads to less subjective (first sense) outcomes, but it can be argued either way.

Assume for a moment that subjectiveness (first sense) sneaks into an analysis via 'choices'. Bayesianism does seem to involve more 'choices'. In the simplest case the choices tally up as: one set of potentially idiosyncratic assumptions for the Frequentist (the Likelihood function or equivalent) and two sets for the Bayesian (the Likelihood and a prior over the unknowns).

However, Bayesians know they're being subjective (in the second sense) about all these choices so they are liable to be more self conscious about the implications which should lead to less subjectiveness (in the first sense).

In contrast, if one looks up a test in a big book of tests, then one could get the feeling that the result is less subjective (first sense), but arguably that's a result of substituting some other subject's understanding of the problem for one's own. It's not clear that one has gotten less subjective this way, but it might feel that way. I think most would agree that that's unhelpful.

Related Question