Solved – When does Fisher’s “go get more data” approach make sense

hypothesis testingintuitionp-valuephilosophical

Quoting gung's great answer

Allegedly, a researcher once approached Fisher with 'non-significant' results, asking him what he should do, and Fisher said, 'go get more data'.

From a Neyman-Pearson perspective, this is blatant $p$-hacking, but is there a use case where Fisher's go-get-more-data approach makes sense?

Best Answer

The frequentist paradigm is a conflation of Fisher's and Neyman-Pearson's views. Only in using one approach and another interpretation do problems arise.

It should seem strange to anyone that collecting more data is problematic, as more data is more evidence. Indeed, the problem lies not in collecting more data, but in using the $p$-value to decide to do so, when it is also the measure of interest. Collecting more data based on the $p$-value is only $p$-hacking if you compute a new $p$-value.

If you have insufficient evidence to make a satisfactory conclusion about the research question, then by all means, go get more data. However, concede that you are now past the NHST stage of your research, and focus instead on quantifying the effect of interest.


An interesting note is that Bayesians do not suffer from this dilemma. Consider the following as an example:

  • If a frequentist concludes no significant difference and then switches to a test of equivalence, surely the false positive rate has increased;
  • A Bayesian can express the highest density interval and region of practical equivalence of a difference simultaneously and sleep just the same at night.