Solved – Statistical significance in an underpowered study, false positive

statistical significancestatistical-power

So, I'm actually a biologist trying to wrap my head around the idea of power of analysis to help design an experiment with the proper sample size. I understand that power of analysis is used to help avoid type II errors, but I came across this paper:

http://www.benthamscience.com/open/toepij/articles/V003/16TOEPIJ.pdf

which seems to say (if I'm not mistaken) that in underpowered studies, you also increase your risk for false positives. The other thing that confuses me, is that if you keep adding more samples to increase your power, of course you make any finding statistically significant, even if your effect is small. Is there some sort of balance between power and number of samples so that one avoids the pitfalls of an underpowered experiment, and also an overpowered experiment where the results will produce a statically significant result that might not be interesting because the effect size is so small?

Best Answer

From a quick skim it seems like they're basically taking a Bayesian viewpoint and computing a particular probability (H0 true|reject if I understood what they were getting at) that they argue must go up as the sample size goes down; if that's what the claim was, then as far as that goes, it's valid, because the denominator in Bayes rule must decrease as the sample size goes down while the significance level and P(H0 true) are presumably fixed.

A frequentist would argue that their rate of reject|H0 true is fixed, and they'd likely say that's what they care about.

On the gripping hand, in the overwhelming majority of studies, the true overall rate of false&positives must be effectively zero at every sample size, since in most circumstances nulls are simply not exactly true.

I guess it comes down to what probability you want.

Related Question