I have submitted a paper to a journal reporting a non-significant finding. One of the reviewers has asked me to include a power analysis on my data to work out how big a sample size I would need to adequately power a study to test a raw mean difference of 1500 for significance. Basically he wants me to "prove" my study was not underpowered. Could someone offer me advice on how to do this?
I ran an unpaired t-test where:
- Group A: (n = 6) mean = 5424; SD = 923
- Group B: (n = 8) mean = 4734; SD = 702
Best Answer
Power analyses exploit an equation with four variables ($\alpha$, power, $N$, and the effect size). When you solve for power by stipulating the others, it is called "post hoc" power analysis. People often use post hoc power analysis to determine the power they had to detect the effect observed in their study after finding a non-significant result, and use the low power to justify why their result was non-significant and their theory might still be right. As @rvl points out in the comments, this involves "circular logic and [is] an empty exercise". However, that is not what you are doing here. Moreover, 'post hoc' power analysis can be a legitimate exercise: for example, I have had cases where a researcher knew they would only be able to get a certain number of patients with a rare disease and wanted to know the power they would be able to achieve to detect a given clinically significant effect. Although that isn't 'post hoc' in the sense of after the fact, it is called "post hoc" power analysis because it solves for power as a function of the other three.
I will go out on a limb and assume your $\alpha$ was $.05$. Clearly, $N = 9$. We can determine the effect size by calculating the pooled SD, and then the standardized mean difference that corresponds to a raw mean difference of $1500$ and the computed pooled SD.
\begin{align} SD_\text{pooled} &= \sqrt{\frac{(n_1-1)s^2_1 + (n_2-1)s^2_2}{(n_1+n_2)-2}} \\[10pt] 2145.041 &= \sqrt{\frac{(5-1)1930^2 + (4-1)2402^2}{(5+4)-2}} \\[30pt] ES &= \frac{\text{mean difference}}{SD_\text{pooled}} \\[10pt] 0.70 &= \frac{1500}{2145.041} \\ \end{align}
Having determined the effect size you want to use, you need some software to do the power analysis calculation for you. (It involves numerical approximations that you cannot do by hand.) A free and convenient application is G*Power.
The power to detect a standardized mean difference of $0.70$ with $\alpha = .05$ using a two-tailed $t$-test when $N = 9$ is $\approx 15\%$. I would say your study is underpowered.