I am trying to conduct power calculations for a "Before-After-Control-Impact" type of study focused on public opinion variables.
Ideally, I want to look at the relationship between effect size and power (in this case our sample size is a given). At a minimum, I want to be able to calculate the minimum detectable effect size for the conventional power = 0.80, alpha = 0.05 thresholds.
I am a bit flummoxed at the best approach to calculate power for a difference in difference of proportions. For example, in the case of "do you agree with Y?" I ultimately want to look at $(p_{ Treatment_{After}} – p_{Treatment_{Before}}) – (p_{Control_{After}} – p_{Control_{Before}})$. The most analogous thing I can think of would be an odds ratios approach, but I'm not confident this makes sense.
I realize this is a simple question, but most of what I can find is about comparing two proportions, rather than comparing the difference between two proportions.
Best Answer
To calculate power for the difference in proportions between two populations, you need to find the standard error of this difference: $\sqrt{p_1 (1 - p_1) / n_1 + p_2 (1 - p_2) / n_2}$
To make this simple, let's assume equal sample size and variance. A little algebra gives us the following conservative bound for the standard error: $ se = 1 / \sqrt(n)$.
Let's assume an effect size of 0.1. Now for power of 80 percent and a significance level of 0.05, you need an effect that's 2.8 standard errors from zero. This gives us the following equation: $2.8 \times 1/ \sqrt{n} = 0.1$ A little algebra gives us: $n = (2.8 / 0.1)^2$ So utilizing this conservative bound on the variance, to detect an effect size 80 percent of the time at the significance level of 0.05, you'd need a sample size of 784 (392 for each population).
Now let's extend this to the case of a difference in the difference between two populations. Again, assuming equal sample sizes and variance, we have:
$$\sqrt{p(1 - p) / (n/4) + p(1 - p) / (n/4) +p(1 - p) / (n/4) +p(1 - p) / (n/4)}$$
This simplifies to $2 / \sqrt{n}$.
So if we replace the standard error for the difference between two proportions with the standard error of the difference in the difference between two proportions, we have: $n = (2.8 \times 2 ) / 0.1)^2$ again assuming an effect size of 0.1. This gives us a required sample of 3136 meaning that for the first survey, you'd need a sample of 1568 (784 for each population) and for the second survey, you'd also need a sample of 1568 (784 for each population).
For a very nice explanation of sample size, see chapter 16 of Regression and Other Stories by Gelmen, Hill, and Vehtari (available on line for free). For the case of difference-in-difference, look at the section on interactions.