Hypothesis Testing – How to Statistically Compare Changes in Two Percentages

hypothesis testingproportion;statistical significance

I plan to conduct an experiment with one between-subjects factor with two levels (control vs. treatment). Respondents will be asked to choose among three options (A vs. B vs. C). (FYI, I plan to collect N = 200 per factor).

Below are predicted/made-up results.

enter image description here

For the choice of option A, there is a 35% percentage-point decrease from control to treatment (45% to 10%). For the choice of option B, there is a 15% percentage-point decrease from control to treatment (45% to 30%).

Question: I would like to test whether 35% and 15% are statistically different. What test should I use?

Note: I am not comparing the choice probability of a specific option between control and treatment group (for this, I think I need to use a multinomial logistic regression). I want to compare the two percentage changes from control to treatment between choice for A and choice for B.

Best Answer

If you want to test whether the distribution among the threee options is signifcantly different between Control and Treatment, you can use a $\chi^2$-test. Here is an example in R:

> data <- data.frame(Control = 200 * c(0.45, 0.45, 0.1),
+                    Treatment = 200 * c(0.1, 0.3, 0.6))
> chisq.test(data, simulate.p.value=TRUE)
X-squared = 121.97, df = NA, p-value = 0.0004998

If you only want to test, whether the change of the proportion in only one option is significant, you can use a proportion test or a t-test. Here are both versions in R for Choice A your example:

> N <- 200; p <- 0.45; q <- 0.1
> prop.test(x=N*c(p,q), n=c(N,N), correct=F)
X-squared = 61.442, df = 1, p-value = 4.56e-15
95 percent confidence interval:
0.2694863 0.4305137

> # t-test
> sx2 <- p*(1-p)*N/(N-1)
> sy2 <- q*(1-q)*N/(N-1)
> T <- abs(q - p) / sqrt(sx2/N + sy2/N)
> k <- (N-1) * (sx2/N + sy2/N)^2 / ((sx2/N)^2 + (sy2/N)^2)
> cat(sprintf("p-value from t-test: %f\n", 2*(1-pt(T,k))))
p-value from t-test: 0.000000

Note that the R function prop.test also gives a confidence interval for the difference, which in most cases is of greater interest than a test for statistical significance.

Related Question