I plan to conduct an experiment with one between-subjects factor with two levels (control vs. treatment). Respondents will be asked to choose among three options (A vs. B vs. C). (FYI, I plan to collect N = 200 per factor).
Below are predicted/made-up results.
For the choice of option A, there is a 35% percentage-point decrease from control to treatment (45% to 10%). For the choice of option B, there is a 15% percentage-point decrease from control to treatment (45% to 30%).
Question: I would like to test whether 35% and 15% are statistically different. What test should I use?
Note: I am not comparing the choice probability of a specific option between control and treatment group (for this, I think I need to use a multinomial logistic regression). I want to compare the two percentage changes from control to treatment between choice for A and choice for B.
Best Answer
If you want to test whether the distribution among the threee options is signifcantly different between Control and Treatment, you can use a $\chi^2$-test. Here is an example in R:
If you only want to test, whether the change of the proportion in only one option is significant, you can use a proportion test or a t-test. Here are both versions in R for Choice A your example:
Note that the R function
prop.test
also gives a confidence interval for the difference, which in most cases is of greater interest than a test for statistical significance.