Solved – Chi Square Sample Size Needed with G*Power

chi-squared-testgpowersample-sizestatistical-power

I was recommended to try G*Power so I could determine the sample size needed in order to achieve an acceptable power. I'm struggling with picking the test and filling out the appropriate values.

Basically, I'm sending a survey to approximately 3,000 people and am testing different times/days for response rate. So, for example, I'd like to send out on Monday, Wednesday, Friday, and Sunday, in the early morning, afternoon, evening, and middle of the night. So a total of 16 groups (4 x 4 conditions). This would give me 187.5 people in each group. Can anyone explain how I use G*Power or other software to figure this out?

When I select $X^2$ (chi-square) and goodness-of-fit and then "A priori" to calculate the required sample size, it asks me to determine p(H1) for each group, and I really don't know what to expect groups to achieve in terms of response rate. Furthermore, is a chi-square the right test, or should I be using an ANOVA (I am under the impression that ANOVA is for continuous variables, and I'd have a proportion)?

Best Answer

I'm going to steer you away from power calculations so that you can reconsider your procedure. Your situation involves 2 independent variables--day and time--, so you need to replace Chi-square with a 2-factor, multivariate model. Which type you use depends on how you operationalize your dependent variable (DV).

Rather than response rate, which applies to groups, your DV, if measured at the individual level, would be response/nonresponse. And to test for days and times that relate to this it would be natural to use logistic regression. As part of this you could build in a term to test for a day*time interaction, i.e., for specific combinations of days and times that have especially high or low response.

If you prefer or need to measure response at the group level via response rate, then I'm guessing loglinear modeling would be your choice.

Related Solutions

Chi-Squared Test – How to Calculate Chi-Square and P-Value: A Comprehensive Guide

No, your use of a chisquare test is inappropriate. One problem is that the same 5000 people appear more than once in your table, so the counts are not independent. Another problem is the use of arbitrarily chosen expected values.

One simple and correct way to test your hypotheses is to compare A with each of the other models one at a time. Comparing A to B, your table shows that 2208 people prefer A and 1570 prefer B. An exact one-sided p-value can be obtained from the binomial probability in R:

pbinom(2208-0.5, prob=0.5, size=2208+1570, lower.tail=FALSE)

which is 1.4e-25. This tests the null hypothesis that people are equally likely to prefer A or B vs the alternative that people are more likely to prefer A. The binomial test conditions on the total number of people who had a preference.

Comparing A to C gives:

pbinom(2970-0.5, prob=0.5, size=2970+1492, lower.tail=FALSE)

which is 1.1e-110. Comparing A to D gives:

pbinom(1890-0.5, prob=0.5, size=1890+1656, lower.tail=FALSE)

which is 4.5e-05.

So, yes, there is strong evidence that the respondents tend to prefer model A over each of the other three alternatives.

Solved – How to determine sample size for Chi-squared test

Complete re-write:

I think the correct approach to calculating Cohen's w is to use the expected values for the P0 values. I looked back at Cohen (1988), and this isn't precisely clear, but I think that's the intention.

So the problem is that your second case (dat_0_better) doesn't represent the expected values for dat, but those for dat_0 does.

chisq.test(dat)$expected

   ###      [,1] [,2]
   ### [1,]   20   20
   ### [2,]   30   30

So the calculation of w in the first case, I believe, is correct † .

library(rcompanion)
cohenW(dat)

   ### Cohen w 
   ### 0.4082

The table that you've constructed with dat includes the information that the control treatment results in 10 out of 50. This is taken into account with the expected values of the table, so I don't think you need to alter the null hypothesis to account for this.

I think what I'm saying makes sense in the standard sample size calculation. It's the case that those before us did the hard work.

† Caveat: I am the author of the rcompanion package. I don't know of another package in R that calculates Cohen's w, though I would suspect there are some.

Best Answer

Related Solutions

Chi-Squared Test – How to Calculate Chi-Square and P-Value: A Comprehensive Guide

Solved – How to determine sample size for Chi-squared test

Related Question