Solved – Minimum population size for chi-squared test

chi-squared-testsample-size

I'm analyzing data from an experiment in which two independent groups were exposed to an experimental setup without and with treatment.

I am testing whether the treatment changed the second group's behaviour by performing a chi-squared test that compares group 2 (the observed) vs group 1 (the expected). The result indicates there is a significant change in behaviour X² p-value < 0.00014.

Now, I am trying to test subgroups to understand better the change, i.e., looking at gender, age, and other self reported metrics.

My question is, given that group 2 N=40 if I look at age for instance I find people in their 20s and their 60s show significant change but other age groups don't. However people in their 20s N=12 and people in their 60s N=5. Is there a heuristic / rule that says there is a minimum number of people needed to consider a result significant? For instance anything below N=5 cannot be considered significant or anything below N=20% of the population?

EDIT: Just to clarify, I am doing a chi-squared test of independence (between group 1&2) not a chi-square goodness of fit test.

EDIT 2: With this edit I consider the question closed. None of the answers / comments gave me a definitive solution, which I believe says more about the question than the answers. I was hoping for a definitive answer along the lines you need at least 5 people or 20% of your sample. It seems the answer is less direct as it is sensitive to many factors.

Best Answer

For small sample sizes, use Fisher's exact test, because the $\chi^2$ test sampling statistics has only approximately the $\chi^2$ distribution, and this approximation is problematic for small sample sizes.

While lower sample size decreases the power of the test, the p-values (and not the sample size) are indicators of the statistical significance. A significant p-value stays significant whatever the sample size; the sample size has been taken care of through the calculation of the test statistic.

However, someone might claim that a small sample size is more likely to be biased. This is not necessarily true, but I think there might exist a correlation between the study sample size and whether the data was collected in an unbiased way as it should.

Related Solutions

Solved – What to do when I have expected count <5 warning for a chi squared test

A lot of the time, you may not need to do anything. The "5" rule is overly conservative, and there are a number of less restrictive (but somewhat more complex) guidelines to be found in the more recent literature (where 'more recent' means 'over the last half century or more').

For example, if all your cells have expected higher than 1 and about 80% are above 5, you're probably safe just treating it as chi-square (in that the p-values will still be roughly correct in instances you'll care to have good accuracy in). If expecteds are close to equal you can go lower.

If you are willing to condition on both margins and have access to something that can generate random tables with fixed margins (such as can be done in R), you can use simulation to estimate p-values without changing anything else. That's often the easiest to do and is built into chi-square testing in R, as an option.

There are a number of other options (some mentioned in other answers), but my usual preference is to simulate if the null distribution of the test statistic won't be adequately described by the chi-square.

Solved – How to determine sample size for Chi-squared test

Complete re-write:

I think the correct approach to calculating Cohen's w is to use the expected values for the P0 values. I looked back at Cohen (1988), and this isn't precisely clear, but I think that's the intention.

So the problem is that your second case (dat_0_better) doesn't represent the expected values for dat, but those for dat_0 does.

chisq.test(dat)$expected

   ###      [,1] [,2]
   ### [1,]   20   20
   ### [2,]   30   30

So the calculation of w in the first case, I believe, is correct † .

library(rcompanion)
cohenW(dat)

   ### Cohen w 
   ### 0.4082

The table that you've constructed with dat includes the information that the control treatment results in 10 out of 50. This is taken into account with the expected values of the table, so I don't think you need to alter the null hypothesis to account for this.

I think what I'm saying makes sense in the standard sample size calculation. It's the case that those before us did the hard work.

† Caveat: I am the author of the rcompanion package. I don't know of another package in R that calculates Cohen's w, though I would suspect there are some.

Best Answer

Related Solutions

Solved – What to do when I have expected count <5 warning for a chi squared test

Solved – How to determine sample size for Chi-squared test

Related Question