Chi-square, effect size and unbalanced groups

chi-squared-testeffect-size

For context: I should note in advance I am a relative beginner with this.

Data context

I have data on some 600 000 persons which includes a column of whether these persons took parental leave or not (coded simply as 1 – took parental leave, 0, took no parental leave). I also have a column coding each person as male or female. I want to know whether persons coded as female are more likely to take parental leave than persons coded as male.

So I made a 2×2 table (female/male; no parental leave/parental leave) and applied the chi-square test which is significant (as expected). The residuals + prop table show that indeed women are overrepresented in taking 'parental leave'. So far so good.

Problem statement

However, the effect size is relatively small (Cramer's V about 0,15). For a number of reasons this seems counterintituive – the difference between men and women in the 'parental leave = 1' group seems quite large. I googled/read a bit about effect size & unbalanced groups. In this case there is a large dataset, with a relatively small proportion of the 600 000 persons taking parental leave. Could this affect the effect size, if yes, is there any measure other than Cramer's V that should be used in this regard?

Note: I am not specifically looking for a large effect size, just wondering whether I am applying the right measure.

Own research I have read the post: Chi-square Test with High Sample Size and Unbalanced Data but it didn't quite answer my question (the issue seems similar though).

Best Answer

Answer from comment thread:

It sounds like you are describing the most useful effect size for your situation:

relative to the proportion of men/women in the overall population, women are about twice as likely to take up parental leave.

If I understand what you are saying, this is the odds ratio.

For future readers, as an example of a 2 x 2 table with Cramer V = 0.15, and OR = 2, the following is code in R:

Matrix = matrix(c(550, 1100, 250, 1000), nrow=2, byrow=TRUE)

library(vcd)

assocstats(Matrix)

oddsratio(Matrix, log=FALSE)

   ###                     X^2 df   P(> X^2)
   ### Likelihood Ratio 64.712  1 8.8818e-16
   ### Pearson          63.294  1 1.7764e-15
   ### 
   ### Phi-Coefficient   : 0.148 
   ### Contingency Coeff.: 0.146 
   ### Cramer's V        : 0.148 
   ###
   ### odds ratio
   ###
   ### 2

OR = (550 / 1100) / (250 / 1000)

names(OR) = "Odds ratio"

OR

   ### Odds ratio 
   ###          2

Related Solutions

Solved – Chi-square with unbalanced design

Chi-square makes no assumptions about equality of group sizes.

The correction rates for the two groups can be compared (and indeed, different amounts of work per teacher within each group can be dealt with by the use of exposures, so if the A group marked twice as much work each as the B group that would also be fine).

Am I right to assume the groups are looking at the work of different students, rather than the same pool of students being marked twice?

I'd be inclined to use Poisson regression (where, for example, the model can be elaborated relatively easily, if required), but if you condition on the total number of corrections it would become a binomial test of a known proportion, which can also be done as a chi-square.

It would be good to explain what the underlying aim is more clearly, without using words like 'test', 'chisquare' or 'design' - you say 'juxtapose' - but that simply means to place unlike things together, which suggests you need a table. What do you want to find out about and why would hypothesis tests answer your underlying questions of interest?

---

Example of how to do the binomial / chi-square calculation:

Possible objection: Assumes the groups are internally homogeneous (i.e. there's no variability in the underlying rate of corrections within group - the observed variation is due to random variation around the shared level). (Other assumptions, like independence, are probably uncontroversial.)

Say the correction counts - on the same set of items, but different students - are as follows:

 A: 27 30 32 34 40 30 24 30 32 19 43 31 29 27 23    total: 451

 B: 32 50 43 37 39 39 38 47 31 38                   total: 394

If the rate of correction is the same for both groups, the total number of corrections should be proportional to the number of teachers.

That is, the sum of the A sample is expected to be a fraction 15/(10+15) (=60%) of the overall number of corrections. The total number of corrections across all teachers is 845.

The expected number of corrections in group A is 845 x 0.6 = 507, and in group B is 845 x 0.4 = 338.

The chisquare (for my made up data!) is

$$(451 - 507)^2/507 + (394 - 338)^2/338 = 15.46$$. The d.f. is 1.

As a binomial, we just test that the A proportion is 60%:

The observed total count in A is binomial(n=845,p=0.6); with a two-tailed test, we could use the normal approximation to the binomial proportion and get:

$Z = \frac{451/845 - 0.6}{\sqrt{0.6 (1-0.6)/845}} = -3.932$

(the square of this Z is the chi=square value above; its two-tailed p-value is the same as the p-value for the chi-square)

The exact binomial calculation is also quite readily done, but I won't labor the point.

---

A more complicated - but more defensible - analysis would be to fit a mixed logistic model, where 'teacher' is a random effect. This would allow for the fact that teachers have individual variation in their correction rate.

Solved – Effect size for goodness of fit chi-square

if for some reason you haven't yet found a solution, there is a good explanation here: http://www.real-statistics.com/chi-square-and-f-distributions/effect-size-chi-square/

Best Answer

Related Solutions

Solved – Chi-square with unbalanced design

Solved – Effect size for goodness of fit chi-square

Related Question