Solved – How to estimate Cohen’s D using the independent groups definition but on a paired samples data

cohens-dconfidence intervalr

I want to calculate cohen's d with confidence intervals for a paired samples designs.

Some authors suggest that you use the paired t test value to adjust for the correlation between measures (Rosenthal, 1991).

However,
Dunlop et al 1996 suggest that the correlation between the paired samples should not be included. In particular, if such a correlation is used, then results are not readily comparable to between subjects effects.
Instead, they recommend using independent samples formula:

$$d= \frac{\mu_1-\mu_2}{\sigma_\textrm{pooled}}$$

Question

How do you estimate confidence intervals for Cohen's d in a paired samples design where Cohen's d uses the formula above?

The way I have estimated it so far is to use the R function ci.smd from the MBESS package. This function is designed for independent samples tests and takes the sizes of two independent groups as input.

In the code below I use i to represent the number of subjects (equal, as the same participants are in each group) and use an effect size of 0.8 as an example.

library(MBESS)
x.ci <- ci.smd(smd=0.8, n.1=i,n.2=i)

Is it appropriate to use such function ci.smd for confidence intervals for paired samples?

Best Answer

This is just about Dunlop et al because I've got no time to go through the code.

They have a point when the design of the study could be between or within. Nevertheless, I think both effect sizes could easily be reported. Minimally, if you follow their advice, you should also report your correlation between your groups so that the repeated measures effect size can be calculated.

But their argument often gets extended to cases where you can only do the experiment within subject. I remember some property of differences between people's ears that gets studied where the literature strongly endorsed Dunlop. In that case, there's no way to do the study as an independent groups design and therefore no concern about comparing the effect sizes observed across repeated measures and independent groups.

It depends on the kind of study you're doing whether you follow Dunlop et al. Are there both independent and repeated measures designs possible? Is there a related literature you want to consider that can have independent measures? If so, include an independent groups effect size.

And also make sure you publish how you pool your effect variance. There are a variety of ways it's done and it's important to specify.

Related Solutions

Effect Size – Calculating Independent Samples Cohen’s d from T & DF

The SD used in the t calculation is that of the effect, not of the individual groups. In the paired case the SD to use in Cohen's d and t are the same. But the SD used in the d calculation for independent samples is the pooled SD of the individual groups, not the theoretical SD of the effect. Under the assumptions of 0 correlation and equal variance (the independent case) the variance of the effect is double the variance of the individual conditions.

Try the following R code to demonstrate the effect

x <- rnorm(1000, 0, 10)
var(x)
y <- rnorm(1000, 5, 10)
var(y)
cor(x,y)
var(x-y)

Run the example a few times. The first two lines get random independent samples of 1000 with variances of 100 (sd = 10). What you'll see is that with the correlation between x and y close to 0 the variance of x-y (the effect) tends toward 200. This is also true for x+y. With the large samples in the code above spurious correlations are rare but in real experiments with large sampled they happen all the time (reduce the n in the sample above and they'll happen there). Therefore, what we do is stick to the theory and average the variance across the groups (pooled variance) and then double it. One could alternatively just sum var(x) + var(y). This turns out to be mathematically the same but hides the assumption of equal variance.

For comparison, try some correlated data made from effect calculations.

m <- rnorm(1000, 2.5, 10)
x <- m - rnorm(1000, 2.5, 10)
y <- m + rnorm(1000, 2.5, 10)
var(x)
var(y)
cor(x,y)
var(x-y)

I didn't bother equating the variances to the above (an sd = sqrt(50) in all samples above would do it). What you'll see this time is that the variance of x and y are each 200. This should result in a final variance of x-y of 400, if all was true as before. However, because x and y are correlated (about 0.5) you'll get a much lower variance. This is the mathematical property that a paired t-test takes advantage of.

That's part of it, but why does cohen's d use different calculations? With the paired measures design, the correlation between conditions is part of the calculation of the effect. You typically only collect enough of an N to really measure the effect and you're not really about measuring the true values in each condition, just the effect. Future experiments will tend to be of a similar design and the effect size from repeated measures provides a more accurate prognosticator of the likelihood of replication. A similar argument holds for the independent design.

Some have argued that you should use a pooled variance of the individual conditions all of the time. There's debate about that with some people coming down firmly that the d formula should always be the same and use the independent groups version. This insures a common reference point for when you do and don't use repeated measures designs. I think the argument has some merit if an independent design is possible, and probable. But I see this argument made in things like differences between people's ears. That can never be an independent groups design, will have very highly correlated measurements, and therefore should always have an effect size calculated through the paired or correlated effect measurement.

Solved – Bootstrapped confidence interval for the difference in means for paired data

The first method is no resampling test of which I'm aware in the literature. It seems like your goal, by resampling $X$ and $Y$ independently, is to generate data under the null hypothesis. This approach is inefficient because you are ignoring pairing in the design.

The preferred resampling method for generating data under the null hypothesis is the permutation test. Permutation testing for paired data is done by randomly negating the $X-Y$ differences; i.e. replacing them with $Y-X$. Here, the between-pair differences are preserved, but the within-pair differences are only preserved if the paired mean difference is 0.

The second example is a proper description of a paired bootstrap.

Question

Best Answer

Related Solutions

Effect Size – Calculating Independent Samples Cohen’s d from T & DF

Solved – Bootstrapped confidence interval for the difference in means for paired data

Related Question