Solved – Confidence of Effect Size Differences

confidence intervaleffect-sizetreatment-effect

Hedges and Olkin (1985) suggest the following procedure for computing confidence intervals for Hedges' g resp. d
$$
\text{CI} = g \pm 1.96 \sqrt{(n_2+n_1)\big/(n_2n_1) + g^2 \times .5\big/(n_1+n_2)}
$$
(with 1.96 as the normal distribution cumulative density value for the confidence coefficient 95%)

When using pre-posttest designs with experimental and control groups, Klauer (1993) defined a corrected effect size d(corr) by subtracting the pre-posttest effect size of the experimental and the control group in order to measure intervention effects:
$$
\text{d(corr) = d(exp) – d(control)}
$$
I could not find literature on computing confidence intervals for effect size differences, though. Does anybody know how to compute confidence intervals of effect size differences? Thanks!

Best Answer

You can compute Cohen’s $d$ as
$$ d = (ME - MC)/SQRT [(SD2E - SD2C )2], $$ where ME and SDE are the posttest mean and SD within the experimental group and MC and SDC are the posttest mean and SD within the control group. This is for equal group sizes.

If the group sizes are unequal, you can use Glass’ $∆$: $$ ∆ = (ME - MC)/SDC. $$

A formula for calculating the confidence interval for an effect size is given by Hedges and Olkin (1985). If the effect size estimate from the sample is $d$, then it is Normally distributed, with standard deviation: $$ \sigma(d) = \sqrt{ \frac{N_E+N_C}{N_E\times N_C}+\frac{d^2}{2(N_E+N_C)} } $$

Hence a 95% confidence interval for $d$ would be from $$ (d – 1.96 σ[d],\; d + 1.96 σ[d]) $$

(Where NE and NC are the numbers in the experimental and control groups, respectively.)

You can find more information in the article Effect Sizes, Confidence Intervals, and Confidence Intervals for Effect Sizes by Bruce Thompson.

I think this article would provide the help you need:

Howell, DC. Confidence intervals on effect size (pdf)

Related Solutions

Solved – Conflict in confidence intervals for mean difference and confidence interval for Cohen’d effect size

There are analogous effect size measures to Cohen's for paired data, sometimes called the "standardized mean change" or "standardized mean gain". This is computed with $$d = \frac{\bar{x}_1 - \bar{x}_2}{SD_D} = \frac{\bar{x}_D}{SD_D},$$ where $\bar{x}_1$ is the mean at time 1 (or under condition 1), $\bar{x}_2$ is the mean at time 2 (or under condition 2), $\bar{x}_D$ is the mean of the change/differences scores, and $SD_D$ is the standard deviation of the change/differences scores.

This is the standardized mean change using "change score standardization". There is also the standardized mean change using "raw score standarization", but the former more directly relates to your use of the dependent samples t-test.

You can use the metafor package to compute this (and the corresponding CI):

summary(escalc(measure="SMCC", m1i=mean(a), sd1i=sd(a), m2i=mean(b), sd2i=sd(b), ni=length(a), ri=cor(a,b)))

yields:

      yi     vi    sei     zi  ci.lb  ci.ub
1 0.4961 0.0401 0.2003 2.4769 0.1035 0.8886

So, now the CI doesn't include 0 anymore, which is consistent with the results from the t-test. (Note: the value under yi is the d-value above, but after using a slight bias correction).

Some references if you want to read more about this:

Morris, S. B., & DeShon, R. P. (2002). Combining effect size estimates in meta-analysis with repeated measures and independent-groups designs. Psychological Methods, 7, 105–125.

Viechtbauer, W. (2007). Approximate confidence intervals for standardized effect sizes in the two-independent and two-dependent samples design. Journal of Educational and Behavioral Statistics, 32, 39-60.

Update: Getting the exact CI for d.

In rare cases, it can happen that the results of the t-test (and the CI for the mean difference) yields a different conclusion than the CI for d obtained above (i.e., the CI for the mean difference includes the value 0, while the CI for d does not, or vice-versa). This is due to the fact that the CI for d is based on an asymptotic approximation using the normal distribution.

One can compute an exact CI for the standardized mean change, but this requires iterative methods (see Viechtbauer, 2007, and the references given therein). The advantage of the exact CI is that it will always agree 100% with the results from the t-test and the CI for the mean difference in its conclusion.

Instead of letting the computer do the iterative work for us (which can be done in a few lines of code), one can also just do this manually by trial and error. For the data given in http://pastebin.com/12J7UghC, the bounds of the exact CI for d can be obtained with:

tval <- t.test(a, b, paired=TRUE)$statistic
pt(tval, df=length(a)-1, ncp=-0.00265265 * sqrt(length(a)), lower.tail=TRUE)
pt(tval, df=length(a)-1, ncp=-0.77193310 * sqrt(length(a)), lower.tail=FALSE)

Essentially, we just need to find those two values of the non-centrality parameter of the t-distribution, so that the observed t-value cuts off .025 in the lower and upper tails of the distribution. With a bit of trial and error (and starting with the CI bounds obtained earlier), we find the exact 95% CI for d is $(-0.003, -0.772)$. And now things are consistent again: The t-test rejects (just barely, with $p=.048$), the CI for the mean difference excludes 0 (just barely), and the CI for d exclude 0 (just barely).