Solved – How to calculate 95% confidence intervals for partial eta squared and eta squared

confidence intervaleffect-size

Following on from this question on reporting eta squared, I would like to report 95% CIs for my effect size point estimates (partial eta and eta squared values).

What statistics do I need to know and is there a straight forward formula for calculation?

Best Answer

As a preliminary answer:

This article seems to explain how to caculate confidence intervals for eta squared http://core.ecu.edu/psyc/wuenschk/SPSS/CI-R2-SPSS.doc using SPSS.

Effect sizes for group mean differences

In general, I find standardised group mean differences (e.g., Cohen's d) a more meaningful effect size measure within the context of group differences. Measures like eta square are influenced by whether group samples sizes are equal, whereas Cohen's d is not. I also think that the meaning of d-based measures are more intuitive when what you are trying to quantify is a difference between group means.
The above point is particularly strong for the case where you only have two groups (e.g., the effect of treatment versus control). If you have more than two groups, then the situation is a little more complicated. I can see the argument for variance explained measures in this case. Alternatively, Cohen's $f^2$ is another option.
A third option is that within the context of experimental effects, even when there are more than two groups, the concept of effect is best conceptualised as a binary comparison (i.e., the effect of one condition relative to another). In this case, you can once again return to d-based measures. The d-based measure is not an effect size measure for the factor, but rather of one group relative to a reference group. The key is to define a meaningful reference group.
Finally, it is important to remember the broader aim of including effect size measures. It is to give the reader a sense of the size of the effect of interest. Any standardised measure of effect should assist the reader in this task. If the dependent variable is on an inherently meaningful scale, then don't shy away from interpreting the size of effect in terms of that scale. E.g., scales like reaction time, salary, height, weight, etc. are inherently meaningful. If you find, as I do, eta squared to be a bit unintuitive within the context of experimental effects, then perhaps choose another index.

Eta squared versus partial eta squared

Partial eta squared is the default effect size measure reported in several ANOVA procedures in SPSS. I assume this is why I frequently get questions about it.
If you only have one predictor variable, then partial eta squared is equivalent to eta squared.
This article explains the difference between eta squared and partial eta squared (Levine and Hullett Eta Squared, Partial Eta Squared..).
In summary, if you have more than one predictor, partial eta squared is the variance explained by a given variable of the variance remaining after excluding variance explained by other predictors.

Rules of thumb for eta squared and partial eta squared

If you only have one predictor then, eta squared and partial eta squared are the same and thus the same rules of thumb would apply.
If you have more than one predictor, then I think that the general rules of thumb for eta squared would apply more to partial eta squared than to eta squared. This is because partial eta squared in factorial ANOVA arguably more closely approximates what eta squared would have been for the factor had it been a one-way ANOVA; and it is presumably a one-way ANOVA which gave rise to Cohen's rules of thumb. In general, including other factors in an experimental design should typically reduce eta squared, but not necessarily partial eta squared due to the fact that the second factor, if it has an effect, increases variability in the dependent variable.
Despite what I say about rules of thumb for eta squared and partial eta squared, I reiterate that I'm not a fan of variance explained measures of effect size within the context of interpreting the size and meaning of experimental effects. Equally, rules of thumb are just that, rough, context dependent, and not to be taken too seriously.

Reporting effect size in the context of significant and non-significant results

In some sense an aim of your research is to estimate various quantitative estimates of the effects of your variables of interest in the population.
Effect sizes are one quantification of a point estimate of this effect. The bigger your sample size is, the more close, in general, your sample point estimate will be to the true population effect.
In broad terms, significance testing aims to rule out chance as an explanation of your results. Thus, the p-value tells you the probability of observing an effect size as or more extreme assuming the null hypothesis was true.
Ultimately, you want to rule out no effect and want to say something about the size of the true population effect. Confidence intervals and credibility intervals around effect sizes are two approaches that get at this issue more directly. However, reporting p-values and point estimates of effect size is quite common and much better than reporting only p-values or only effect size measures.
With regards to your specific question, if you have non-significant results, it is your decision as to whether you report effect size measures. I think if you have a table with many results then having an effect size column that is used regardless of significance makes sense. Even in non-significant contexts effect sizes with confidence intervals can be informative in indicating whether the non-significant findings could be due to inadequate sample size.

Solved – Whether to report confidence intervals of effect sizes such as $r$ and $\eta^2$

The answer is almost always: report both. This way, your audience can decide on the interestingness and importance of your results, instead of just having to believe you. Confidence intervals are similarly always useful, because they give a neat indication of both effect size, and significance, in one. Even better if it's on a graph :)

It may be best to look at all the possible outcomes, remembering that a significance level ($\alpha$) is an arbitrary threshold, and you can choose anyone you like (higher ones are of course harder to defend).

Low p-value, low effect size: You've got a result, you're pretty sure it's not down to chance, but it doesn't really say anything interesting. An example might be a new drug, that significantly improves on an old one. But if the improvement is a only 2%, then your result might not means so much when weighed up against other factors (like extra costs or new side effects)
High p-value, high effect size: Looks like you're on to some thing, but you can't say for certain that it wasn't just the result of chance. For this drug, things look promising, but you DEFINITELY want to do some more testing, probably with a modified experiment that (hopefully) will reduce some of that ridiculous variability you're seeing.
High p-value, low effect size: Nothing interesting here. Go and design a better experiment.
Low p-value, large effect size: Win. You've got a big effect, and you're sure it's not down to chance. If it's a new drug, then you've got a good chance that your results will make a big impact, and it'll get pushed out to market quickly, even if it costs more, or if there are some side effects.

(Note: not pushing drugs here, I'm as mistrustful as the next paranoiac about big pharma, but drugs make for good statistical examples :)

Best Answer

Related Solutions

Solved – How to interpret and report eta squared / partial eta squared in statistically significant and non-significant analyses

Effect sizes for group mean differences

Eta squared versus partial eta squared

Rules of thumb for eta squared and partial eta squared

Reporting effect size in the context of significant and non-significant results

Solved – Whether to report confidence intervals of effect sizes such as $r$ and $\eta^2$

Related Question