Effect Size – How to Interpret Large Cohen’s d When p-value is Non-Significant and CIs are Close to 0

cohens-dconfidence intervaleffect-sizet-testwilcoxon-mann-whitney-test

I have a sample size of 23, with 11 and 12 participants in each group. I conducted t-tests for several continuous dependent variables. All of them are non-significant, but some of them have quite high Cohen's d values (for example 0.6 or above). In addition, the confidence intervals in these cases, while still spanning zero, usually have one case which is very very close to zero.

As an example, one of the tests has a Mann-Whitney U value of 41, a p-value of .11, CIs of -0.00004 and 1.2, and cohen's d of 0.75. How should I interpret this finding? Intuitively I would say it is a power problem, but with medium-large effect sizes I am not sure, since even small samples would theoretically be enough to detect such effects.

I used the jamovi package in R to conduct the tests

ttestIS(data = dat, vars = vars(variable_1, variable_2, variable_3, variable_4, variable_5), group = condition, students = FALSE, mann = TRUE, meanDiff = TRUE, desc = TRUE, effectSize = TRUE, ci = TRUE, plots = TRUE)

Update: I went off the effect size interpretations specified by Cohen (0.2 = small, 0.5 = medium, 0.8 = large). The dependent variables are self-report questionnaire items on a 5 point likert scale. The responses were non-normally distributed, hence the use of Mann-Whitney U tests.

Best Answer

Short answer: You don't.

Since your effect is not significant (you fail to reject the null hypothesis that there is no effect), if you're following the rules of null hypothesis significance testing, you cannot conclude that there is any effect here.

Additional points

  • You're running 5 tests, without adjusting for multiple comparisons, so the chances of making a type 1 error here are quite high. This makes it even more likely that your large effect size is large just by chance.
  • The wide confidence intervals show that even though the effect size might be large (and might even be larger than 0.75), it might also be small, zero, or negative, since your data does not rule out these possibilities.
Related Question