I am planning a meta-analysis. I read some books (e.g., Cumming, 2012) and papers (e.g., Johnson & Eagly, 2000). I learned that standardized measure of effect size like d and r are sensitive to resaerch design so that standardized effect size for within-subject design are usually larger than those for between-subject design in spite of the fact that the unstandardized effect sizes of both design are exactly the same (e.g., Baguley, 2009). For my meta-analysis, I have some studies with a between-subject design and other with a within-subject design. I know that I can correct standardized effect size for research design (e.g., Cumming, 2012, p. 291; Olejnik & Algina, 2000).
My question is : Do I need to do more than correcting for research design in order to combine my effect sizes? A similar question was already asked by Jemma (A meta-regression with effect sizes from variously designed studies) and mdewey suggested to convert the effect size into the same scale. I am ok with that and correcting for research design is a part of the job. But is it necessary to take into account in the computation of the combined effect size that some studies followed a within-subject design or a between-subject design?
Edit: I maybe found something consistent with mdewey's answer to Jemma in Borenstein et al. (2009, p. 30). They stated that:
a single systematic review can include studies that used independent groups and also studies that used matched groups. From a statistical perspective the effect size (d or g) has the same meaning regardless of the study design. Therefore, we can compute the effect size and variance from each study using the appropriate formula, and then include all studies in the same analysis. While there are no technical barriers to using studies with different designs in the same analysis, there may be a concern that these studies could differ in substantive ways as well (see Chapter 40)
Does someone else have some relevant references allowing to corroborate this statement?
Thanks a lot in advance
Best Answer
Morris and DeShon have a paper that answers this question:
Morris, S. B., & DeShon, R. P. (2002). Combining effect size estimates in meta-analysis with repeated measures and independent-groups designs. Psychological methods, 7(1), 105-125.
They recommend that: 1) You convert the effect sizes into a common metric, and 2) You assess design as a moderator.
If design is a significant moderator then you need to either meta-analyse the within-subjects and between-subjects design studies separately, or use multiple-meta regression where all moderators are assessed simultaneously (i.e., so that when you assess each moderator, the rest are controlled for).