I have two related questions, both of which are related to a meta-analysis I am performing where where the primary outcomes are expressed in terms of the standardized mean difference.
My studies have multiple variables that are available with which to calculate the standardized mean difference. I would like to know the extent to which the standardized mean differences calculated on one variable are consistent with the standardized mean differences on the other. To my mind, this question could be expressed as as a meta-analysis on the difference between two sets of standardized mean differences. However, I am having trouble determining the effect size and sampling error for the difference between two standardized mean differences within the same study.
To express my problem in a different way, consider a two-condition study with groups $g_1$ and $g_2$ and outcome variables $var_1$ and $var_2$. These two outcome variables are correlated as $cor(var_1, var_2)$. We can calculate standardized mean differences for $var_1$ and $var_2$ across $g_1$ and $g_2$, yielding $d_{var1}$, $d_{var_2}$, and their sampling variances $v_{d_{var_1}}$ and $v_{d_{var_2}}$. I have included a very simple schematic of the situation below.
Now let's say that we calculate a difference between $var_1$ and $var_2$ as $diff$. I can calculate the standardized mean difference across $g_1$ and $g_2$ as $d_{diff}$, which has sampling variance $v_{d_{diff}}$.
What I would like to do is express $d_{diff}$ and $v_{d_{diff}}$ in terms of the following variables:
- Effect sizes $d_{var_1}$ and $d_{var_2}$,
- Sampling variances $v_{d_{var_1}}$ and $v_{d_{var_2}}$, and
- Correlation $cor(var_1, var_2)$
I feel like this goal should be possible given the fact that, in a simple (non meta-analytic) context, the standard deviation of the difference between $var_1$ and $var_2$ is given as
$sd(var_1)^2 + sd(var_2)^2 – 2 * cor(var_1, var_2) * sd(var_1) * sd(var_2)$
I am also interested in a slightly more complicated situation where one has studies with 3 (or more) groups, and where one therefore calculates two sets of standardized mean differences between the two candidate variables.
To express this second question in a different way, assume that a given study has three groups $g_1$, $g_2$, and $g_3$ and two outcome variables $var_1$ and $var_2$. Furthermore, assume once again that $var_1$ and $var_2$ are correlated as $cor(var_1, var_2)$.
Choose group $g_1$ as the reference group and, for $var_1$, calculate effect sizes for group $g_1$ vs $g_2$ and $g_1$ vs $g_3$. This will yield two sets of effect sizes for each of $var_1$ and $var_2$ — for $var_1$, $d_{var1_{g_1 – g_2}}$ and $d_{var1_{g_1 – g_3}}$, and, for $var_2$, $d_{var2_{g_1 – g_2}}$ and $d_{var2_{g_1 – g_3}}$. This will also yield two sampling variances for each set of effect sizes (for $var_1$, $v_{d_{var1_{g_1 – g_2}}}$ and $v_{d_{var1_{g_1 – g_3}}}$, and, for $var_2$, $v_{d_{var2_{g_1 – g_2}}}$ and $v_{d_{var2_{g_1 – g_3}}}$) and one sampling covariance for each variable (for $var_1$, $cov(d_{var1_{g_1 – g_2}}, d_{var1_{g_1 – g_3}})$, and, for $var_2$, $cov(d_{var2_{g_1 – g_2}}, d_{var2_{g_1 – g_3}})$). I have included a very simple schematic of the situation below.
Once again, I can create a difference score between $var_1$ and $var_2$, yielding $diff$. I can then calculate two sets of effect sizes on this difference score as above, calculating a standardized mean difference for the comparison between $g_1$ and $g_2$ (yielding $d_{diff_{g_1 – g_2}}$) and a standardized mean difference for the comparison between $g_1$ and $g_3$ (yielding $d_{diff_{g_1 – g_3}})$. This procedure will, of course, also yield corresponding sampling variances and covariances.
What I would like is to express the effect sizes, sampling variances, and sampling covariances for $diff$ in terms of:
- Effect sizes $d_{var1_{g_1 – g_2}}$, $d_{var1_{g_1 – g_3}}$, $d_{var2_{g_1 – g_2}}$, and $d_{var2_{g_1 – g_3}}$
- Sampling variances $v_{d_{var1_{g_1 – g_2}}}$, $v_{d_{var1_{g_1 – g_3}}}$, $v_{d_{var2_{g_1 – g_2}}}$, and $v_{d_{var2_{g_1 – g_3}}}$,
- Sampling covariances $cov(d_{var1_{g_1 – g_2}}, d_{var1_{g_1 – g_3}})$ and $cov(d_{var2_{g_1 – g_2}}, d_{var2_{g_1 – g_3}})$, and
- Correlation $cor(var_1, var_2)$
Once again, I feel like my goal should be feasible given the fact that it is possible to calculate the standard deviation of a difference score between $var_1$ and $var_2$ given $sd(var_1)$, $sd(var_2)$, and $cor(var_1, var_2)$.
I realize that my questions are a little complicated, but I feel like they could be answered given a bit of clever algebra. Let me know if I can clarify my question and / or notation in any way.
Best Answer
I can certainly give you an answer to the first part of your question.
Using your notation, let $d_{var_1}$ and $d_{var_2}$ denote the two d values (computed for the same two groups) based on two different dependent variables and let $v_{var_1}$ and $v_{var_2}$ denote the corresponding sampling variances, which are typically computed/estimated with: $$v_{var_1} = \frac{1}{n_1} + \frac{1}{n_2} + \frac{d_{var_1}^2}{2(n_1 + n_2)}$$ and $$v_{var_2} = \frac{1}{n_1} + \frac{1}{n_2} + \frac{d_{var_2}^2}{2(n_1 + n_2)},$$ where $n_1$ and $n_2$ are the two group sizes.
Let $r = cor(var_1, var_2)$ denote the correlation between the two variables. Then the covariance between the two d values can be estimated with: $$cov(d_{var_1}, d_{var_2}) = \left(\frac{1}{n_1} + \frac{1}{n_2}\right)r + \left(\frac{d_{var_1}d_{var_2}}{2(n_1 + n_2)} \right)r^2.$$ See equation (19.27) in the chapter on stochastically dependent effect sizes by Gleser and Olkin (2009) in The handbook of research synthesis and meta-analysis (2nd ed.).
Therefore, the sampling variance of $$d_{diff} = d_{var_1} - d_{var_2}$$ can be computed/estimated with: $$v_{d_{diff}} = v_{var_1} + v_{var_2} - 2 cov(d_{var_1}, d_{var_2}).$$
The chapter by Gleser and Olkin also partly addresses your second question. Essentially, you have what the authors call a 'multiple-treatment study' and they provide equations for the covariance in that case as well (see Expectation of correlated variables). However, your case is actually a combination of the 'multiple-treatment' and 'multiple-endpoint' case. Deriving the necessary covariance equations would require some additional work.