Solved – Analyzing Difference in Change between two groups with unpaired samples

hypothesis testingt-test

I'm currently working with a data set where there is a control group and an intervention group. There are a couple of data points where we're interested in seeing if the intervention group has a statistically significant difference in effect (basically is $x_2 – x_1$ in intervention different than $x_2-x_1$ in control).

Unfortunately, the samples collected within the groups are not paired – in other words, the same people were not interviewed at baseline and endline. I can use an unpaired t-test to compare endline-to-endline, or baseline-to-endline for the groups individually. Were the samples paired, I could just take the difference and use an unpaired t-test to compare the differences. Since they're unpaired, I'm at a bit of a loss.

Is there a way to adjust the t-test for this?

If not, are there appropriate statistical tests that can be done for this situation without artificially pairing up the samples?

(As is too commonly the case, I was not involved in the design of this experiment, but I hope I've conveyed to the group the importance of paired sampling for the future.)

Best Answer

Let's use $C$ for 'control' and $T$ for treatment (i.e. 'intervention')

It seems like you have the following situation.

There are four quantities you have measurements on:

1) Control, baseline - from which you can estimate $\mu_{C1}$

2) Control, endline - from which you can estimate $\mu_{C2}$

3) Intervention, baseline - from which you can estimate $\mu_{T1}$

4) Intervention, endline - from which you can estimate $\mu_{T2}$

Let $\delta_i=\mu_{i2}-\mu_{i1}$ be the after-before difference.

You're interested in testing against the alternative $\delta_T\neq\delta_C$ (or maybe $\delta_T>\delta_C$ in a one-tailed test).

This is straightforward; it may be done in a number of ways.

  1. Since you say you'd normally use a t-test, you can do that here, but you'll need some assumptions; the suitability of those assumptions will be a question you'll need to consider carefully.

    The numerator of your t-statistic will be

    $\hat{\delta}_T-\hat{\delta}_C \,= \hat{\mu}_{T2}-\hat{\mu}_{T1}-(\hat{\mu}_{C2}-\hat{\mu}_{C1}) $

    $\quad\quad\quad\quad= \bar{x}_{T2}-\bar{x}_{T1}-(\bar{x}_{C2}-\bar{x}_{C1}) $

    The questions come down to whether you will assume independence of all four sets of measurements, and whether you will assume equality of variances (in the second case, if not, some Welch-type adjustment will be required).

    a. If you assume independence and equal variances for all four measurements, your denominator would work similarly to the way it does in an ordinary two-sample t-test, so you'd get:

    $$t = \frac{\bar{x}_{T2}-\bar{x}_{T1}-(\bar{x}_{C2}-\bar{x}_{C1})}{ s_{p} \cdot \sqrt{\frac{1}{n_{T1}}+\frac{1}{n_{T2}}+\frac{1}{n_{C1}}+\frac{1}{n_{C2}}}}$$

    where

    $$s_p^2= \frac{(n_{T1}-1)s_{{T1}}^2+(n_{T2}-1)s_{{T2}}^2+(n_{C1}-1)s_{{C1}}^2+(n_{C2}-1)s_{{C2}}^2}{n_{T1}+n_{T2}+n_{C1}+n_{C2}-4}$$

    and you have $n_{T1}+n_{T2}+n_{C1}+n_{C2}-4$ d.f. for the $t$ distribution.

    b. if they don't all necessarily have the same variance, the Welch-Satterthwaite approximation to the d.f. for the natural test statistic can be used:

    $$t = {\overline{x}_{T2} - \overline{x}_{T1}-(\overline{x}_{C2} - \overline{x}_{C1})} \over s_{d}$$

    $$s_{d} = \sqrt{{s_{T1}^2 \over n_{T1}} + {s_{T2}^2 \over n_{T2}}+{s_{C1}^2 \over n_{C1}} + {s_{C2}^2 \over n_{C2}}}$$

    with d.f:

    $$ \frac{(\frac{s_{T1}^2}{n_{T1}} + \frac{s_{T2}^2}{n_{T2}}+\frac{s_{C1}^2}{n_{C1}} + \frac{s_{C2}^2}{n_{C2}})^2}{\frac{(s_{T1}^2/n_{T1})^2}{n_{T1}-1} + \frac{(s_{T2}^2/n_{T2})^2}{n_{T2}-1}+\frac{(s_{C1}^2/n_{C1})^2}{n_{C1}-1} + \frac{(s_{C2}^2/n_{C2})^2}{n_{C2}-1}}$$

    It might even make sense to assume that the two baseline measurements have equal variance and that the two endline measurements have equal variance, but that the baseline and endline might not have the same variance; that, too, could be done fairly readily (as could the assumption that the two control measurements had equal variance and the two intervention measurements had equal variance but control and treatment variances might differ).

  2. Alternatively, the equality of that linear combination can be tested as a contrast in a one-way ANOVA. Many packages make this simple. Again, some packages will support the option of doing this with unequal variances.

  3. Another approach is to do it as a two-way ANOVA type model; you can fit a model like:

    response ~ group + time + group:time

    where group is control or treatement, and time is the baseline/endline ('1' or '2'). In this case the group:time interaction is the thing you're interested in testing for.

  4. You could also construct a permutation test of this hypothesis, or a variety of other tests can be similarly adapted in the way I did above.

If you had paired data (before and after), many people would suggest a different approach to the differences-in-differences style of test here (such as repeated measures, or at least putting the baseline in the model as a control variable), but I don't think those sort of approaches are an option in this situation.

That said, if your sample sizes are equal, you perhaps shouldn't fear pairing up your samples artificially; simulations suggest that actually works surprisingly well.

Related Question