I've recently come across a method known as "Controlled-experiment Using Pre-Existing Data" (CUPED), which is a regression adjustments technique to create unbiased estimators in experiments. Likewise, I have prior familiarity with difference-in-differences (DiD).
As I understand it, DiD makes the assumption that treatment groups and control groups are changing at the same rate; the means of treatment and control might differ before the experiment begins. However, we're interested in the change in the treatment group (while accounting for the existing trend) and the change in the control group (only the existing trend).
CUPED seems to accomplish much the same thing. But I've heard that instead of modeling the difference in trends, it tries to account for trends such that in the absence of treatment exposure, the treatment group is corrected to match the control group. And this adjustment carries through the experiment such that treatment and control groups could be comparable directly. Is any of this true?
Further, I have a few questions:
- Does CUPED assume non-zero trends exist in both treatment and control groups?
- If [1], does CUPED assume the same trend?
- Does CUPED's adjustment require data from pre- and post-experiment start (like DiD)? Or does CUPED instead use purely pre-experiment panel data to make the adjustment?
- Do both treatment and control groups receive an adjustment?
Best Answer
i'm try to answer this question from a practitioner's perspective: in A/B testing, or online controlled experiment, when people mention CUPED, or DID, what are they actually doing and the assumptions reflected in these practices.
for simplicity, we start from a simulated data:
Diff-in-Diff
in practical A/B testing, some people do use DID because it seems intuitive, although many other people criticize this method as being inappropriate in the context of randomized experiments.
here is summary stat, the raw delta y is 0.197327 - 0.00625198 = 0.191075, pre-exist delta x is -0.00297056 - 0.0151793 = -0.0181499, close but not 0.
DID correct the pre-exist delta x by this: (0.197327 - 0.00625198) - (-0.00297056 - 0.0151793) = 0.209225.
here is so called DID model, also as change score model. the estimate of t is the same as the value we calculated manually before
in DID model, if without treat effect, y = x + intercept, here intercept is time effect, which is constant for all.
CUPED
the main idea of CUPED is:
$$\theta = \frac{\operatorname{cov}(Y, X)}{\operatorname{var}(X)} = \operatorname{corr}(X, Y) \cdot \operatorname{var}(Y) = \rho \cdot \operatorname{var}(Y)$$
$\theta$ turns out to be the ordinary least square (OLS) solution of regressing Y on X
the result cuped-adjusted estimate treat effect is
$$\tau = (\overline Y_1 - \theta \cdot (\overline X_1 - \mu_X)) - (\overline Y_0 - \theta \cdot (\overline X_0 - \mu_X)) \\= (\overline Y_1 - \overline Y_0) - \theta \cdot (\overline X_1 - \overline X_0) $$
in our case, $\tau = 0.197327 - 0.00625198 - (-0.00297056 - 0.0151793) * \theta = 0.201874 $, the same as below
DID vs CUPED
the only difference is $\theta$, in DID, $\theta$ fixed to 1, in CUPED, $\theta = \frac{\operatorname{cov}(Y, X)}{\operatorname{var}(X)}$, there are also other calculation methods.
because of the difference in $\theta$, the standard errors of the CUPED and DID estimates are different. Generally, the cuped estimate is more accurate.
because of the difference in $\theta$, there are also slight differences in the treatment effects estimated by CUPED and DID. refer to (Best practice when analysing pre-post treatment-control designs)[https://stats.stackexchange.com/questions/3466/best-practice-when-analysing-pre-post-treatment-control-designs]
Back to your question
yes. $\theta$ is common for both treatment and control groups. But I think we can also estimate $\theta$ in the experimental group and the control group separately.
CUPED's adjustment require data from pre- and post-experiment start and both treatment and control groups receive an adjustment