Experiment-Design – How to Design Experiments for Two Types of Treatment and Holdout Effectively

case-control-studycausalityexperiment-designtreatment-effect

Let's say there are two types of treatment, namely treatment A and treatment B.

A subject can be in one of these categories:

  1. get treatment A and then treatment B.
  2. get treatment B and then treatment A.
  3. get only treatment A.
  4. get only treatment B.
  5. get no treatment.

There are several ways we can construct the control group. For example, we can:

  1. construct a pure control group who gets no treatment.
  2. construct control groups for treatment A and treatment B independently. A portion of the control subjects for treatment A may have gotten treatment B. In other words there can be overlap.

Edit: Note that I only have ability to design and assign control group in the experiment. The treatment order happens naturally in the test group without moderation.

Also the post-experiment analysis to estimate treatment effects get tricky since there are two types of treatment, and they may interact with each other.

Anyone have good pointers for reference I can read on?

Edit:

Context was requested in the comment section, so I'd like to provide some exemplar cases:

Weight study

  • Treatment A: doing yoga daily
  • Treatment B: running 3 miles daily
  • Metric: body weight in kg

Button on a web page

  • Treatment A: change button's location
  • Treatment B: change button's color
  • Metric: click-through-rate

Completing online course

  • Treatment A: chunking hour-long videos into smaller sessions
  • Treatment B: sending reminder emails
  • Metric: course completion rate

In general, carrying out multiple experiments to estimate effect of two types of treatment on the same metric requires more time and/or subjects and increases cost. It also ignores potential interaction effect of treatment A and treatment B.

Best Answer

OP mentioned that they ended up selecting control groups for the treatments independently.

In this case, if there is no clear bias in the assignment mechanisms of treatment A and treatment B (e.g. somehow applying treatment B also increases probability of treatment A), the subtraction in the calculation of average treatment effect should naturally cancel out the impact of the other treatment on both treatment and control group.

If one suspects that there is some dependency between treatment A and treatment B, it is always possible to manually test for the independence of the two random events.

In the case of treatment A and treatment B being dependent, without loss of generality, one can adjust for the bias from treatment B by conducting regression adjustment with the propensity scores of treatment A given treatment B. I suspect that's what OP went for directly without testing for independence.

Related Question