Lurking-Variables – Examples of Lurking Variables in Controlled Experiments

confoundingrandom allocation

In this paper:

Lurking Variables: Some Examples
Brian L. Joiner
The American Statistician
Vol. 35, No. 4, Nov., 1981 227-233

Brian Joiner claims that "randomization is not a panacea". This is contrary to common statements such as the one below:

A well-designed experiment includes design features that allow
researchers to eliminate extraneous variables as an explanation for
the observed relationship between the independent variable(s) and the
dependent variable. These extraneous variables are called lurking
variables.

The quote was taken from this question and does not have a source but in my experience it is representative of the prevailing attitude:
Examples of Lurking Variable and Influential Observation

One example given is that when testing the safety (specifically carcinogenesis) of red #40 food dye on rodents in the seventies an effect of cage position was found to confound the study. Now I have read many journal articles studying carcinogenesis in rodents and have never seen anyone report controlling for this effect.

Further discussion of these studies can be found here:
A case study of statistics in the regulatory process: the FD&C Red No. 40 experiments.

I could not find a non-paywalled version but here is an excerpt:

At the January meeting, we presented a preliminary analysis (14) that
disclosed a strong correlation between cage row and RE (reticulo-endothelial tumor) death rates,
which varied from 17% (bottom row) to 32% (top row) (table 2). We
could not explain this strong association by sex, dosage group, or
rack column or position. A subsequent analysis (18) also indicated
that cage position (front vs. back) might be correlated with non-RE
mortality and that position was correlated with time to non-RE death.

I am specifically interested in why there seems to be such a problem with replication in the medical literature, but examples from all fields would be welcome. Note that I am interested in examples from randomized controlled experiments, not observational studies.

Best Answer

A few examples from clinical research might be variables that arise after randomization - randomization doesn't protect you from those at all. A few off the top of my head, that have been raised as either possibilities or been noted:

  • Changes in behavior post voluntary adult male circumcision for the prevention of HIV
  • Differential loss to follow-up between treatment and control arms of an RCT
  • A more specific example might include the recent "Benefits of Universal Gowning and Gloving" study looking at prevention of hospital acquired infections (blog commentary here, the paper is behind a paywall). In addition to the intervention, and potentially because of it, both hand hygiene rates and contact rates between patients and staff/visitors changed.

Randomization protects against none of those effects, because they arise post-randomization.

Related Question