I'm working through a textbook (Regression and Other Stories) and have come across a particular problem that I am having difficulty convincing myself I understand.
I am specifically interested in part (b), but I include (a) as context.
It is as follows
'Before-after comparisons: The folder Sesame contains data from an experiment in which a randomly selected group of children was encouraged to watch the television program Sesame Street and the randomly selected control group was not.
(a) The goal of the experiment was to estimate the effect on child cognitive development of watching more Sesame Street. In the experiment, encouragement but not actual watching was randomized. Briefly explain why you think this was done. Think of practical as well as statistical reasons.
(b) Suppose that the investigators instead had decided to test the effectiveness of the program simply by examining how test scores changed from before the intervention to after. What assumption would be required for this to be an appropriate causal inference? Use data on just the control group from this study to examine how realistic this assumption would have been.'
I think this question is hinting at the problem of attaining an unbiased estimate for ATE from a difference in means from pre-post scores with systematic differences in pre-treatment variables between the treated and control groups, leading to differences in the outcome that is dependent on these differences rather than the effect of treatment – either through sampling bias. For example, more intelligent children are encouraged to watch the TV program because their parents are aware of its supposed effects, or through heterogeneous treatment effects such as socioeconomic factors.
The question suggests looking at control group data – which I have done, but I am not entirely sure what I am looking for in the data to test the above assumptions given the question suggests only looking at control data.
Best Answer
The validity of the post-pre estimator in the intervention group depends on there being no change in the control group. If there was change in the control group, then any changes you see in the treatment group could be due either to the treatment or to whatever caused changes in the control group. For example, if cognitive performance naturally increases in this period of a child's life, even in the absence of watching Sesame Street, then even if the intervention had no effect, you would expect to see increases in scores in the intervention group. Only by also observing the control group and noting the test score changes in the absence of the intervention can one ascribe causality to the intervention for score changes in the intervention group.
So, what you should do is to look in the control group for changes in scores between post and pre. If there are no changes, this suggests that just using the differences in the intervention group is sufficient to establish causality. If there are changes, then just using the intervention is not sufficient to establish causality; the competing explanation that the observed changes are due simply to the same forces that change scores in the control group remains, and you cannot say the intervention was effective. Subtracting the post-pre changes in the control group from the change in the intervention group would give you a causal effect; this is also known as the method of difference-in-differences.
In an experiment, this is not the best way to adjust for pre-intervention scores, but it does not invalidate the inference. If the experiment was randomized, comparing the post-difference means (assuming perfect compliance) is sufficient to estimate the causal effect. Adjusting for pre-treatment scores using regression/ANCOVA is even better. Things, of course, are a bit more complicated in the face of non-compliance.