Say I have an obstacle course, which not everyone completes, though, globally, most do. I hypothesize that the treatment, drinking Gatorade, will cause an increase in the obstacle course completion rate (CR).
Due to a fluke in randomization, the test group has a pre-experiment 95% CR but an in-experiment 98% CR. Whereas the control has a pre-experiment 94% CR and a 96% in-experiment CR.
In both time windows, the test group is beating the control group; given that this bias exists, I cannot simply do a Z proportions test on the experimental data. I need to correct for this pre-experiment bias in some way.
My question is: How is this most frequently / easily done?
Edit 1: DAG added (courtesy Excalidraw). The fundamental issue is that I believe that two factors affect Completion Rate, both ability and treatment (gatorade.) Because pre-experiment bias exists in my data (it could be anything from better running shoes, increased stamina, height, etc) which is observed by the CR of test > CR control pre-experiment. I believe that ability is moderating the effect of gatorade such that drawing conclusions from the experimental results alone, might tell an incomplete story.
Edit 2: Perhaps this situation is best solved with a regression model, eg the x=pre-experiment ratios, y=post-experiment ratios and what really matters are the slopes for exposure (yes/no)?
Best Answer
I ended up using logistic regression and controlling for the confounding variables, which were likely to have caused the pre-experiment bias.