I'm a pharmacologist and, in my experience, almost all papers in basic biomedical research use Student's t-test (either to support inference or to conform to expectations…). A couple of years ago it came to my attention that Student's t-test is not the most efficient test that might be used: sequential tests offer much more power for any sample size, or a far smaller sample size on average for equivalent power.
Sequential procedures of varying complexity are used in clinical research but I've never seen one used in a basic biomedical research publication. I note that they are also absent from the introductory level statistics textbooks that are all that most basic scientists are likely to see.
My question is three-fold:
- Given the very substantial efficiency advantage of sequential tests, why are they not more widely used?
- Is there a drawback associated with the used of sequential methods that would mean that their use by non-statisticians is to be discouraged?
- Are statistics students taught about sequential testing procedures?
Best Answer
I don't know much of sequential tests and their application outside of interim analysis (Jennison and Turnbull, 2000) and computerized adaptive testing (van der Linden and Glas, 2010). One exception is in some fMRI studies that are associated to large costs and difficulty to enroll subjects. Basically, in this case sequential testing primarily aims at stopping the experiment earlier. So, I am not surprised that these very tailored approaches are not taught in usual statistical classes.
Sequential tests are not without their pitfalls, though (type I and II error have to be specified in advance, choice of the stopping rule and multiple look at results should be justified, p-values are not uniformly distributed under the null as in a fixed sample design, etc.). In most design, we work with a pre-specified experimental setting or a preliminary power study was carried out, to optimize some kind of cost-effectiveness criterion, in which case standard testing procedures apply.
I found, however, the following paper from Maik Dierkes about fixed vs. open sample design very interesting: A claim for sequential designs of experiments.