Imagine you have a study with two groups (e.g., males and females) looking at a numeric dependent variable (e.g., intelligence test scores) and you have the hypothesis that there are no group differences.
Question:
- What is a good way to test whether there are no group differences?
- How would you determine the sample size needed to adequately test for no group differences?
Initial Thoughts:
- It would not be enough to do a standard t-test because a failure to reject the null hypothesis does not mean that the parameter of interest is equal or close to zero; this is particularly the case with small samples.
- I could look at the 95% confidence interval and check that all values are within a sufficiently small range; perhaps plus or minus 0.3 standard deviations.
Best Answer
I think you are asking about testing for equivalence. Essentially you need to decide how large a difference is acceptable for you to still conclude that the two groups are effectively equivalent. That decision defines the 95% (or other) confidence interval limits, and sample size calculations are made on this basis.
There is a whole book on the topic.
A very common clinical "equivalent" of equivalence tests is a non-inferiority test/ trial. In this case you "prefer" one group over the other (an established treatment) and design your test to show that the new treatment is not inferior to the established treatment at some level of statistical evidence.
I think I need to credit Harvey Motulsky for the GraphPad.com site (under "Library").