I have tested a group of participants over various time points and I want to investigate whether mean score on each subsequent time point is significantly different to the mean score at baseline (eg. score on day 0 versus day 1, score on day 0 versus day 2 etc). My plan is to perform a repeated measures 1 way ANOVA and the Dunnett's posthoc test to compare the mean scores at day with the basline day score. However, the n number for the group is 6. Is a one way ANOVA appriate with a low n number like this? Is there a minimum sample size for a 1 way ANOVA? If so, is there a better alternative statistic to use?
Solved – Minimum sample size for 1 way ANOVA
anovapost-hocsample-size
Best Answer
Described below are three approaches to estimating sample size for completely randomized designs. Note that the procedures differ in terms of the information you must provide.
Approach #1 (requires most information)
To calculate sample size, the researcher first needs to specify:
1) level of significance, α (alpha)
2) power, 1-β
3) size of the population variance, σ2
4) sum of the squared population treatment effects.
In practice, 3 and 4 are unknown. However, you can estimate both from a pilot stud. Alternatively, you might estimate these parameters from previous research.
As an example, let's assume we conducted a pilot study and estimated the population variance and sum of squared population treatment effects. If we let α = 0.05 and 1-β = 0.80, then we can use trial and error to calculate the required sample size. The test statistic you calculate is phi (Φ), where:
Φ = (n^0.5)[(average of squared treatment effects/population variance)], where n is a sample size value. The Φ test statistic can then be used to look up power that corresponds to the sample size in Tang's Charts (citation below).
Approach #2
If accurate estimates of #3 and #4 are not available from a pilot study or previous research, then one can use an alternative approach that requires a general idea about the size of the difference between the largest and smallest population means relative to the standard deviation of the population standard deviation:
μmax - μmin = dσ, where d is a multiple of the population standard deviation. In other words, this approach allows you to calculate the sample size if you wanted to detect a difference between the highest and smallest means that would equal to some multiplier of the pop. standard deviation (whether it's one half, or 1.5, or anything else). For the math for this approach, see Kirk (2013) [I have a PDF].
Approach 3
If you know nothing about #3 and #4 from Approach 1, and are unable to express μmax - μmin as a multiple of pop. standard deviation, then you can use strength of association or the effect size to calculate the sample size. This approach also requires the researcher to specify the level of significance, α, as well as the power, 1 - β.
Remember that the strength of an association indicates the proportion of the population variance in the dependent variable that is accounted for by the independent variable. Omega squared is used to measure the strength of association in analysis of variances with fixed treatment effects, whereas intraclass correlation is used in analysis of variance with random treatment effects.
Based on Cohen (1988), we know that (for strength of association):
ω^2 = 0.010 is a small association
ω^2 = 0.059 is a medium association
ω^2 = 0.138 or larger is a large association
And for effect size:
f = 0.10 is a small effect size
f = 0.25 is a medium effect size
f = 0.40 or larger is a large effect size.
Back to the Approach 3: if we have a completely randomized design with p treatment levels, then we can calculate the sample size necessary to detect any magnitude of strength of association OR any magnitude of effect size (the mathematics are equivalent). Again, if you are interested for working out the math for this approach, I refer you to Kirk (2013).
Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Hillsdale, NJ: Lawrence Erlbaum
Kirk, R.E. (2013) Experimental Design: Procedures for the Behavioral Sciences
Tang, P. C. The power function of the analysis of variance test with tables and illustrations of their use.Statist. res. Memoirs. 1938,2, 126–149.