I'm trying to compute the minimum sample size for a psychometric test based on 7 point Likert scales. I'd like to run ANOVA on each scale to look for differences between groups.
Most online survey sample size calculators seem to be designed for polls, e.g. Yes/No, Agree/Disagree. They take as input population size, a confidence interval and a proportion (50% Yes/50% no) and then return the required sample size.
Most statistical books suggest using power tests (such as R's power.t.test), which take as input a minimum effect size, alpha, beta and a statistical test and then return the required sample size.
For my purposes power tests seems to make the most sense, but what has me concerned is that none of them take into account the population size, which seems like it ought to have at least some effect on the outcome.
So my question is, what is the correct calculation to use in my specific survey situation and more generally what is the connection between power tests and these online survey sample size calculators, does population size matter in some way, perhaps helping to capture the notion of representative sample?
Best Answer
The commonly used statistical methods assume that you take a sample of an infinite or very large population. ANOVA, too, has this assumption. When the subjects of your survey can be viewed as a representative sample of an existing or hypothetical much larger population, you do not need the finite population methods.
The second question is if ANOVA is appropriate to analyse the data collected. 7 point Likert scales are strictly speaking ordinal scales, so methods for ordinal dependent variables may be best. However, in psychometry it's usual to assume that the values from a Likert scale will follow a distribution that may be approximated with a normal distribution. In this case ANOVA is an acceptable method; the t-test too, although the latter compares two groups only. (The methods designed for binary (yes/no) outcomes may be used after setting a threshold in your Likert scale and dichotomising your data, however unless this threshold also exists in the psychological mechanism it will lead to loss of detail in your data and loss of power in your test. So not generally recommended.)
You need to check or think over if the homoscedasticity assumption of ANOVA is likely to be met. If yes, use a power formula for ANOVA and you need not worry about not having to specify the population size.