I have sent a survey to 92 persons but so far I have received only 40 responses. By using sample size requirement calculation I need 75 responses for the desired criteria
Margin of error = 5%
Confidence level = 95%
Response distribution = 50%
Is it appropriate to use this sample size calculation for a small population (N=92)?
I used this Sample size calculator
Thanks in advance for your help !
Best Answer
Take a step back and consider what you really need to know. Remember, statistics is concerned with Type I (false positive) and Type II (false negative) errors. Sample size calculators like the one you linked to focus on Type I errors expressed as p-values or confidence intervals, i.e. the probability that any effect you detect is due to random error vs. "true" population differences. That is important because you want to make sure you have statistical significance.
But you also need to be able to detect a difference if it is there - that is where you consider Type II errors - you need enough sample to detect an effect when it is really present. This is the Power of your test. Power analysis tells you the minimally detectable effect size your sample can attain.
Power analysis can get incredibly confident, but for your purpose, it is relatively straightforward. With teaching to fish in mind, go ahead and download G*Power, a free power analysis program that will work great for your purposes.
For example, assuming you are doing a test of proportions, this chart can help you out:
The three lines show you a curve of how much sample size you need to attain a given power (0.80 is kind of a standard, but it is more a convention than a rule, much like why we choose 0.05 rather than 0.051 or 0.06 with pvalues), for a given difference in proportions. The chart assumes that the reference sample (p1) is 0.5, so the lines represent the proportion for the comparison group (p2).
The main point here is: depending on the size of the differences between the two groups you are looking at, you may need different sample sizes. Power analysis for means, regression, etc., involves other factors. Cohen's work (e.g. this is the key reference here, though there are tons of other primers out there.
A couple of other points:
You have a small population (92) - that could call for a finite population correction, which would reduce your standard errors further and make your "small" sample size not so small, and help you out. Of course, you have to make the argument that the pop of 92 doesn't reflect a broader super-population. That's somewhat of a philosophical argument. See Andrew Gelman's short piece or an older article that seems to kick it off.
I hope you are thinking about response rates - you've got a response rate of 44%. Some fields that's good, others that's bad. Response rate is a concern in that it may reflect non-response bias. There is a ton of literature on that in survey research it wouldn't hurt to be familiar with. I've got a blog post summarizing it.