I've inherited a survey from an old employee that consists of 27 questions. We have on the order of 500 respondents but not all of them answered every question as some questions are of the type 'if yes then please answer this'. The survey has an overall reported margin of error of +/- 4% at the 95% confidence interval and I understand that this was derived from a table based on the CI and the number of respondents.
I wanted to understand what the table was doing and I found this formula to calculate margin of error:
$$
C =z\sqrt{\frac{p (1-p)}{N}}
$$
Where z is 1.96, N is the number of respondents and p is the response distribution (I think) (for example 45% of respondents say 'Yes').
My problem is that after reading about margin of error it seems incorrect to have calculated a margin for the entire survey. It seems like a margin of error would need to be calculated for every response (p) to every question.
So, my question: is it an acceptable practice to calculate an overall margin of error for a survey or should I, in a survey with 27 questions, calculate a margin of error for each question (given they have different respondent numbers) and then take the max as the margin of error for the whole survey? Or should I do something else entirely?
Best Answer
Technically, you are quite correct. The confidence interval depends on both N and p. Furthermore, the classical equation for the confidence interval of a binomial proportion isn't always all that good. There is quite a literature on this subject, a good starting point is this article by Agresti and Coull. `R' has implemented various of the suggested alternatives to the simple formula.
However, in practice, it is quite common to cite one CI for all the questions. Often the CI for the different questions do not vary that much. The question then becomes which CI to use as the "general" one. I would hope that they chose one calculated with a relatively small N.