Sampling Techniques – Why Larger Sample Sizes Are Needed for Monte Carlo Simulations Compared to Surveys

monte carlosamplingsurvey-sampling

I’m a bit of a stats-noob, so I am not sure I will manage to formulate this question properly, but let me do my best.

I‘m trying to develop an intuition for sample sizes and when they are sufficient to get to a reasonable degree of representativeness of a larger population.

I know that when doing surveys or polls, a sample size of mere hundreds or thousands is often sufficient, even for very large populations. (This is at least what calculators such as this one are telling me.)

So now what happens if we extend this notion to Monte Carlo simulations of a multi-parameter model? Every simulation can be thought of as representing a sample of the larger population of all possible simulations (the solution space).

The solution space of a multi-dimensional problem can be very large, but the survey/poll scenario tells me that this doesn’t mean that the sample size will need to be large.

This seems to suggest that any application of the Monte Carlo method can be concluded within several hundred/thousand of simulations as well. But I feel this cannot be true, as it would appear to defeat the purpose of the plethora of “more efficient alternatives” to Monte Carlo (such as Markov Chain Monte Carlo).

What am I missing? What is the difference between surveying a large population of people on their election vote (for example) and “surveying” a large population of parameter permutations on their performance (when plugged into a certain model)?

Why does a surprisingly small number of surveys suffice to get a decent idea of the likely outcome of an election, while a small number of Monte Carlo simulations does not suffice (I think) to get a decent idea of the likely (top) performance of a model?

My apologies for needing too many words, but it’s the best I can do I am afraid. I hope you catch my drift regardless and can steer me in the right direction. Thanks a lot!

Best Answer

I know that when doing surveys or polls, a sample size of mere hundreds or thousands is often sufficient, even for very large populations.

In the calculator you linked to, it only considers estimating a single proportion. The relationship between sample size, confidence level, and desired margin of error is simple for estimating a single proportion from iid binomial data. Political polls, at least, tend to be focused on just a few proportions (what % of voters favor candidate A over candidate B). If they assume random sampling and ignorable nonresponse, they can use such a calculator to find that, say, ~1000 respondents will get you a 95% MOE of $\pm$ 3 percentage points. Even if you wanted to do multiple comparisons corrections (though they often don't) for reporting a handful of proportions at once, ~1000 respondents is typically still good enough for reasonably narrow MOEs.

So now what happens if we extend this notion to Monte Carlo simulations of a multi-parameter model?

If you're studying a higher-dimensional space, you need more data if you want to honestly account for the uncertainty in studying many estimates at once. If your estimators have some complicated intractable distribution, sample-size calculations may be sketchy and you want to err on the side of more data. If you want to characterize the space in more detail than just a single (multi-dimensional) point estimate, you need more data.

In fact... this is true of surveys as well. There are plenty of much larger surveys, such as the ones that are run by national statistical offices (such as the US Census Bureau). Some ask simple binary questions where the above sample-size calculator works, but they want bigger samples in order to account for asking many such questions. Other questions are quantitative measurements and might need a different approach to estimate the sample size. They also often need bigger sample sizes in order to get precise sub-group estimates (think small geographic regions or small demographic groups). So just a few hundred responses is not always enough. For instance, the American Community Survey collects roughly 3 million responses each year.

Related Question