Solved – Calculating necessary sample size using bootstrap

bootstrapsample-size

I have recently come across a website (http://www.surveysystem.com/sscalc.htm) that returns the sample size given the following inputs: confidence level, confidence interval, and population. I assume this is done by rearranging a hypothesis test under a CDF, I guess using the standard normal distribution? But, if one does not believe the data reflects this particular CDF, how would you go about using bootstrapping to arrive at a version of the sample size that is data driven and not limited to the ~N(0,1) distribution?

Further, I am interested in this procedure for stratifications of a sample.

Help on either, especially the first paragraph, would be appreciated.

Best Answer

Ok, so this answer might not be exactly what you were after based on the detail of your question, but I stumbled across your question based on just the title and so this might help other people who also come across it in a similar fashion.

The only way I know of determining sample size using a bootstrap is via a power analysis approach. That is you:

  1. State the null hypothesis and alternative hypothesis
  2. State the alpha level (typically 5%)
  3. If necessary shift the pilot study data so that you know the null hypothesis is false
  4. Re-sample with replacements from the pilot study
  5. Perform the test on the this sample and record the result
  6. Repeat 1000 or so times to build up probability distribution
  7. Count how many times the null hypothesis is rejected

With many possible "variations on a theme of..."

And that gives you the statistical power (for that sample size and that particular test), because the definition of statistical power is "probability that the test will reject the null hypothesis when the alternative hypothesis is true". So you can then vary the sample size until you achieve the desired power.

Here's an approach in R that I did based on this paper, Sample Size / Power Considerations, by Elizabeth Colantuoni.

I had two groups of non-normal, non-parametric data. A pilot study of each showed them to have differing medians and a Mann Whitney Wilcoxon test rejected the null hypothesis that they were the same, but I wanted to determine the sample size required so I could say this for "sure". Since the test already rejected the null hypothesis on the pilot data I did not see any need to shift or manipulate the data to ensure the alternative hypothesis was true.

power = function(group1.pilot, group2.pilot, reps=1000, size=10) {
    results  <- sapply(1:reps, function(r) {
        group1.resample <- sample(group1.pilot, size=size, replace=TRUE) 
        group2.resample <- sample(group2.pilot, size=size, replace=TRUE) 
        test <- wilcox.test(group1.resample, group2.resample, paired=FALSE)
        test$p.value
    })
    sum(results<0.05)/reps
}

#Find power for a sample size of 100
power(data1, data2, reps=1000, size=100)

Necessary disclaimer: I'm not a statistician and I'm still learning about bootstrapping so feedback, corrections and pointing and laughing are welcome.

Related Question