Assume I want to use a (nonparametric) bootstrap test for a hypothesis with a sample size of $n_1$ and I already have $n_0$ actual samples on which to base my power estimates. Usually, we would also have $n_1 > n_0$.
Is it a valid procedure to estimate the power of my test by using nested bootstrapping? Basically, I would repeatedly sample $n_1$ samples with replacement from my $n_0$ samples and apply my non-parametric bootstrap test to each of these samples. Finally, I would look at the percentage of the bootstrap tests which was significant at my $\alpha$ level. Are there pitfalls I have to look out for?
I already did some googling without much success, perhaps because I do not know the proper search terms. Therefore it would be also nice to know how the procedure (if it exists) is called, so I can find references.
Best Answer
I don't think use bootstrap to artificially increase your sample size would be a good idea. Any violation of the assumption of the independence of the observations dramatically increase the odds od a spurious result (which would be the case when n1 is significantly greater than n0).
I would estimate the confidence interval of the effect size (the strength of the difference / relationship you are trying to use) and assume the lower bound as the true effect. Then it would be easy to estimate the power.
[note: I assume you already have a significan result with n0 observations. Otherwise your data is compatible* with the null hypothesis and there is no way to have a conservative estimate of the powere - unless you are using the wrong test. Power analysis assume the knowledge of the "real" effect size so there is no way to use theme to "bypass" inferential statistics]
*likely to be observed if the null hypotesis is true