The short answer, if I understand the questions, is "no". Out of sample error is out of your sample and no bootstrapping or other analytical effort with your sample can calculate it.
In answer to your comment on whether the bootstrap can be used in checking a model with data outside a training set: two possible interpretations.
It would be fine, and absolutely standard, to fit a model on your training set with traditional methods and then use bootstrapping on the training set to check for things like distribution of your estimators, etc. Then use your final model from that training set to test against the test set.
It would be possible to do a bootstrap-like procedure that involves a loop around:
- selecting a subset of the whole sample as your training set
- fit a model to that training set of the data
- compare that model to the testing set of the remaining data and generate some kind of test statistic that says how well the model from the training set goes against the test set.
And then considering the results of doing that many times. Certainly, it would give you some insight into the robustness of your train/test process. It would reassure you that the particular model you got was not just due to the chance of what ended up in the test set in your one split.
However, it's difficult to say exactly why but there seems to me to be a philosophical clash between the idea of a testing/training division and the bootstrap. Perhaps if I didn't think of it as a bootstrap, but just a robustness test of the train/test process it would be ok...
The answer given by miura is not entirely accurate so I am answering this old question for posterity:
(2). These are very different things. The empirical cdf is an estimate of the CDF (distribution) which generated the data. Precisely, it is the discrete CDF which assigns probability $1/n$ to each observed data point, $\hat{F}(x) = \frac{1}{n}\sum_{i=1}^n I(X_i\leq x)$, for each $x$. This estimator converges to the true cdf: $\hat{F}(x) \to F(x) = P(X_i\leq x)$ almost surely for each $x$ (in fact uniformly).
The sampling distribution of a statistic $T$ is instead the distribution of the statistic you would expect to see under repeated experimentation. That is, you perform your experiment once and collect data ${X_1,\ldots,X_n}$. $T$ is a function of your data: $T = T(X_1,\ldots,X_n)$. Now, suppose you repeat the experiment, and collect data ${X'_1,\ldots,X'_n}$. Recalculating T on the new sample gives $T' = T({X'_1,\ldots,X'_n})$. If we collected 100 samples we would have 100 estimates of $T$. These observations of $T$ form the sampling distribution of $T$. It is a true distribution. As the number of experiments goes to infinity its mean converges to $E(T)$ and its variance to $Var(T)$.
In general of course we don't repeat experiments like this, we only ever see one instance of $T$. Figuring out what the variance of $T$ is from a single observation is very difficult if you don't know the underlying probability function of $T$ a priori. Bootstrapping is a way to estimate that sampling distribution of $T$ by artificially running "new experiments" on which to calculate new instances of $T$. Each new sample is actually just a resample from the original data. That this provides you with more information than you have in the original data is mysterious and totally awesome.
(1). You are correct--you would not do this. The author is trying to motivate the parametric bootstrap by describing it as doing "what you would do if you knew the distribution" but substituting a very good estimator of the distribution function--the empirical cdf.
For example, suppose you know that your test statistic $T$ is normally distributed with mean zero, variance one. How would you estimate the sampling distribution of $T$? Well, since you know the distribution, a silly and redundant way to estimate the sampling distribution is to use R to generate 10,000 or so standard normal random variables, then take their sample mean and variance, and use these as our estimates of the mean and variance of the sampling distribution of $T$.
If we don't know a priori the parameters of $T$, but we do know that it's normally distributed, what we can do instead is generate 10,000 or so samples from the empirical cdf, calculate $T$ on each of them, then take the sample mean and variance of these 10,000 $T$s, and use them as our estimates of the expected value and variance of $T$. Since the empirical cdf is a good estimator of the true cdf, the sample parameters should converge to the true parameters. This is the parametric bootstrap: you posit a model on the statistic you want to estimate. The model is indexed by a parameter, e.g. $(\mu, \sigma)$, which you estimate from repeated sampling from the ecdf.
(3). The nonparametric bootstrap doesn't even require you to know a priori that $T$ is normally distributed. Instead, you simply draw repeated samples from the ecdf, and calculate $T$ on each one. After you've drawn 10,000 or so samples and calculated 10,000 $T$s, you can plot a histogram of your estimates. This is a visualization of the sampling distribution of $T$. The nonparametric bootstrap won't tell you that the sampling distribution is normal, or gamma, or so on, but it allows you to estimate the sampling distribution (usually) as precisely as needed. It makes fewer assumptions and provides less information than the parametric bootstrap. It is less precise when the parametric assumption is true but more accurate when it is false. Which one you use in each situation you encounter depends entirely on context. Admittedly more people are familiar with the nonparametric bootstrap but frequently a weak parametric assumption makes a completely intractable model amenable to estimation, which is lovely.
Best Answer
Your permutation test is correct for testing if there is a difference in the quality of the 2 models. I don't understand enough about your bootstrap approach to know if it is correct or not. Another book to consider is "Bootstrap Methods and their Application" by Davison and Hinkley. It is a bit more recent and I believe more applied than Efron and Tibshirani.