The parameter tolerance.method determines how the tolerance threshold (which is used to determine the optimal number of trees) shall be calculated.
Let's look at the paper and the source code to first understand the purpose of the tolerance threshold, and then how is it calculated.
Purpose of the tolerance threshold
The paper (Elith, Leathwick & Hastie 2008) states (p. 807) that the function gbm.step implements cross-validation to determine the optimal number of trees (as detailed in Figure 4, which I will paste here since the PDF treats it as an image, not text.)
The tolerance threshold helps determine when "the average of the more recent set is higher than the average of the previous set" (step 5).
The source code for gbm.step (line 159) shows that the algorithm will continue to build trees while (delta.deviance > tolerance.test & n.fitted < max.trees).
tolerance.test is the tolerance thresholddelta.deviance is defined as default of 1 (line 150), which will never fall below the tolerance threshold, but when at least 20 trees have been built, then:
(on line 220)
if (j >= 20) {
test1 <- mean(cv.loss.values[(j - 9):j])
test2 <- mean(cv.loss.values[(j - 19):(j - 9)])
delta.deviance <- test2 - test1
}
In other words, the reduction in means of loss functions by the most recent 10 iterations as compared to the 10 iterations before that.
It's worth noting the apparent discrepancy in the source code (v 1.1.1) which compares the current to 10th previous iterations against the 11th-20th iterations, and step 5 in the figure, which compares the current to 5th against the 6th-10th. So the code is a little more conservative in that it uses a larger window for averaging the loss function.
How is the tolerance threshold calculated?
By default, tolerance.method=auto and tolerance=0.001.
On line 77:
mean.total.deviance <- total.deviance/n.cases
tolerance.test <- tolerance
if (tolerance.method == "auto") {
tolerance.test <- mean.total.deviance * tolerance
}
Since there is no corresponding adjustment for tolerance.method == "fixed", the algorithm would use the default or user-provided argument without adjustment.
So you can specify tolerance.test as an absolute deviance (via tolerance.method='fixed') or relative to the mean total deviance (via tolerance.method='auto').
For the case of neural networks, this is a promising approach: WTTE-RNN - Less hacky churn prediction.
The essence of this method is to use a Recurrent Neural Network to predict parameters of a Weibull distribution at each time-step and optimize the network using a loss function that takes censoring into account.
The author also released his implementation on Github.
Best Answer
According to
gbm's reference manual: While indeedtype="response"thengbmconverts back to the same scale as the outcome this currently will be returning probabilities forbernoulliand expected counts forpoissononly. For all other distributionsresponseandlinkreturn the same. That said,gbm.predictwill indeed transform the response when the assumed cost function is Adaboost.Here is a small R example about it:
As we can see the
responsepredictions are indeed between $[0,1]$ for Adaboost too. The manually conversion to probabilities between thelinkpredictions andresponsepredictions in the case of Bernoulli requires the using the inverse logit: $\frac{1}{1+exp(-x)}$ while for the Adaboost is $\frac{1}{1+exp(-2x)}$. The $2$ comes directly from the inversion of $\frac{1}{2} \ln(\frac{1-\epsilon}{\epsilon})$, see Schapire (2013) "Explaining Adaboost" for more details. We can see the the manual calculations here: