boostingcartmachine learning
I'm reading Introduction to Statistical Learning, James, G., et al. (2013), in which they describe the Boosted Regression Tree algorithm as following. What I do not understand is Eq 8.10 and 8.11. What do "adding the new tree to the old tree" & "update the residual", as signified by $\leftarrow$, mean mathematically?
The working guide to boosted regression tree by the same authors do not explain how exactly to add the trees either.
I expect that there would be some difference in the training and CV AUC scores, but should this much of a difference be of concern? If not, how should I interpret and report these results? If it is of concern, what are some possible reasons for the differences and strategies I can take to fix them?
You are overfitting the training data. The stark decrease in AUC shows that given new data, your model would likely not perform as well as it does for the training data.
- The data are ordered by date, but I permutated the data row-wise before using gbm.fixed and predict.gbm. Also, from what I understand gbm.step also randomized the data.
Structured dependency in the data is something that you should try to capture in the model. If date/time is important then you should find a way to include it. Admittedly this is ignored by most machine learning algorithms.
- Could I have too few observations or too many variables? Could overfitting be an issue?
Yes, your results are the definition of overfitting.
- All of the individual animals are pooled together, could differences in the preferences of each individual animal be contributing.
It is possible. Another consideration for model development.
- Could the number CV folds in the gbm.step or gbm. simplify be at play?
Yes, read about the bias-variance trade-off.
CART can capture interaction effects. An interaction effect between $X_1$ and $X_2$ occurs when the effect of explanatory variable $X_1$ on response variable $Y$ depends on the level of $X_2$. This happens in the following example:
The effect of poor economic conditions (call this $X_1$) depends on what type of building is being purchased ($X_2$). When investing in an office building, poor economic conditions decrease the predicted value of the investment by 140,000 dollars. But when investing in an apartment building, the predicted value of the investment decreases by 20,000 dollars. The effect of poor economic conditions on the predicted value of your investment depends on the type of property being bought. This is an interaction effect.
Best Answer
They assume that you're keeping track of a "current estimator" $\hat f$, which is a sum of all the trees you've seen so far. (In code you would just store this as an array of all the trees you've seen so far.) The $\leftarrow$ sign just means "takes the new value"--so when they say "add the new tree" they mean, basically, append the new tree to the array of trees you already store, so that where you previously would have computed $\hat f$ with that array, you now compute $\hat f + \lambda \hat f^b$. (The $\leftarrow$ sign just means "takes the new value".)
The residual is just the difference between the response and your current prediction $\hat f$. So if you add something to $\hat f$ you need to subtract it from the residual so that they continue to sum up to the target response. Again, the $\leftarrow$ sign just means "takes the new value"--so $r_i \leftarrow r_i - \lambda \hat f^b(x_i)$ would translate in code to
r[i] -= lambda * tree_prediction[i]or something.