Solved – How to use residuals to train a model

modelingrresiduals

I've read in a couple places about people using residuals from predictions of other models to improve the model's accuracy. Anyone have a link about how to carry this out in R, ideally with some example code?

UPDATE: This is in the context of reading the papers about the Netflix prize where k-nearest neighbors (KNN) models were trained using residuals from other KNN models.

Best Answer

This technique is used in a variety of ways. For example, boosting. There are also various multi-level classifications algorithms that do similar things. The idea is to see where your first level makes mistakes and concentrate further learning in that area, keeping what you have and building on it rather than changing what you have.

Related Solutions

Solved – Combining machine learning models

It actually boils down to one of the "3B" techniques: bagging, boosting or blending.

In bagging, you train a lot of classifiers on different subsets of object and combine answers by average for regression and voting for classification (there are some other options for more complex situations, but I'll skip it). Vote proportion/variance can be interpreted as error approximation since the individual classifiers are usually considered independent. RF is in fact a bagging ensemble.

Boosting is a wider family of methods, however their main point is that you build next classifier on the residuals of the former, this way (in theory) gradually increasing accuracy by highlighting more and more subtle interactions. The predictions are thus usually combined by summing them up, something like calculating a value of a function in x by summing values of its Taylor series' elements for x.
Most popular versions are (Stochastic) Gradient Boosting (with nice mathematical foundation) and AdaBoost (well known, in fact a specific case of GB). From a holistic perspective, decision tree is a boosting of trivial pivot classifiers.

Blending is an idea of nesting classifiers, i.e. running one classifier on an information system made of predictions of other classifiers. As so, it is a very variable method and certainly not a defined algorithm; may require a lot of objects (in most cases the "blender" classifier must be trained on a set of objects which were not used to build the partial classifiers to avoid embarrassing overfit).
The predictions of partial classifiers are obviously combined by melding them into an information system which is predicted by the blender.

Calculating R-Squared – How to Calculate $R^2$ in Mixed Models Using R2glmm Method

I am answering by pasting Douglas Bates's reply in the R-Sig-ME mailing list, on 17 Dec 2014 on the question of how to calculate an $R^2$ statistic for generalized linear mixed models, which I believe is required reading for anyone interested in such a thing. Bates is the original author of the lme4package for R and co-author of nlme, as well as co-author of a well-known book on mixed models, and CV will benefit from having the text in an answer, rather than just a link to it.

I must admit to getting a little twitchy when people speak of the "R2 for GLMMs". R2 for a linear model is well-defined and has many desirable properties. For other models one can define different quantities that reflect some but not all of these properties. But this is not calculating an R2 in the sense of obtaining a number having all the properties that the R2 for linear models does. Usually there are several different ways that such a quantity could be defined. Especially for GLMs and GLMMs before you can define "proportion of response variance explained" you first need to define what you mean by "response variance". The whole point of GLMs and GLMMs is that a simple sum of squares of deviations does not meaningfully reflect the variability in the response because the variance of an individual response depends on its mean.

Confusion about what constitutes R2 or degrees of freedom of any of the other quantities associated with linear models as applied to other models comes from confusing the formula with the concept. Although formulas are derived from models the derivation often involves quite sophisticated mathematics. To avoid a potentially confusing derivation and just "cut to the chase" it is easier to present the formulas. But the formula is not the concept. Generalizing a formula is not equivalent to generalizing the concept. And those formulas are almost never used in practice, especially for generalized linear models, analysis of variance and random effects. I have a "meta-theorem" that the only quantity actually calculated according to the formulas given in introductory texts is the sample mean.

It may seem that I am being a grumpy old man about this, and perhaps I am, but the danger is that people expect an "R2-like" quantity to have all the properties of an R2 for linear models. It can't. There is no way to generalize all the properties to a much more complicated model like a GLMM.

I was once on the committee reviewing a thesis proposal for Ph.D. candidacy. The proposal was to examine I think 9 different formulas that could be considered ways of computing an R2 for a nonlinear regression model to decide which one was "best". Of course, this would be done through a simulation study with only a couple of different models and only a few different sets of parameter values for each. My suggestion that this was an entirely meaningless exercise was not greeted warmly.

Best Answer

Related Solutions

Solved – Combining machine learning models

Calculating R-Squared – How to Calculate $R^2$ in Mixed Models Using R2glmm Method

Related Question