rregressionridge regression
I'm very impressed by this plot: Why does ridge estimate become better than OLS by adding a constant to the diagonal?
Does someone has any clue about how to plot this on R? I mean, how to get RSS values for differents parameter estimation? With lm function, I will have only one solution!
Thanks for your help!
In an unpenalized regression, you can often get a ridge* in parameter space, where many different values along the ridge all do as well or nearly as well on the least squares criterion.
* (at least, it's a ridge in the likelihood function -- they're actually valleys$ in the RSS criterion, but I'll continue to call it a ridge, as this seems to be conventional -- or even, as Alexis points out in comments, I could call that a thalweg, being the valley's counterpart of a ridge)
In the presence of a ridge in the least squares criterion in parameter space, the penalty you get with ridge regression gets rid of those ridges by pushing the criterion up as the parameters head away from the origin:
In the first plot, a large change in parameter values (along the ridge) produces a miniscule change in the RSS criterion. This can cause numerical instability; it's very sensitive to small changes (e.g. a tiny change in a data value, even truncation or rounding error). The parameter estimates are almost perfectly correlated. You may get parameter estimates that are very large in magnitude.
By contrast, by lifting up the thing that ridge regression minimizes (by adding the $L_2$ penalty) when the parameters are far from 0, small changes in conditions (such as a little rounding or truncation error) can't produce gigantic changes in the resulting estimates. The penalty term results in shrinkage toward 0 (resulting in some bias). A small amount of bias can buy a substantial improvement in the variance (by eliminating that ridge).
The uncertainty of the estimates are reduced (the standard errors are inversely related to the second derivative, which is made larger by the penalty).
Correlation in parameter estimates is reduced. You now won't get parameter estimates that are very large in magnitude if the RSS for small parameters would not be much worse.
There are lots of penalized approaches that have all kinds of different penalty functions now (ridge, lasso, MCP, SCAD). The question of why is one of a particular form is basically "what advantages/disadvantages does such a penalty provide?".
Properties of interest might be:
1) nearly unbiased estimators (note all penalized estimators will be biased)
2) Sparsity (note ridge regression does not produce sparse results i.e. it does not shrink coefficients all the way to zero)
3) Continuity (to avoid instability in model prediction)
These are just a few properties one might be interested in a penalty function.
It is a lot easier to work with a sum in derivations and theoretical work: e.g. $||\beta||_2^2=\sum |\beta_i|^2$ and $||\beta||_1 = \sum |\beta_i|$. Imagine if we had $\sqrt{\left(\sum |\beta_i|^2\right)}$ or $\left( \sum |\beta_i|\right)^2$. Taking derivatives (which is necessary to show theoretical results like consistency, asymptotic normality etc) would be a pain with penalties like that.
Best Answer
It's a plot of the objective function (i.e. the sum of squares, the thing you're trying to minimize with OLS) as a function of the coefficients. One way to compute this is to generate a "grid" of possible coefficients and then compute the sum of squares at each point on the grid.
That is, we fit a regression by finding $\hat\beta \equiv \arg \min_\beta \{ y-x\beta \}$, i.e. the $\hat\beta$ that minimizes $SS_{reg}$. But we can compute $y-x\beta$ for any $\beta$ value. So we can generate a bunch of different $\beta$ values and just plug each one into that $SS_{reg}$ formula. In this case, $\beta$ has two components (i.e. the intercept and slope), so you can visualize all of the possible $SS_{reg}$s as a 3D surface.
As for generating the plot itself, look here for a complete example with R code.