Solved – R Ridge Regression: Choosing best lambda

rridge regression

I am doing ridge regression with Mass package and stuck with the problem trying to find the best lambda.
I know that it should look somehow like thi

lm.ridge(y ~x1 + x2+ x3...data = data, lambda=seq(0,1,.1))
help <- lm.ridge(y ~x1 + x2+ x3...data = data, lambda=seq(0,1,.1))
plot(help)
select(help)

select(help) gives me the following output:

modified HKB estimator is 13.50453 
modified L-W estimator is 85.33649 
smallest value of GCV  at 1

But what is my best lambda the? The smallest value for GCV?
Also I don't understand what values I should pick for

lambda=seq(0,1,.1)

I hope someone can help me!

Best Answer

(Changing a comment to an answer.) Yes, you want the lambda that minimizes GCV. MASS's lm.ridge doesn't choose a default lambda sequence for you. Look at this question which talks about good default choices for lambda. Also, I'd suggest using cv.glmnet with alpha = 0 (meaning ridge penalty) from glmnet package which will do this cross validation with some good lambdas for you. Use glmnetUtils package in addition if you like a formula interface like lm.ridge. See ?cv.glmnet after installing glmnet (and glmnetUtils, if you want).

Related Solutions

Solved – Poor predictions from lm.ridge

The results are comparable to those of a standard regression. I guess this is not a situation in which ridge regression improves the estimators: the number of variables is small compared to the number of observations, and there are no colinearity problems.

pairs(wine_train,gap=0)
d <- cbind(
  wine_test[, 12], pred.test, 
  predict( lm(quality~., wine_train), wine_test )
)
pairs(d, gap=0 )
cor(d)

The scale of your forecasts is wrong, however: you need center=TRUE.

Solved – Interpretation of ridge regularization in regression

Good questions!

Yes, this is exactly correct. You can see ridge penalty as one possible way to deal with multicollinearity problem that arises when many predictors are highly correlated. Introducing ridge penalty effectively lowers these correlations.
I think this is partly tradition, partly the fact that ridge regression formula as stated in your first equation follows from the following cost function: $$L=\| \mathbf y - \mathbf X \beta \|^2 + \lambda \|\beta\|^2.$$ If $\lambda=0$, the second term can be dropped, and minimizing the first term ("reconstruction error") leads to the standard OLS formula for $\beta$. Keeping the second term leads to the formula for $\beta_\mathrm{ridge}$. This cost function is mathematically very convenient to deal with, and this might be one of the reasons for preferring "non-normalized" lambda.
One possible way to normalize $\lambda$ is to scale it by the total variance $\mathrm{tr}(\mathbf X^\top \mathbf X)$, i.e. to use $\lambda \mathrm{tr}(\mathbf X^\top \mathbf X)$ instead of $\lambda$. This would not necessarily confine $\lambda$ to $[0,1]$, but would make it "dimensionless" and would probably result in optimal $\lambda$ being less then $1$ in all practical cases (NB: this is just a guess!).
"Attacking only small eigenvalues" does have a separate name and is called principal components regression. The connection between PCR and ridge regression is that in PCR you effectively have a "step penalty" cutting off all the eigenvalues after a certain number, whereas ridge regression applies a "soft penalty", penalizing all eigenvalues, with smaller ones getting penalized more. This is nicely explained in The Elements of Statistical Learning by Hastie et al. (freely available online), section 3.4.1. See also my answer in Relationship between ridge regression and PCA regression.
I have never seen this done, but note that you could consider a cost function in the form $$L=\| \mathbf y - \mathbf X \beta \|^2 + \lambda \|\beta-\beta_0\|^2.$$ This shrinks your $\beta$ not to zero, but to some other pre-defined value $\beta_0$. If one works out the math, you will arrive to the optimal $\beta$ given by $$\beta = (\mathbf X^\top \mathbf X + \lambda \mathbf I)^{-1} (\mathbf X^\top \mathbf y + \lambda \beta_0),$$ which perhaps can be seen as "regularizing cross-covariance"?

Best Answer

Related Solutions

Solved – Poor predictions from lm.ridge

Solved – Interpretation of ridge regularization in regression

Related Question