Solved – Standardized LASSO in R still has intercept

glmnetinterceptlassorstandardization

I understand the need to standardize variables when performing LASSO in R (I'm specifically using cv.glmnet, and setting standardize=TRUE).

The resulting model, however, still fits an intercept. From what I understand, we center the data so that there's no intercept and the model has more freedom to fit coefficients (without the intercept taking up some of the $\Sigma{\beta_j}$), am I doing this correctly? Do I need to set intercept=FALSE?

Best Answer

The intercept should generally only be omitted if all the predictors and the response have mean=0 (in which case the intercept must necessarily be 0).

Setting standardize=TRUE, which is the default option for glmnet::glmnet, only standardizes the predictors. The function has another parameter to standardize the response, but by default this is set to standardize.response=FALSE. So you would want to estimate an intercept unless you have specified standardize.response=TRUE.

Related Solutions

Standardized vs Centered Variables – Key Differences in Regression

Yes
Yes
You standardize variables to compare the importance of independent variables in determining the outcome variables.
You may want to center a variable when you use an interaction term--its effect will be meaningfully interpretable if the minimum value of one of the interacted variables is not zero.
If you are regressing different outcome variables (with different scales) on the same set of independent variables, you can meaningfully compare the estimated coefficients.
Yes
Yes.
Yes.
Yes, but bare in mind point 4.

Lasso Centering and Standardization – How to Perform Lasso Centering and Standardization with R

If you use glmnet, the scaling is performed by the package. You don't need to worry about scaling the test set because the "coefficients are always returned on the original scale".

By default:

glmnet(x, y, [...]
standardize = TRUE,
intercept = TRUE,
standardize.response = FALSE [...])

As for the standardization of the response, it should not change the performance of your model after cross validating over $\lambda$ so you can set standardize.response = FALSE

Indeed the LASSO solves

$$ \min_\beta\; \| Y - X\beta \|^2_2 + \lambda \|\beta\|_1 $$

Scaling $Y$ by a factor $\alpha > 0$, the problem becomes

$$ \min_\beta\; \| \alpha Y - X\beta \|^2_2 + \lambda \|\beta\|_1 $$

which is equivalent to

$$ \min_\beta\; \alpha \| Y-X\beta/\alpha \|^2_2 + \lambda \|\beta\|_1 $$

$$ \min_\beta\; \| Y - X\beta/\alpha \|^2_2 + \lambda \|\beta/\alpha\|_1 $$

So it has the same value of $\lambda$

Best Answer

Related Solutions

Standardized vs Centered Variables – Key Differences in Regression

Lasso Centering and Standardization – How to Perform Lasso Centering and Standardization with R

Related Question