How do ridge, LASSO and elasticnet regularization methods compare? What are their respective advantages and disadvantages? Any good technical paper, or lecture notes would be appreciated as well.
References – Comprehensive Guide to Ridge, Lasso, and Elastic Net Regularization
elastic netlassoreferencesregularizationridge regression
Related Question
- Solved – Grid fineness and overfitting when tuning $\lambda$ in LASSO, ridge, elastic net
- Solved – elastic net regularization, and how does it solve the drawbacks of Ridge ($L^2$) and Lasso ($L^1$)
- Bridge Penalty vs Elastic Net – Comparison and Applications in Regularization
- Regularization – Why Lasso or ElasticNet Perform Better Than Ridge with Correlated Features
- Elastic Net vs Lasso – Exploring Disadvantages of Elastic Net
Best Answer
In The Elements of Statistical Learning book, Hastie et al. provide a very insightful and thorough comparison of these shrinkage techniques. The book is available online (pdf). The comparison is done in section 3.4.3, page 69.
The main difference between Lasso and Ridge is the penalty term they use. Ridge uses $L_2$ penalty term which limits the size of the coefficient vector. Lasso uses $L_1$ penalty which imposes sparsity among the coefficients and thus, makes the fitted model more interpretable. Elasticnet is introduced as a compromise between these two techniques, and has a penalty which is a mix of $L_1$ and $L_2$ norms.