Description
Fits regressions that penalise large coefficients, trading a little bias for much lower variance; the lasso also shrinks weak predictors exactly to zero.
Instead of choosing a subset of predictors, regularised regression fits all of them while minimising the residual sum of squares plus a penalty on coefficient size: lambda times the sum of squared coefficients for ridge (L2), or of absolute coefficients for the lasso (L1). The intercept is not penalised. As lambda rises from zero (ordinary least squares) to infinity (null model), variance falls and bias rises; with many predictors relative to observations a well-chosen lambda can cut test error substantially. Because the lasso's constraint region has corners, it sets some coefficients exactly to zero, giving sparse, more interpretable models. Neither dominates: the lasso wins when few predictors matter, ridge when many have similar effects, and cross-validation decides. Predictors must be standardised because penalised estimates depend on scale. Logistic and Cox versions follow the same logic. Selected variables are one predictive set among many, not proof of which variables matter.
Attributes
- Discipline
- Statistics & Modelling
- Loss type
- Siloed
- Stage
- Discovery, Post-launch
- Data
- Quantitative
- Effort
- Days
Steps
- Standardise every predictor to standard deviation one.
- Choose a grid of lambda values spanning from almost no penalty to the null model.
- Fit ridge or lasso at each lambda, leaving the intercept unpenalised (use logistic or Cox versions for binary or time-to-event outcomes).
- Compute k-fold (typically 10-fold) or leave-one-out cross-validated error for each lambda.
- Select the lambda with lowest error, or the largest lambda within one standard error of it for a sparser model.
- Refit on all data at that lambda, plot coefficient paths, and report coefficients and predictive accuracy.
Sources
- An Introduction to Statistical Learning with Applications in Python (James et al., 2023)
- Mixed Models: Theory and Applications (Demidenko, 2004)
- Low Sample Size and Regression: A Monte Carlo Approach (Riveros Gavilanes, 2019)
- Bayesian Statistical Modelling (Congdon, 2006)
- Applied Predictive Modeling (Kuhn & Johnson, 2013)
- Pattern Recognition and Machine Learning (Bishop, 2006)
Also known as
- Ridge regression and the lasso (regularised regression)
- Shrinkage
- Regularisation
- L1 penalty
- L2 penalty
- Penalised regression
- glmnet
Categories
- Analytics & Behaviour
- Statistics & Modelling