1.2.2 The penalty: α and λ
For the Gaussian family, glmnet solves
min_{β₀,β} 1/(2n) ‖y − β₀ − Xβ‖² + λ [ α‖β‖₁ + (1−α)/2 ‖β‖₂² ]
over the intercept β₀ and the coefficients β. The other families replace the squared-error term with their negative log-likelihood; the penalty is the same for all of them. It has two knobs:
#:alpha, α ∈ [0, 1], mixes the penalty. α = 0 is the ridge (L2) penalty, which shrinks every coefficient but zeroes none; α = 1 is the lasso (L1) penalty, which sets coefficients exactly to zero; values in between are the elastic net. elnet-fit and the other families’ fits default to 1.0.
#:lambda, λ ≥ 0, sets the penalty’s strength. Every single fit but ols requires it; to fit a whole sequence of values at once, use a regularization path (Regularization paths). λ = 0 is the unpenalized fit.
The four Gaussian models are one routine, elnet-fit, with different arguments: ols fixes λ = 0, ridge and lasso fix α, and elastic-net requires it:
Procedure |
| α |
| λ |
| (ignored) |
| 0 | |
| 0 |
| > 0 | |
| 1 |
| > 0 | |
| 0 < α < 1 |
| > 0 |
> (equal? (lasso X y #:lambda 0.1) (elnet-fit X y #:alpha 1.0 #:lambda 0.1)) #t
> (elastic-net X y #:lambda 0.1) application: required keyword argument not supplied
procedure: elastic-net
required keyword: #:alpha
arguments...:
(array #[#[1.0 2.0] #[2.0 1.0] #[3.0 4.0] #[4.0 3.0]
#[5.0 6.0]])
'(1.2 3.9 3.1 5.8 5.1)
#:lambda 0.1