1.2.5 Regularization paths🔗ℹ

The right λ is rarely known in advance. A regularization path fits a whole decreasing sequence of λ in one call, starting each fit from the solution before it, which is much cheaper than fitting each value separately. It is how R’s glmnet(x, y) is normally used. Every family has a path fitter: elnet-path, logistic-path, multinomial-path, cox-path, poisson-path and mgaussian-path. They take the same keywords as the single fits, except that #:lambda is optional and, when given, a list.

Without #:lambda, glmnet chooses the sequence the way R does: #:nlambda values (default 100) from λ_max, the smallest λ at which every coefficient is zero, down to #:lambda-min-ratio times λ_max (default 0.01 when there are fewer observations than predictors, otherwise 0.0001). A path prints as R prints one: a row for each fitted λ, with the number of nonzero coefficients (Df) and the percentage of the null deviance explained (%Dev):

> (define path (elnet-path X y #:nlambda 12))

> path

#<glmnet-path:gaussian   Df  %Dev  Lambda 1  0  0.00 1.37200 2  1 59.05 0.59380 3  2 71.29 0.25700 4  2 94.60 0.11130 5  2 98.97 0.04817 6  2 99.79 0.02085 7  2 99.94 0.00902>

At the first λ every coefficient is zero, and the predictors enter as the penalty relaxes, here x₁ first and then x₂. The path has fewer values than asked for because glmnet, like R, stops an automatic sequence once another λ would barely change the fit. For the Gaussian family that is when R² gains less than 1e-5 of its own value from one λ to the next, or passes 0.999; the reference lists each family’s rule (Regularization paths).

in-path walks a path, giving each fitted λ with its coefficients as coef gives them. With #:lambda, the path fits those values, largest first:

> (define user-path (elnet-path X y #:lambda '(0.01 0.5 0.1)))

> (for ([(λ β) (in-path user-path)])     (printf "λ = ~a: ~a\n" λ β))

λ = 0.5: #(1.970660171779822 0.6164466094067259 0.0) λ = 0.1: #(1.228876781353367 1.4306354476222993 -0.531494726318833) λ = 0.01: #(1.0966635809629564 1.788234241387329 -0.8254269703515447)

Each point agrees with the single fit at that λ to within the solver’s tolerance. The early-stopping rule does not apply to a user sequence, but any path can still end early, for example where glmnet does not converge within #:max-iters passes at one of its values. It then keeps the values before that one and logs a warning (Convergence and errors):

> (elnet-path X y #:lambda '(0.5 0.1 0.01) #:max-iters 1)

#<glmnet-path:gaussian   Df  %Dev Lambda 1  1 63.01    0.5>

predict and coef evaluate a path at any λ (Predictions and coefficients). Cross-validation chooses among the values on a path (Choosing λ by cross-validation). plot-coefficient-path, from glmnet/plot, plots a path’s coefficients against λ, as R’s plot does (Coefficient paths).