1.6.5.1 Fitting🔗ℹ

The running fixture again, with the irrelevant x₃ = x₁²:

> (define X '((1.0 2.0  1.0)               (2.0 1.0  4.0)               (3.0 4.0  9.0)               (4.0 3.0 16.0)               (5.0 6.0 25.0)               (6.0 5.0 36.0)))

> (define y '(1.0 4.0 3.0 6.0 5.0 8.0))

elastic-net takes an explicit #:alpha. coef gives the intercept β₀ and then the coefficients β, here shown to three decimals:

> (define fit (elastic-net X y #:alpha 0.5 #:lambda 0.5))

> (displayln (coef-line (coef fit)))

β₀ = 1.650, β = (0.577 0.000 0.055)

The fit both shrinks its coefficients, as ridge does, and sets one exactly to zero, as the lasso does.

1.6.5.2 Between ridge and lasso🔗ℹ

At a shared λ, the number of coefficients the elastic net zeroes lies between ridge, which zeroes none, and the lasso, which zeroes the most. The count skips the intercept:

> (define (zeros model)     (match-define (vector _ β ...) (coef model))     (count zero? β))

> (map zeros (list (ridge X y #:lambda 0.5)                    fit                    (lasso X y #:lambda 0.5)))

'(0 1 2)

Sweeping α at that λ moves between the two:

> (for ([α (in-list '(0.0 0.25 0.5 0.75 1.0))])     (define model (elastic-net X y #:alpha α #:lambda 0.5))     (printf "α = ~a: ~a\n" α (coef-line (coef model))))

α = 0.0: β₀ = 1.932, β = (0.712 -0.335 0.082) α = 0.25: β₀ = 1.806, β = (0.649 -0.183 0.070) α = 0.5: β₀ = 1.650, β = (0.577 0.000 0.055) α = 0.75: β₀ = 1.682, β = (0.623 0.000 0.042) α = 1.0: β₀ = 1.425, β = (0.879 0.000 0.000)

1.6.5.3 Correlated predictors🔗ℹ

The case the elastic net was designed for is a group of predictors that carry the same signal. Take the extreme: duplicate x₁, so the first two columns are identical, and drop x₃:

> (define X-dup     (map (match-lambda [(list x₁ x₂ _) (list x₁ x₁ x₂)]) X))

> (define lasso-dup (lasso X-dup y #:lambda 0.1))

> (define enet-dup (elastic-net X-dup y #:alpha 0.5 #:lambda 0.1))

> (printf "lasso:       ~a\n" (coef-line (coef lasso-dup)))

lasso:       β₀ = 1.000, β = (1.658 0.000 -0.658)

> (printf "elastic net: ~a\n" (coef-line (coef enet-dup)))

elastic net: β₀ = 1.006, β = (0.870 0.855 -0.726)

The lasso puts all of x₁’s weight on one copy and leaves the other within rounding of zero: the two identical columns tie exactly at this λ, and which copy wins is an accident of the column order. The elastic net’s L2 term makes splitting the weight cheaper than concentrating it, so the two copies get almost equal coefficients. On real data the copies are merely correlated, but the effect is the same.

1.6.5.4 Choosing α and λ🔗ℹ

elnet-cv chooses λ for one α (Choosing λ by cross-validation). To choose α too, cross-validate each candidate on the same folds, so that the errors differ only because the models do. Take the 40 noisy observations X40 and y40 of Choosing λ by cross-validation:

> (define folds (random-fold-ids 40))

> (define alpha-cvs     (for/list ([α (in-list '(0.0 0.25 0.5 0.75 1.0))])       (cons α (elnet-cv X40 y40 #:alpha α #:fold-ids folds))))

> (for ([(α cv) (in-dict alpha-cvs)])     (printf "α = ~a: ~a\n" α cv))

α = 0.0: #<glmnet-cv:gaussian Measure: Mean-Squared Error     Lambda Index Measure      SE Nonzero min 0.2841   100  0.5117 0.08342       3 1se 0.4523    95  0.5889 0.09539       3> α = 0.25: #<glmnet-cv:gaussian Measure: Mean-Squared Error      Lambda Index Measure      SE Nonzero min 0.02448    67  0.3390 0.05769       2 1se 0.10847    51  0.3933 0.06771       3> α = 0.5: #<glmnet-cv:gaussian Measure: Mean-Squared Error      Lambda Index Measure      SE Nonzero min 0.02348    60  0.3379 0.05756       2 1se 0.12529    42  0.3902 0.06925       3> α = 0.75: #<glmnet-cv:gaussian Measure: Mean-Squared Error      Lambda Index Measure      SE Nonzero min 0.02069    57  0.3379 0.05732       2 1se 0.14596    36  0.3915 0.07030       3> α = 1.0: #<glmnet-cv:gaussian Measure: Mean-Squared Error      Lambda Index Measure      SE Nonzero min 0.01703    56  0.3378 0.05704       2 1se 0.14472    33  0.3890 0.06871       2>

Every α above 0 reaches nearly the same error, within one standard error of the others, so on these data the choice among them matters much less than the choice of λ. Ridge’s default path stops before its error bottoms out, so its best λ is the last one; a longer #:lambda grid closes the gap:

> (define longer-grid     (for/list ([i (in-range 160)])       (expt 10.0 (- 3.5 (* 0.05 i)))))

> (define longer-ridge-cv     (elnet-cv X40 y40 #:alpha 0.0 #:fold-ids folds #:lambda longer-grid))

> longer-ridge-cv

#<glmnet-cv:gaussian Measure: Mean-Squared Error      Lambda Index Measure      SE Nonzero min 0.00398   119  0.3361 0.05516       3 1se 0.07943    93  0.3856 0.06455       3>

1.6.5.5 When to use it🔗ℹ

The elastic net is a good default when you want selection but expect groups of correlated predictors, as with gene expression, sensor arrays or technical indicators. α is a second tuning parameter alongside λ; values around 0.5 are a common starting point, and cross-validation on shared folds chooses between candidates.