1.1.2 A First Regression🔗ℹ

A fit is the result of solving an optimization problem on a dataset. For a Gaussian model, the objective described in the R glmnet vignette is:

min_β₀,β 1/(2n) ∑ᵢ₌₁ⁿ (yᵢ − β₀ − xᵢᵀβ)² + λ [α‖β‖₁ + (1−α)/2 ‖β‖₂²]

The solution gives the intercept β₀ and the predictor coefficients β. Each of the n observations has predictors xᵢ and a response yᵢ. The penalty’s strength is λ ≥ 0, and α ∈ [0, 1] mixes the lasso and ridge penalties. Ordinary least squares (OLS) sets λ = 0; see The penalty: α and λ for the other choices.

R’s mtcars dataset measures 32 cars. Load it as a Polars dataframe:

> (require datasets)

> (define cars (load-dataset 'mtcars #:format 'polars))

> cars

shape: (32, 12) ┌───────────────────┬──────┬─────┬───────┬───┬─────┬─────┬──────┬──────┐ │ model             ┆ mpg  ┆ cyl ┆ disp  ┆ … ┆ vs  ┆ am  ┆ gear ┆ carb │ │ ---               ┆ ---  ┆ --- ┆ ---   ┆   ┆ --- ┆ --- ┆ ---  ┆ ---  │ │ str               ┆ f64  ┆ i64 ┆ f64   ┆   ┆ i64 ┆ i64 ┆ i64  ┆ i64  │ ╞═══════════════════╪══════╪═════╪═══════╪═══╪═════╪═════╪══════╪══════╡ │ Mazda RX4         ┆ 21.0 ┆ 6   ┆ 160.0 ┆ … ┆ 0   ┆ 1   ┆ 4    ┆ 4    │ │ Mazda RX4 Wag     ┆ 21.0 ┆ 6   ┆ 160.0 ┆ … ┆ 0   ┆ 1   ┆ 4    ┆ 4    │ │ Datsun 710        ┆ 22.8 ┆ 4   ┆ 108.0 ┆ … ┆ 1   ┆ 1   ┆ 4    ┆ 1    │ │ Hornet 4 Drive    ┆ 21.4 ┆ 6   ┆ 258.0 ┆ … ┆ 1   ┆ 0   ┆ 3    ┆ 1    │ │ Hornet Sportabout ┆ 18.7 ┆ 8   ┆ 360.0 ┆ … ┆ 0   ┆ 0   ┆ 3    ┆ 2    │ │ …                 ┆ …    ┆ …   ┆ …     ┆ … ┆ …   ┆ …   ┆ …    ┆ …    │ │ Lotus Europa      ┆ 30.4 ┆ 4   ┆ 95.1  ┆ … ┆ 1   ┆ 1   ┆ 5    ┆ 2    │ │ Ford Pantera L    ┆ 15.8 ┆ 8   ┆ 351.0 ┆ … ┆ 0   ┆ 1   ┆ 5    ┆ 4    │ │ Ferrari Dino      ┆ 19.7 ┆ 6   ┆ 145.0 ┆ … ┆ 0   ┆ 1   ┆ 5    ┆ 6    │ │ Maserati Bora     ┆ 15.0 ┆ 8   ┆ 301.0 ┆ … ┆ 0   ┆ 1   ┆ 5    ┆ 8    │ │ Volvo 142E        ┆ 21.4 ┆ 4   ┆ 121.0 ┆ … ┆ 1   ┆ 1   ┆ 4    ┆ 2    │ └───────────────────┴──────┴─────┴───────┴───┴─────┴─────┴──────┴──────┘

Predict fuel economy (mpg, miles per US gallon) from weight (wt, in thousands of pounds) and horsepower (hp). ols fits without a penalty; name the response column, then the predictor columns:

> (define fit (ols cars "mpg" #:predictors '("wt" "hp")))

> fit

#<glmnet:gaussian λ=0 dev=0.8268 nz=2/2>

> (coef fit)

'#(37.22729275591213 -3.8778430581971657 -0.03177283120189043)

The fit prints its family, λ, the fraction of variance explained (R²) and how many coefficients are nonzero. coef gives the intercept, then one coefficient per predictor, in the order named. Name them with match-define:

> (match-define (vector β₀ β-wt β-hp) (coef fit))

> (~r (* 100 β-hp) #:precision 2)

"-3.18"

> (deviance-ratio fit)

0.8267854518805285

(* 100 β-hp) is the change in predicted fuel economy for 100 more horsepower at the same weight. deviance-ratio is R².

A penalty shrinks the coefficients. lasso applies the L1 penalty, which can set coefficients to zero; at λ = 4 it drops horsepower:

> (define penalized-fit (lasso cars "mpg" #:predictors '("wt" "hp") #:lambda 4.0))

> (coef penalized-fit)

'#(23.92234052081157 -1.1909909148532365 0.0)

A regularization path fits many values of λ at once. elnet-path uses the lasso penalty by default. Its summary shows, for each λ, the number of nonzero coefficients and the percentage of deviance explained:

> (elnet-path cars "mpg" #:predictors '("wt" "hp") #:nlambda 12)

#<glmnet-path:gaussian   Df  %Dev Lambda 1  0  0.00 5.1470 2  2 65.67 2.2280 3  2 79.49 0.9645 4  2 82.08 0.4175 5  2 82.57 0.1807 6  2 82.66 0.0782 7  2 82.67 0.0339 8  2 82.68 0.0147 9  2 82.68 0.0063>