1.6.2.1 The data🔗ℹ

The response is an exact linear function of two predictors, y = 1 + 2x₁ − x₂, with no error term, so least squares should recover the intercept 1.0 and the coefficients 2.0 and -1.0. The design matrix is a list of rows, and the response the matching list:

> (define X '((1.0 2.0)               (2.0 1.0)               (3.0 4.0)               (4.0 3.0)               (5.0 6.0)))

> (define y '(1.0 4.0 3.0 6.0 5.0))

1.6.2.2 Fitting🔗ℹ

ols is elnet-fit with #:lambda 0.0. The fit prints as a summary; coef gives the intercept β₀ and then the coefficients β, here shown to five decimals, and deviance-ratio the R², here to ten:

> (define fit (ols X y))

> fit

#<glmnet:gaussian λ=0 dev=1.0000 nz=2/2>

> (displayln (coef-line (coef fit) 5))

β₀ = 1.00001, β = (1.99998 -0.99999)

> (fmt (deviance-ratio fit) 10)

"0.9999999999"

The intercept and coefficients match the generating equation to about five decimal places, and R² is 1 to within 1e-10. They are not exact because coordinate descent stops once a pass changes the objective by less than #:thresh. A tighter threshold costs more passes and gets closer:

> (define tight (ols X y #:thresh 1e-14))

> (displayln (coef-line (coef tight) 5))

β₀ = 1.00000, β = (2.00000 -1.00000)

> (map elnet-result-num-passes (list fit tight))

'(29 41)

1.6.2.3 Fitted values🔗ℹ

A fitted value is the intercept plus the dot product of a row with the coefficients. predict computes it for each row, here the training rows, whose fitted values reproduce y to the same five decimal places:

> (fmt (predict fit X) 5)

'("1.00002" "3.99999" "3.00001" "5.99998" "5.00000")

elnet-predict is the same computation under the Gaussian family’s own name. On new rows the fit extrapolates the plane it found:

> (fmt (elnet-predict fit '((6.0 5.0) (0.0 0.0))) 5)

'("7.99997" "1.00001")

1.6.2.4 When to use it🔗ℹ

Least squares is the baseline the penalized models are measured against. It has no answer to overfitting or to collinear predictors: when two predictors carry the same information, or when there are more predictors than rows, the least-squares coefficients are not unique. Ridge regression (L2, α = 0) handles that case, and Lasso (L1, α = 1) handles the case where most predictors are noise.