1.6.6.1 The data🔗ℹ

Class 1 has high x₁ and low x₂, class 0 the reverse, and x₃ is noise, identically distributed in both classes:

> (define X '((1.0 5.0 2.0)               (2.0 6.0 1.0)               (2.0 5.0 3.0)               (1.0 4.0 1.0)               (3.0 6.0 2.0)               (2.0 4.0 2.0)               (6.0 2.0 2.0)               (5.0 1.0 1.0)               (6.0 1.0 3.0)               (5.0 2.0 1.0)               (4.0 1.0 2.0)               (6.0 3.0 2.0)))

> (define y '(0 0 0 0 0 0 1 1 1 1 1 1))

1.6.6.2 Fitting🔗ℹ

> (define fit (logistic-fit X y #:lambda 0.04))

> fit

#<glmnet:binomial λ=0.04 dev=0.9266 nz=2/3>

> (coef fit)

'#(-0.7794670198037468 0.9979652738654424 -0.8362224983591743 0.0)

coef lists the intercept and then the coefficients, all on the log-odds scale. x₁ raises the odds of class 1 and x₂ lowers them; the noise predictor is exactly 0.0. Exponentiating a coefficient gives an odds ratio, the factor by which one more unit of the predictor multiplies the odds of class 1:

> (match-define (vector _ β ...) (coef fit))

> (map exp β)

'(2.7127564925605556 0.43334439467246694 1.0)

deviance-ratio is the fraction of null deviance explained, the logistic analogue of R²:

> (deviance-ratio fit)

0.9266146849101543

1.6.6.3 Predicting🔗ℹ

predict with #:type 'response returns the class-1 probability for each row, and with #:type 'class the label:

> (predict fit X #:type 'response)

'(0.018658581830764638   0.02186262725172905   0.04904870960935964   0.04203165380714618   0.057167332847504866   0.10636446519086237   0.9716915927629215   0.9668863972296748   0.9875327213177286   0.9267573826785769   0.9149921263903765   0.9370064088370706)

> (equal? (predict fit X #:type 'class) y)

#t

Every training point is classified correctly. logistic-predict-proba is the first call. logistic-predict is the second with a #:threshold on the probability, 0.5 unless given; for a point near the boundary, where it sits decides the label:

> (define borderline '((3.5 3.5 2.0)))

> (predict fit borderline #:type 'response)

'(0.44685962480034513)

> (logistic-predict fit borderline)

'(0)

> (logistic-predict fit borderline #:threshold 0.4)

'(1)

Lowering the threshold trades false negatives for false positives, which is what you want when missing a class-1 case costs more than a false alarm.

1.6.6.4 Separation and the penalty🔗ℹ

These two classes are linearly separable: a line in the (x₁, x₂) plane splits them perfectly. Unpenalized logistic regression has no finite solution on such data, because making the coefficients larger always fits a little better. The penalty is what keeps the fit well defined, as logistic-path shows by fitting a list of λ in one call, largest first (Regularization paths). in-path walks it, giving each λ with its coefficients, the intercept β₀ first and then β, and deviance-ratio gives the deviance ratio at each λ, all shown to three decimals:

> (define path (logistic-path X y #:lambda '(0.0 0.001 0.01 0.04 0.1 0.3)))

> (for ([(λ coefs) (in-path path)]         [dev (in-vector (deviance-ratio path))])     (printf "λ = ~a: ~a, dev-ratio = ~a\n" λ (coef-line coefs) (fmt dev)))

λ = 0.3: β₀ = -0.355, β = (0.246 -0.158 0.000), dev-ratio = 0.412 λ = 0.1: β₀ = -0.566, β = (0.669 -0.547 0.000), dev-ratio = 0.817 λ = 0.04: β₀ = -0.780, β = (0.998 -0.836 0.000), dev-ratio = 0.927 λ = 0.01: β₀ = -1.160, β = (1.503 -1.267 0.000), dev-ratio = 0.981 λ = 0.001: β₀ = -1.902, β = (2.386 -2.008 0.000), dev-ratio = 0.998 λ = 0.0: β₀ = -14.039, β = (7.975 -5.369 1.842), dev-ratio = 1.000

At λ = 0 the solver returns large coefficients, the noise predictor among them, with a deviance ratio of essentially 1: the model has memorized the training labels. With no finite optimum, which large coefficients it returns depends on where it starts. The path starts from its fit at λ = 0.001; a single fit starts from zero and returns different ones:

> (define unpenalized (logistic-fit X y #:lambda 0.0))

> (displayln (coef-line (coef unpenalized)))

β₀ = 1.065, β = (6.677 -5.589 -3.542)

Even a small penalty removes the noise and keeps the coefficients finite. On harder data, glmnet can instead fail with an error saying a class probability collapsed; a larger #:lambda fixes that too.

1.6.6.5 When to use it🔗ℹ

Use the binomial family for any yes/no outcome: default or not, spam or not, malignant or benign. The lasso logistic is a standard tool for building a sparse, interpretable classifier from many candidate features. For more than two classes, see Multinomial classification (K classes).