1.6.7.1 The data🔗ℹ

Three separable clusters: class 0 at low x₁, class 1 at high x₁, class 2 at high x₂:

> (define X '((1.0 1.0) (2.0 1.0) (1.0 2.0) (2.0 2.0)               (5.0 1.0) (6.0 1.0) (5.0 2.0) (6.0 2.0)               (3.0 5.0) (4.0 5.0) (3.0 6.0) (4.0 6.0)))

> (define y '(0 0 0 0 1 1 1 1 2 2 2 2))

1.6.7.2 Fitting🔗ℹ

> (define fit (multinomial-fit X y #:lambda 0.01))

> fit

#<glmnet:multinomial λ=0.01 dev=0.9814 nz=2/2>

> (coef fit)

'#(#(6.334774419914438 -1.2558971734686988 0.0)    #(-2.453188136398622 1.2550226289748376 0.0)    #(-3.881586283515815 0.0 2.0229335782284603))

coef gives one vector per class, the intercept first. Each class keeps only the predictor that sets it apart. Classes 0 and 1 split on x₁, with opposite signs; class 2 is picked out by x₂; every other coefficient is exactly 0.0.

Only differences between the classes’ linear predictors matter: adding the same vector to every class’s coefficients leaves the softmax unchanged. glmnet resolves that ambiguity with the penalty, which prefers the smallest coefficients, so the class-0 and class-1 weights on x₁ come out nearly equal and opposite rather than, say, zero and twice as large. The penalty does not touch the intercepts, which have the same freedom; as R does, the bindings centre them to sum to zero. To three decimals, the x₁ weights of classes 0 and 1, and the sum of the three intercepts:

> (match-define (vector (vector b₀ w₀ _) (vector b₁ w₁ _) (vector b₂ _ _))     (coef fit))

> (fmt (list w₀ w₁))

'("-1.256" "1.255")

> (fmt (+ b₀ b₁ b₂))

"0.000"

1.6.7.3 Predicting🔗ℹ

predict with #:type 'response returns one probability per class for each row, summing to 1, and with #:type 'class the most probable class. multinomial-predict-proba and multinomial-predict are the same two calls:

> (define new-points '((1.5 1.5) (5.5 1.5) (3.5 5.5) (3.5 1.5)))

> (predict fit new-points #:type 'response)

'((0.9885378101387693 0.006518498377706142 0.004943691483524595)   (0.006515125658992151 0.9885338196834262 0.0049510546575815845)   (0.004915825133083355 0.0049170874442380445 0.9901670874226786)   (0.4849885303770453 0.4851130682552687 0.029898401367686074))

> (predict fit new-points #:type 'class)

'(0 1 2 1)

> (equal? (predict fit X #:type 'class) y)

#t

The first three points sit at the cluster centres and are classified with confidence. The fourth lies between clusters 0 and 1; its probabilities show how close the call is.

1.6.7.4 Class labels🔗ℹ

The labels must be the integers 0 to K−1, with every class present. A gap is an error, not an empty class:

> (multinomial-fit X '(0 0 0 0 2 2 2 2 2 2 2 2) #:lambda 0.01)

multinomial-fit: class 1 has no observations; labels must

cover 0..2 contiguously

1.6.7.5 When to use it🔗ℹ

Use the multinomial family for a categorical outcome with more than two unordered levels: species, document topics, product categories. For two classes, use Binomial logistic regression (classification).