3.2 Transducers
(require rebellion/streaming/transducer) | |
package: rebellion |
A transducer is an object that can incrementally transform one (potentially infinite) sequence of elements into another sequence. Transducers are state machines; performing a transduction involves starting the transducer to get an initial state, then repeatedly updating that state by either consuming an element from the input sequence or by emitting an element to the output sequence. When the input sequence is exhausted, the transducer enters a half closed state where it may emit more output elements but it will never consume more input elements. When the transducer stops emitting elements, its finisher is called to clean up any resources held in the final transduction state. Optionally, a transducer may half close early, before the input sequence is fully consumed.
procedure
(transducer? v) → boolean?
v : any/c
procedure
seq : sequence? trans : transducer? red : reducer?
> (transduce (in-range 1 20) (filtering even?) (mapping number->immutable-string) #:into (join-into-string ", ")) "2, 4, 6, 8, 10, 12, 14, 16, 18"
procedure
(in-transduced seq trans) → sequence?
seq : sequence? trans : transducer?
procedure
(into-transduced trans ... #:into red) → reducer?
trans : transducer? red : reducer?
(define into-first-ten-letters (into-transduced (filtering char-alphabetic?) (mapping char-downcase) (taking 10) #:into into-string))
> (transduce "The quick brown fox" #:into into-first-ten-letters) "thequickbr"
3.2.1 Transforming Elements with Transducers
procedure
(mapping f) → transducer?
f : (-> any/c any/c)
procedure
(append-mapping f) → transducer?
f : (-> any/c sequence?)
This is similar to Java’s Stream.flatMap method.
> (transduce (set 'red 'green 'blue) (append-mapping symbol->immutable-string) #:into into-string) "bluegreenred"
procedure
(batching batch-reducer) → transducer?
batch-reducer : reducer?
If batch-reducer refuses to consume any elements and immediately terminates the reduction every time it’s started, then the returned transducer raises exn:fail:contract.
> (transduce (in-range 10) (batching (reducer-limit into-list 4)) #:into into-list) '((0 1 2 3) (4 5 6 7) (8 9))
procedure
(windowing window-size [ #:into window-reducer]) → transducer? window-size : exact-positive-integer? window-reducer : reducer? = into-list
If window-reducer is provided, it is used to reduce each window. By default each window’s elements are collected into a list. Only full windows are reduced and emitted downstream: if window-reducer finishes early for a window, before consuming window-size elements, the reduction result is not emitted downstream until after enough elements are consumed from the transduced sequence to verify that the result corresponds to a full window.
> (transduce "hello world" (windowing 3 #:into into-string) #:into into-list) '("hel" "ell" "llo" "lo " "o w" " wo" "wor" "orl" "rld")
> (transduce "hello world" (windowing 3 #:into (into-index-of #\w)) #:into into-list)
(list
#<absent>
#<absent>
#<absent>
#<absent>
(present 2)
(present 1)
(present 0)
#<absent>
#<absent>)
value
> (transduce "cat" enumerating #:into into-list)
(list
(enumerated #:element #\c #:position 0)
(enumerated #:element #\a #:position 1)
(enumerated #:element #\t #:position 2))
procedure
(enumerated? v) → boolean?
v : any/c
procedure
(enumerated #:element element #:position position) → enumerated? element : any/c position : natural?
procedure
(enumerated-element enum) → any/c
enum : enumerated?
procedure
(enumerated-position enum) → natural?
enum : enumerated?
3.2.2 Removing Elements with Transducers
procedure
(filtering pred) → transducer?
pred : predicate/c
procedure
(taking amount) → transducer?
amount : natural?
> (transduce "hello world" (taking 5) #:into into-string) "hello"
procedure
(taking-while pred) → transducer?
pred : predicate/c
> (transduce "The quick brown fox" (taking-while char-alphabetic?) #:into into-string) "The"
procedure
(taking-maxima [ comparator #:key key-function]) → transducer? comparator : comparator? = real<=> key-function : (-> any/c any/c) = values
procedure
(taking-minima [ comparator #:key key-function]) → transducer? comparator : comparator? = real<=> key-function : (-> any/c any/c) = values
> (transduce (list 2 512 3 5 512) (taking-maxima) #:into into-list) '(512 512)
Elements are compared using comparator. If key-function is provided, then elements are not compared directly. Instead, a key is extracted from each element using key-function and the keys of elements are compared instead of the elements themselves.
> (transduce (list "cat" "dog" "zebra" "horse") (taking-maxima string<=>) #:into into-list) '("zebra")
> (transduce (list "red" "yellow" "blue" "purple" "green") (taking-maxima #:key string-length) #:into into-list) '("yellow" "purple")
procedure
(taking-local-maxima [ comparator #:key key-function]) → transducer? comparator : comparator? = real<=> key-function : (-> any/c any/c) = values
procedure
(taking-local-minima [ comparator #:key key-function]) → transducer? comparator : comparator? = real<=> key-function : (-> any/c any/c) = values
> (transduce (list 1 8 4 7 1) (taking-local-maxima) #:into into-list) '(8 7)
Elements are compared using comparator. If key-function is provided, then elements are not compared directly. Instead, a key is extracted from each element using key-function and the keys of elements are compared instead of the elements themselves.
> (transduce (list "cat" "dog" "aardvark" "zebra" "horse") (taking-local-maxima string<=>) #:into into-list) '("dog" "zebra")
> (transduce (list "a" "long" "b" "longer" "longest" "c") (taking-local-maxima #:key string-length) #:into into-list) '("long" "longest")
procedure
(taking-duplicates [#:key key-function]) → transducer?
key-function : (-> any/c any/c) = values
> (transduce "hello world" (taking-duplicates) #:into into-string) "lol"
If key-function is provided, then it is used to extract a key from each element and only elements with duplicate keys are emitted downstream.
> (transduce (list "red" "yellow" "blue" "purple" "green") (taking-duplicates #:key string-length) #:into into-list) '("purple")
procedure
(dropping amount) → transducer?
amount : natural?
> (transduce "hello world" (dropping 5) #:into into-string) " world"
procedure
(dropping-while pred) → transducer?
pred : predicate/c
> (transduce "The quick brown fox" (dropping-while char-alphabetic?) #:into into-string) " quick brown fox"
procedure
(deduplicating [#:key key-function]) → transducer?
key-function : (-> any/c any/c) = values
> (transduce "Hello world!" (deduplicating) #:into into-string) "Helo wrd!"
If key-function is provided, is is applied to each element and uniqueness is based on the returned key value.
> (transduce (list "cat" "dog" "CAT" "HORSE" "horse") (deduplicating #:key string-foldcase) #:into into-list) '("cat" "dog" "HORSE")
procedure
(deduplicating-consecutive [#:key key-function]) → transducer?
key-function : (-> any/c any/c) = values
> (transduce "Mississippi" (deduplicating-consecutive) #:into into-string) "Misisipi"
If key-function is provided, it is applied to each element and uniqueness is based on the returned key value.
> (transduce (list "cat" "Cat" "CAT" "dog" "HORSE" "horse" "Dog") (deduplicating-consecutive #:key string-foldcase) #:into into-list) '("cat" "dog" "HORSE" "Dog")
3.2.3 Adding Elements with Transducers
procedure
(adding-between separator) → transducer?
separator : any/c
> (transduce "BEHOLD" (adding-between #\space) #:into into-string) "B E H O L D"
procedure
(splicing-between seperator-seq) → transducer?
seperator-seq : (sequence/c any/c)
> (transduce "BEHOLD" (splicing-between " ") #:into into-string) "B E H O L D"
Beware that if separator-seq is inserted into the input sequence multiple times, it is initiated each time it’s inserted. This can have unexpected behavior with sequences that have side effects, sequences that are nondeterministic, and sequences that observe concurrently-mutated state. This behavior can be suppressed by converting the sequence to a stream using sequence->stream.
(struct shoes-or-hat () ; This sequence randomly picks between a pair of shoes or a single hat #:property prop:sequence (λ (this) (random-ref (list (list 'shoe 'shoe) (list 'hat)))))
> (transduce (list 1 2 3 4 5 6) (splicing-between (shoes-or-hat)) #:into into-list) '(1 hat 2 hat 3 shoe shoe 4 hat 5 shoe shoe 6)
> (transduce (list 1 2 3 4 5 6) (splicing-between (sequence->stream (shoes-or-hat))) #:into into-list) '(1 hat 2 hat 3 hat 4 hat 5 hat 6)
3.2.4 Rearranging Elements with Transducers
procedure
(sorting [ comparator #:key key-function #:descending? descending?]) → transducer? comparator : comparator? = real<=> key-function : (-> any/c any/c) = values descending? : boolean? = #f
> (transduce (list 4 1 2 5 3) (sorting) #:into into-list) '(1 2 3 4 5)
> (transduce (list "the" "quick" "brown" "fox") (sorting string<=>) #:into into-list) '("brown" "fox" "quick" "the")
If key-function is provided, it is applied to each element and the result is tested with comparator rather than the element itself.
(define-record-type gem (kind weight)) (define gems (list (gem #:kind 'ruby #:weight 17) (gem #:kind 'sapphire #:weight 9) (gem #:kind 'emerald #:weight 13) (gem #:kind 'topaz #:weight 17)))
> (transduce gems (sorting #:key gem-weight) #:into into-list)
(list
(gem #:kind 'sapphire #:weight 9)
(gem #:kind 'emerald #:weight 13)
(gem #:kind 'ruby #:weight 17)
(gem #:kind 'topaz #:weight 17))
> (transduce gems (sorting #:key gem-weight #:descending? #t) #:into into-list)
(list
(gem #:kind 'ruby #:weight 17)
(gem #:kind 'topaz #:weight 17)
(gem #:kind 'emerald #:weight 13)
(gem #:kind 'sapphire #:weight 9))
procedure
(shuffling) → transducer?
> (transduce "hello my friend" (shuffling) #:into into-string) "nr hdelileyfm o"
procedure
(transposing #:into column-reducer [ #:ordered? preserve-column-order?]) → (transducer/c (sequence/c any/c) any/c) column-reducer : reducer? preserve-column-order? : boolean? = #true
> (transduce (list (list 1 2 3) (list 4 5 6) (list 7 8 9)) (transposing #:into into-list) #:into into-list) '((1 4 7) (2 5 8) (3 6 9))
> (transduce (list "abc" "xyz") (transposing #:into into-string) #:into into-list) '("ax" "by" "cz")
If column-reducer finishes early while building a column, that column may be sent downstream before additional rows are consumed. If preserve-column-order? is true, a column is only sent downstream once all columns to the left of it have been sent downstream. If preserve-column-order? is false, columns are sent downstream as soon as they’re ready. In either case, if all columns finish early the transducer stops consuming input rows from upstream.
(define into-list-while-positive (into-transduced (taking-while positive?) #:into into-list)) (define rows (list (list 1 2 3) (list 1 0 3) (list 0 0 3) (list 0 0 0)))
> (transduce rows (transposing #:into into-list-while-positive) #:into into-list) '((1 1) (2) (3 3 3))
> (transduce rows (transposing #:into into-list-while-positive #:ordered? #false) #:into into-list) '((2) (1 1) (3 3 3))
3.2.5 Transducer Composition
procedure
(transducer-pipe trans ...) → transducer?
trans : transducer?
> (transduce (in-naturals) (transducer-pipe (filtering even?) (taking 5)) #:into into-list) '(0 2 4 6 8)
procedure
(transducer-compose trans ...) → transducer?
trans : transducer?
> (transduce (in-naturals) (transducer-compose (filtering even?) (taking 5)) #:into into-list) '(0 2 4)
3.2.6 The Transduction Protocol
procedure
(make-transducer #:starter starter #:consumer consumer #:emitter emitter #:half-closer half-closer #:half-closed-emitter half-closed-emitter #:finisher finisher [ #:name name]) → transducer? starter : (-> transduction-state/c) consumer : (-> any/c transduction-state/c) emitter : (-> any/c emission?) half-closer : (-> any/c half-closed-transduction-state/c) half-closed-emitter : (-> any/c half-closed-emission?) finisher : (-> any/c void?) name : (or/c interned-symbol? #f) = #f
value
=
(variant/c #:consume any/c #:emit any/c #:half-closed-emit any/c #:finish any/c)
value
=
(variant/c #:half-closed-emit any/c #:finish any/c)
procedure
state : transduction-state/c value : any/c
procedure
em : emission?
procedure
(emission-value em) → any/c
em : emission?
procedure
v : any/c
procedure
(half-closed-emission state value) → half-closed-emission?
state : half-closed-transduction-state/c value : any/c
procedure
→ half-closed-transduction-state/c em : half-closed-emission?
procedure
(half-closed-emission-value em) → any/c
em : half-closed-emission?
3.2.7 Transducer Contracts
procedure
(transducer/c domain-contract range-contract) → contract? domain-contract : contract? range-contract : contract?
(define/contract squaring (transducer/c number? number?) (mapping sqr))
> (transduce (in-range 1 5) squaring #:into into-list) '(1 4 9 16)
> (transduce "abcd" squaring #:into into-list) squaring: contract violation
expected: number?
given: #\a
in: an element consumed by
(transducer/c number? number?)
contract from: (definition squaring)
blaming: top-level
(assuming the contract is correct)
at: eval:2:0
3.2.8 Transducer Chaperones and Impersonators
procedure
(transducer-impersonate transducer [ #:domain-guard domain-guard #:range-guard range-guard #:properties properties #:chaperone? chaperone?]) → transducer? transducer : transducer? domain-guard : (or/c (-> any/c any/c) #f) = #f range-guard : (or/c (-> any/c any/c) #f) = #f
properties : (hash/c impersonator-property? any/c #:immutable #t) = empty-hash
chaperone? : boolean? = (and (false? domain-guard) (false? range-guard))
If chaperone? is true, then the returned impersonator is a chaperone. In that case, both domain-guard and range-guard must always return values equal to whatever they’re given. Additionally, if a returned value is an impersonator, it must also be a chaperone.
(define (print-domain v) (printf "Consuming ~a\n" v) v) (define (print-range v) (printf "Emitting ~a\n" v) v) (define summing/printing (transducer-impersonate (folding + 0) #:domain-guard print-domain #:range-guard print-range))
> (transduce (in-range 1 6) summing/printing #:into into-list)
Consuming 1
Emitting 1
Consuming 2
Emitting 3
Consuming 3
Emitting 6
Consuming 4
Emitting 10
Consuming 5
Emitting 15
'(1 3 6 10 15)
3.2.9 Debugging Transducers
procedure
(peeking handler) → transducer?
handler : (-> any/c void?)
This function is intended mostly for debugging, as (peeking displayln) can be inserted into a transduction pipeline to investigate what elements are consumed in that part of the pipeline.
> (transduce (list 1 2 3 'apple 4 5 6) (peeking displayln) (taking-while number?) #:into into-sum)
1
2
3
apple
6
3.2.10 Testing Transducers
(require rebellion/streaming/transducer/testing) | |
package: rebellion |
This module provides utilities for testing transducers.
procedure
(observing-transduction-events subject)
→ (transducer/c any/c transduction-event?) subject : transducer?
The first emitted event is always start-event and the last one is always finish-event. A half-close-event is emitted if subject was half-closed, which means the upstream sequence ran out of elements while subject was trying to consume one.
> (transduce (in-naturals) (observing-transduction-events (taking 3)) #:into into-list)
(list
#<start-event>
(consume-event 0)
(emit-event 0)
(consume-event 1)
(emit-event 1)
(consume-event 2)
(half-closed-emit-event 2)
#<finish-event>)
> (transduce (list 1 2) (observing-transduction-events (taking 3)) #:into into-list)
(list
#<start-event>
(consume-event 1)
(emit-event 1)
(consume-event 2)
(emit-event 2)
#<half-close-event>
#<finish-event>)
procedure
(transduction-event? v) → boolean?
v : any/c
value
value
value
value
procedure
(consume-event v) → consume-event?
v : any/c
procedure
(consume-event-value event) → any/c
event : consume-event?
value
procedure
(emit-event v) → emit-event?
v : any/c
procedure
(emit-event-value event) → any/c
event : emit-event?
value
procedure
v : any/c
procedure
(half-closed-emit-event-value event) → any/c
event : half-closed-emit-event?