chk: a minimal tester
| (require chk) | package: chk-lib |
This package defines a minimal testing library with a simple implementation, sensible defaults, useful flexibilty, and beautiful failure reporting. It is independent of all other Racket testing libraries, but it does cooperate with raco test and raco cover for displaying summary information.
syntax
(chk . t)
| t | = | (test ...) | ||
| test | = | strict-test | ||
| | | (strict-test) | |||
| | | (ex ex) | |||
| | | (ex) | |||
| strict-test | = | (#:t ex) | ||
| | | (#:? ex ex) | |||
| | | (#:= ex ex) | |||
| | | (#:eq ex ex ex) | |||
| | | (#:! test) | |||
| | | (#:x ex ex) | |||
| | | (#:do e) | |||
| ex | = | (#:stx e e) | ||
| | | (#:src e e) | |||
| | | e |
The syntax of t determines the test executed.
1 Single Value Tests
The simplest form is a #:t e test, where chk ensures that e evaluates to exactly one value that is not #f.
> (chk #:t 5) > (chk #:t #f)
FAILURE
kind : "predicate"
predicate : #<procedure:identity>
actual : #f
from
eval:2:0: #f
value : #f
predicate : "not false"
> (chk #:t (values 1 2))
FAILURE
kind : "predicate"
predicate : #<procedure:identity>
actual : (values 1 2)
from
eval:3:0: (values 1 2)
part : "how many values"
actual : 2
expected : 1
> (chk #:t (/ 1 0))
FAILURE
kind : "predicate"
predicate : #<procedure:identity>
actual : /: division by zero
context...:
/home/root/user/.local/share/racket/8.16.0.1/pkgs/chk-lib/chk/main.rkt:119:4
body of top-level
/home/root/racket/collects/racket/private/more-scheme.rkt:148:2: call-with-break-parameterization
/home/root/racket/share/pkgs/sandbox-lib/racket/sandbox.rkt:921:7
/home/root/racket/share/pkgs/sandbox-lib/racket/sandbox.rkt:891:2: user-process
from
eval:4:0: (/ 1 0)
predicate : "not exception"
This may be abbreviated by removing the #:t.
> (chk 5) > (chk #f)
FAILURE
kind : "predicate"
predicate : #<procedure:identity>
actual : #f
from
eval:6:0: #f
value : #f
predicate : "not false"
A #:t e test is a special case of a #:? p e test, where chk ensures that e evaluates to exactly one value and that the predicate p does not return #f on it. This form is robust to errors in p, which must evaluate to exactly one value that is a procedure of one argument.
> (chk #:? even? 0) > (chk #:? even? 1)
FAILURE
kind : "predicate"
predicate : #<procedure:even?>
actual : 1
from
eval:8:0: 1
value : 1
predicate : "not false"
> (chk #:? even? (values 1 2))
FAILURE
kind : "predicate"
predicate : #<procedure:even?>
actual : (values 1 2)
from
eval:9:0: (values 1 2)
part : "how many values"
actual : 2
expected : 1
> (chk #:? even? "two")
FAILURE
kind : "predicate"
predicate : #<procedure:even?>
actual : "two"
from
eval:10:0: "two"
value : "two"
part : "evaluating predicate"
exn : even?: contract violation
expected: integer?
given: "two"
context...:
/home/root/user/.local/share/racket/8.16.0.1/pkgs/chk-lib/chk/main.rkt:350:47
/home/root/racket/collects/racket/private/more-scheme.rkt:148:2: call-with-break-parameterization
/home/root/racket/share/pkgs/sandbox-lib/racket/sandbox.rkt:921:7
/home/root/racket/share/pkgs/sandbox-lib/racket/sandbox.rkt:891:2: user-process
> (chk #:? #f 0)
FAILURE
kind : "valid predicate expected"
actual : #f
from
eval:11:0: #f
predicate : "procedure of 1 arg"
> (chk #:? (values 3 4) 0)
FAILURE
kind : "valid predicate expected"
actual : (values 3 4)
from
eval:12:0: (values 3 4)
part : "how many values"
actual : 2
expected : 1
> (chk #:? list-ref 0)
FAILURE
kind : "valid predicate expected"
actual : #<procedure:list-ref>
from
eval:13:0: list-ref
predicate : "procedure of 1 arg"
> (chk #:? (error 'boo) 0)
FAILURE
kind : "valid predicate expected"
actual : error: boo
context...:
body of top-level
/home/root/racket/collects/racket/private/more-scheme.rkt:148:2: call-with-break-parameterization
/home/root/racket/share/pkgs/sandbox-lib/racket/sandbox.rkt:921:7
/home/root/racket/share/pkgs/sandbox-lib/racket/sandbox.rkt:891:2: user-process
from
eval:14:0: (error 'boo)
predicate : "not exception"
2 Expected Value Tests
An #:= actual expected test compares an actual evaluation with an expected evaluation. If expected raises an exception, then actual must also, with the same exn-message. Otherwise, the two expression must return the same number of values, which must be equal?.
> (chk #:= 1 1) > (chk #:= 1 "two")
FAILURE
kind : "compare"
compare : #<procedure:equal?>
actual : 1
from
eval:16:0: 1
expected : "two"
from
eval:16:0: "two"
part : "value 0"
actual : 1
expected : "two"
> (chk #:= (/ 1 0) (error '/ "division by zero")) > (chk #:= (/ 1 0) +inf.0)
FAILURE
kind : "compare"
compare : #<procedure:equal?>
actual : /: division by zero
context...:
/home/root/user/.local/share/racket/8.16.0.1/pkgs/chk-lib/chk/main.rkt:119:4
body of top-level
/home/root/racket/collects/racket/private/more-scheme.rkt:148:2: call-with-break-parameterization
/home/root/racket/share/pkgs/sandbox-lib/racket/sandbox.rkt:921:7
/home/root/racket/share/pkgs/sandbox-lib/racket/sandbox.rkt:891:2: user-process
from
eval:18:0: (/ 1 0)
expected : +inf.0
from
eval:18:0: +inf.0
predicate : "values"
> (chk #:= (/ 1 0) (error '/ "divided by zero"))
FAILURE
kind : "compare"
compare : #<procedure:equal?>
actual : /: division by zero
context...:
/home/root/user/.local/share/racket/8.16.0.1/pkgs/chk-lib/chk/main.rkt:119:4
body of top-level
/home/root/racket/collects/racket/private/more-scheme.rkt:148:2: call-with-break-parameterization
/home/root/racket/share/pkgs/sandbox-lib/racket/sandbox.rkt:921:7
/home/root/racket/share/pkgs/sandbox-lib/racket/sandbox.rkt:891:2: user-process
from
eval:19:0: (/ 1 0)
expected : /: divided by zero
context...:
body of top-level
/home/root/racket/collects/racket/private/more-scheme.rkt:148:2: call-with-break-parameterization
/home/root/racket/share/pkgs/sandbox-lib/racket/sandbox.rkt:921:7
/home/root/racket/share/pkgs/sandbox-lib/racket/sandbox.rkt:891:2: user-process
from
eval:19:0: (error '/ "divided by zero")
part : "exn-message"
actual : "/: division by zero"
expected : "/: divided by zero"
> (chk #:= (values 1 2) (values 1 2)) > (chk #:= (values 1 2) 1)
FAILURE
kind : "compare"
compare : #<procedure:equal?>
actual : (values 1 2)
from
eval:21:0: (values 1 2)
expected : 1
from
eval:21:0: 1
part : "how many values"
actual : 2
expected : 1
> (chk #:= 1 (values 1 2))
FAILURE
kind : "compare"
compare : #<procedure:equal?>
actual : 1
from
eval:22:0: 1
expected : (values 1 2)
from
eval:22:0: (values 1 2)
part : "how many values"
actual : 1
expected : 2
> (chk #:= (values 1 2) (values "one" "two"))
FAILURE
kind : "compare"
compare : #<procedure:equal?>
actual : (values 1 2)
from
eval:23:0: (values 1 2)
expected : (values "one" "two")
from
eval:23:0: (values "one" "two")
part : "value 0"
actual : 1
expected : "one"
An #:= actual expected test may be abbreviated as actual expected.
> (chk 1 1) > (chk 1 "two")
FAILURE
kind : "compare"
compare : #<procedure:equal?>
actual : 1
from
eval:25:0: 1
expected : "two"
from
eval:25:0: "two"
part : "value 0"
actual : 1
expected : "two"
An #:= actual expected test is a special case of a #:eq equal actual expected test where equal is equal?. equal must evaluate to exactly one value, which is a procedure of two arguments. If equal does not evaluate as expected, then the test is considered a failure.
> (chk #:eq = 1 1) > (chk #:eq eq? (cons 1 2) (cons 1 2))
FAILURE
kind : "compare"
compare : #<procedure:eq?>
actual : (1 . 2)
from
eval:27:0: (cons 1 2)
expected : (1 . 2)
from
eval:27:0: (cons 1 2)
part : "value 0"
actual : (1 . 2)
expected : (1 . 2)
> (chk #:eq (/ 1 0) 1 2)
FAILURE
kind : "valid compare expected"
actual : /: division by zero
context...:
/home/root/user/.local/share/racket/8.16.0.1/pkgs/chk-lib/chk/main.rkt:119:4
body of top-level
/home/root/racket/collects/racket/private/more-scheme.rkt:148:2: call-with-break-parameterization
/home/root/racket/share/pkgs/sandbox-lib/racket/sandbox.rkt:921:7
/home/root/racket/share/pkgs/sandbox-lib/racket/sandbox.rkt:891:2: user-process
from
eval:28:0: (/ 1 0)
predicate : "not exception"
> (chk #:eq #f 1 1)
FAILURE
kind : "valid compare expected"
actual : #f
from
eval:29:0: #f
predicate : "procedure of 2 args"
> (chk #:eq add1 1 1)
FAILURE
kind : "valid compare expected"
actual : #<procedure:add1>
from
eval:30:0: add1
predicate : "procedure of 2 args"
> (chk #:eq + "one" "two")
FAILURE
kind : "compare"
compare : #<procedure:+>
actual : "one"
from
eval:31:0: "one"
expected : "two"
from
eval:31:0: "two"
part : "value 0"
actual : "one"
expected : "two"
part : "evaluating check"
exn : +: contract violation
expected: number?
given: "one"
context...:
/home/root/user/.local/share/racket/8.16.0.1/pkgs/chk-lib/chk/main.rkt:126:0: *check=
/home/root/user/.local/share/racket/8.16.0.1/pkgs/chk-lib/chk/main.rkt:350:47
/home/root/racket/collects/racket/private/more-scheme.rkt:148:2: call-with-break-parameterization
/home/root/racket/share/pkgs/sandbox-lib/racket/sandbox.rkt:921:7
/home/root/racket/share/pkgs/sandbox-lib/racket/sandbox.rkt:891:2: user-process
3 Expected Exception Tests
An #:x actual exn-pred test is form when you expect actual to evaluate to an exception, but cannot (or do not want to) predict the exact error message. exn-pred must evaluate to exactly one value. If that value is a string, then it must be appear in the message of the exception that actual raises. If that value is a regular expression, then it must match the message of exception that actual raises. If the value is not a string or regular expression, it must be a procedure of one argument that does not return #f on the exception that actual raises.
> (chk #:x (/ 1 0) "division") > (chk #:x (/ 1 0) "divided")
FAILURE
kind : "exception"
actual : /: division by zero
context...:
/home/root/user/.local/share/racket/8.16.0.1/pkgs/chk-lib/chk/main.rkt:119:4
body of top-level
/home/root/racket/collects/racket/private/more-scheme.rkt:148:2: call-with-break-parameterization
/home/root/racket/share/pkgs/sandbox-lib/racket/sandbox.rkt:921:7
/home/root/racket/share/pkgs/sandbox-lib/racket/sandbox.rkt:891:2: user-process
from
eval:33:0: (/ 1 0)
part : "exn-message"
actual : "/: division by zero"
expected : #rx"divided"
> (chk #:x (/ 1 0) #rx"di.ision") > (chk #:x (/ 1 0) #rx"(one|two)")
FAILURE
kind : "exception"
actual : /: division by zero
context...:
/home/root/user/.local/share/racket/8.16.0.1/pkgs/chk-lib/chk/main.rkt:119:4
body of top-level
/home/root/racket/collects/racket/private/more-scheme.rkt:148:2: call-with-break-parameterization
/home/root/racket/share/pkgs/sandbox-lib/racket/sandbox.rkt:921:7
/home/root/racket/share/pkgs/sandbox-lib/racket/sandbox.rkt:891:2: user-process
from
eval:35:0: (/ 1 0)
part : "exn-message"
actual : "/: division by zero"
expected : #rx"(one|two)"
> (chk #:x (/ 1 0) exn:fail?) > (chk #:x (/ 1 0) exn:fail:syntax?)
FAILURE
kind : "exception"
actual : /: division by zero
context...:
/home/root/user/.local/share/racket/8.16.0.1/pkgs/chk-lib/chk/main.rkt:119:4
body of top-level
/home/root/racket/collects/racket/private/more-scheme.rkt:148:2: call-with-break-parameterization
/home/root/racket/share/pkgs/sandbox-lib/racket/sandbox.rkt:921:7
/home/root/racket/share/pkgs/sandbox-lib/racket/sandbox.rkt:891:2: user-process
from
eval:37:0: (/ 1 0)
predicate : exn:fail:syntax?
> (chk #:x 0 exn:fail?)
FAILURE
kind : "exception"
actual : 0
from
eval:38:0: 0
predicate : "exception"
> (chk #:x (/ 1 0) 0)
FAILURE
kind : "valid exception expected"
actual : 0
from
eval:39:0: 0
predicate : "exception expectation"
> (chk #:x (/ 1 0) cons)
FAILURE
kind : "valid exception expected"
actual : #<procedure:cons>
from
eval:40:0: cons
predicate : "exception expectation"
> (chk #:x (/ 1 0) add1)
FAILURE
kind : "exception"
actual : /: division by zero
context...:
/home/root/user/.local/share/racket/8.16.0.1/pkgs/chk-lib/chk/main.rkt:119:4
body of top-level
/home/root/racket/collects/racket/private/more-scheme.rkt:148:2: call-with-break-parameterization
/home/root/racket/share/pkgs/sandbox-lib/racket/sandbox.rkt:921:7
/home/root/racket/share/pkgs/sandbox-lib/racket/sandbox.rkt:891:2: user-process
from
eval:41:0: (/ 1 0)
part : "evaluating exn predicate"
exn : add1: contract violation
expected: number?
given: (exn:fail:contract:divide-by-zero "/: division by zero" #<continuation-mark-set>)
context...:
/home/root/user/.local/share/racket/8.16.0.1/pkgs/chk-lib/chk/main.rkt:350:47
/home/root/racket/collects/racket/private/more-scheme.rkt:148:2: call-with-break-parameterization
/home/root/racket/share/pkgs/sandbox-lib/racket/sandbox.rkt:921:7
/home/root/racket/share/pkgs/sandbox-lib/racket/sandbox.rkt:891:2: user-process
4 Meta Forms
There are two meta-forms provided by chk.
A #:! t test, where t is another test, represents the expectation that test t will fail.
> (chk #:! #:t #f) > (chk #:! #:t #t)
FAILURE
inverted : #t
kind : "predicate"
predicate : #<procedure:identity>
actual : #t
from
eval:43:0: #t
value : #t
> (chk #:! #:= 1 0) > (chk #:! #:= 1 1)
FAILURE
inverted : #t
kind : "compare"
compare : #<procedure:equal?>
actual : 1
from
eval:45:0: 1
expected : 1
from
eval:45:0: 1
> (chk #:! #:x (/ 1 0) exn:fail:syntax?) > (chk #:! #:x (/ 1 0) "division")
FAILURE
inverted : #t
kind : "exception"
actual : /: division by zero
context...:
/home/root/user/.local/share/racket/8.16.0.1/pkgs/chk-lib/chk/main.rkt:119:4
/home/root/user/.local/share/racket/8.16.0.1/pkgs/chk-lib/chk/main.rkt:351:54: body of top-level
/home/root/racket/collects/racket/private/more-scheme.rkt:148:2: call-with-break-parameterization
/home/root/racket/share/pkgs/sandbox-lib/racket/sandbox.rkt:921:7
/home/root/racket/share/pkgs/sandbox-lib/racket/sandbox.rkt:891:2: user-process
from
eval:47:0: (/ 1 0)
A #:do e form is equivalent to e. (This does not appear to be useful, unless you read the next section!)
5 Combining Tests
A chk form can contain any number of sub-forms. This is why #:do can be useful. They may be wrapped in parentheses, or not.
> (chk 1 1 2 2 #t)
> (chk #:= 1 1 #:! #:t #f #:? even? 2)
> (chk [#:= 1 1] [#:! #:t #f] [#:? even? 2])
> (chk #:! [#:t #f] #:! [#:= 1 2])
> (chk #:= (+ 2 3) 5 #:do (define x 1) #:= (+ 5 x) 6)
6 Test Sets
syntax
(chk* body ...+)
> (chk* (define x "one") (chk #:? number? x #:do (printf "Hey, listen!\n") #:? even? x) (chk #:? odd? (add1 x)))
FAILURE
kind : "predicate"
predicate : #<procedure:number?>
actual : "one"
from
eval:54:0: x
value : "one"
predicate : "not false"
value
chk-fail-first? : (parameter/c boolean?)
7 Contextual Failure Reporting
syntax
(with-chk ([key value] ...) body ...+)
> (with-chk (['author "jay"]) (chk #:= 1 1 #:= 1 2) (with-chk (['suite "1337"]) (chk #:? even? "two")))
FAILURE
author : "jay"
kind : "compare"
compare : #<procedure:equal?>
actual : 1
from
eval:55:0: 1
expected : 2
from
eval:55:0: 2
part : "value 0"
actual : 1
expected : 2
FAILURE
author : "jay"
suite : "1337"
kind : "predicate"
predicate : #<procedure:even?>
actual : "two"
from
eval:55:0: "two"
value : "two"
part : "evaluating predicate"
exn : even?: contract violation
expected: integer?
given: "two"
context...:
/home/root/user/.local/share/racket/8.16.0.1/pkgs/chk-lib/chk/main.rkt:350:47
/home/root/racket/collects/racket/private/more-scheme.rkt:148:2: call-with-break-parameterization
/home/root/racket/share/pkgs/sandbox-lib/racket/sandbox.rkt:921:7
/home/root/racket/share/pkgs/sandbox-lib/racket/sandbox.rkt:891:2: user-process
value
> (with-chk (['author "jay"] [chk-inform! (λ (d) (printf "Woah:\n~v\n" d))]) (chk #:= 1 1 #:= 1 2))
Woah:
'((author . "jay") (chk-inform!6529 . #<procedure>) (kind . "compare") (compare . #<procedure:equal?>) (actual . #s((res:values res 1) #<syntax:eval:56:0 1> (1))) (expected . #s((res:values res 1) #<syntax:eval:56:0 2> (2))) (part . "value 0") (actual . 1) (expected . 2))
FAILURE
author : "jay"
kind : "compare"
compare : #<procedure:equal?>
actual : 1
from
eval:56:0: 1
expected : 2
from
eval:56:0: 2
part : "value 0"
actual : 1
expected : 2
8 Filtering Tests with Command-line Arguments
The key/value pairs specified in the current context by with-chk can also be used to group tests into catagories which can be selected and run by supplying command-line arguments to the module running the tests. Before each test is run, chk will use the value of current-command-line-arguments to construct a list of names and key/value pairs.
Key/value pairs are specified by arguments of the form key=value. If a non-zero amount of key/value pairs are specified, then only tests which match the set of key/value pairs specified on the command line will be executed. The command-line argument "foo=bar", for example, corresponds with the racket pair ('foo . "bar").
> (parameterize ([current-command-line-arguments (vector "foo=bar")]) (with-chk (['foo "bar"]) (chk #:= 1 2)) (chk #:= 2 3))
FAILURE
foo : "bar"
kind : "compare"
compare : #<procedure:equal?>
actual : 1
from
eval:57:0: 1
expected : 2
from
eval:57:0: 2
part : "value 0"
actual : 1
expected : 2
In the above example, the former test runs because it has been called inside of an expression where the key 'foo matches the value "bar", but the latter test is skipped.
Keys read from the command line are always treated as symbols and compared using eq?. Values are treated as strings, and the runtime values that they are compared against are cast to string with ~a before the comparison. Because of this, it is simple to filter against values of any transparent data type, not just strings.
> (parameterize ([current-command-line-arguments (vector "data=(1 2 3)")]) (with-chk (['data '(1 2 3)]) (chk #:= 1 2)))
FAILURE
data : (1 2 3)
kind : "compare"
compare : #<procedure:equal?>
actual : 1
from
eval:58:0: 1
expected : 2
from
eval:58:0: 2
part : "value 0"
actual : 1
expected : 2
If multiple key/value pairs are present on the command line, then tests will only run if they match against all specified key/value pairs.
> (parameterize ([current-command-line-arguments (vector "foo=bar" "number=6")]) (with-chk (['foo "bar"]) (chk #:= 1 2) (with-chk (['number 6]) (chk #:= 2 3))))
FAILURE
foo : "bar"
number : 6
kind : "compare"
compare : #<procedure:equal?>
actual : 2
from
eval:59:0: 2
expected : 3
from
eval:59:0: 3
part : "value 0"
actual : 2
expected : 3
In the above example, the former test does not run because it is defined in a context in which only the key 'foo has a value. The latter test runs because its context contains the correct values for both 'foo and 'number.
The keys 'file and 'line are special cases when present in the command-line arguments. They are case insensitive and are matched against the file name and line number of current test. Files are treated as regular expressions. Multiple file names and line numbers may be specified on the command line to allow more tests through in a single test run.
Any command-line arguments which do not fit the name=value pattern are treated as regular expressions and used as a set of names to run. In the execution context of the test, these names are compared against the value corresponding with the 'name key. Unlike with key/value pairs, a test’s name needs only to match against one of the names specified on the command line in order to be run. This way, multiple tests can be specified to be run. Tests with no 'name key specified will only be run in the case that no names are present in the command-line arguments.
> (parameterize ([current-command-line-arguments (vector "test1" "test2")]) (with-chk (['name "test1"]) (chk #:= 1 2)) (with-chk (['name "test2"]) (chk #:= 2 3)) (chk #:= 3 4))
FAILURE
name : "test1"
kind : "compare"
compare : #<procedure:equal?>
actual : 1
from
eval:60:0: 1
expected : 2
from
eval:60:0: 2
part : "value 0"
actual : 1
expected : 2
FAILURE
name : "test2"
kind : "compare"
compare : #<procedure:equal?>
actual : 2
from
eval:60:0: 2
expected : 3
from
eval:60:0: 3
part : "value 0"
actual : 2
expected : 3
If both names and key/value pairs are present in the command-line arguments, then a test must satisfy both predicates in order to be run.
> (parameterize ([current-command-line-arguments (vector "test1" "foo=bar")]) (with-chk (['name "test1"]) (chk #:= 1 2)) (with-chk (['foo "bar"]) (chk #:= 2 3)))
In the above example, the command-line arguments are specifying that only tests which match the name "name1" and the key/value pair ('foo . "bar") should be run. Neither test satisfies both conditions, so neither test is run.
9 Controlling Source Location and Syntax Display
Everywhere in the preceeding discussion where we referred to an expression evaluating to something, that position may be replaced with (#:stx stx e) or (#:src src e). In both cases, the expression evaluated is e. However, if there is an error, then a different syntax (stx, which is simply a datum) or source location (src which is evaluate to something that is a source-location?) is reported.
> (chk #:= (#:stx one 1) (#:stx two 2))
FAILURE
kind : "compare"
compare : #<procedure:equal?>
actual : 1
from
eval:62:0: one
expected : 2
from
eval:62:0: two
part : "value 0"
actual : 1
expected : 2
> (chk #:= (#:src #'here 1) 2)
FAILURE
kind : "compare"
compare : #<procedure:equal?>
actual : 1
from
eval:63:0: 1
expected : 2
from
eval:63:0: 2
part : "value 0"
actual : 1
expected : 2
> (chk #:= (#:src '(there 99 1 22 1) "one") "two")
FAILURE
kind : "compare"
compare : #<procedure:equal?>
actual : "one"
from
there:99:1: "one"
expected : "two"
from
eval:64:0: "two"
part : "value 0"
actual : "one"
expected : "two"
This is intended to be used by macros that produce chks.