Version: 4.1.5.3
3 SchemeUnit API
3.1 Overview of SchemeUnit
There are three basic data types in SchemeUnit:
A check is the basic unit of a test. As the name suggests, it checks some condition is true.
A test case is a group of checks that form one conceptual unit. If any check within the case fails, the entire case fails.
A test suite is a group of test cases and test suites that has a name.
3.2 Checks
Checks are the basic building block of SchemeUnit. A check
-checks some condition. If the condition holds the check
-evaluates to #t. If the condition doesn’t hold the
-check raises an instance of exn:test:check with
-information detailing the failure.
Although checks are implemented as macros, which is
-necessary to grab source location, they are conceptually
-functions. This means, for instance, checks always evaluate
-their arguments. You can use check as first class
-functions, though you will lose precision in the reported
-source locations if you do so.
The following are the basic checks SchemeUnit provides. You
-can create your own checks using define-check.
| (check op v1 v2 [message]) → #t |
| op : (-> any any (or/c #t #f)) |
| v1 : any |
| v2 : any |
| message : string? = "" |
The simplest check. Succeeds if op applied to
-v1 and v2 is not #f, otherwise
-raises an exception of type exn:test:check. The
-optional message is included in the output if the
-check fails.
For example, the following check succeeds:
(check < 2 3)
| (check-eq? v1 v2 [message]) → #t |
| v1 : any |
| v2 : any |
| message : string? = "" |
| (check-not-eq? v1 v2 [message]) → #t |
| v1 : any |
| v2 : any |
| message : string? = "" |
| (check-eqv? v1 v2 [message]) → #t |
| v1 : any |
| v2 : any |
| message : string? = "" |
| (check-equal? v1 v2 [message]) → #t |
| v1 : any |
| v2 : any |
| message : string? = "" |
| (check-not-equal? v1 v2 [message]) → #t |
| v1 : any |
| v2 : any |
| message : string? = "" |
Checks that v1 is (not) eq?,
-eqv?, or equal? to v2. The
-optional message is included in the output if the
-check fails.
For example, the following checks all fail:
| (check-eq? (list 1) (list 1) "allocated data not eq?") |
| (check-not-eq? 1 1 "integers are eq?") |
| (check-eqv? 1 1.0 "not eqv?") |
| (check-equal? 1 1.0 "not equal?") |
| (check-not-equal? (list 1) (list 1) "equal?") |
| (check-pred pred v [message]) → #t |
| pred : (-> any (or/c #t #f)) |
| v : any |
| message : string? = "" |
Checks that pred returns #t when applied to v. The optional message is included in the output if the check fails.
Here’s an example that passes and an example that fails:
| (check-= v1 v2 epsilon [message]) → #t |
| v1 : any |
| v2 : any |
| epsilon : number? |
| message : string? = "" |
Checks that v1 and v2 are within
-epsilon of one another. The optional
-message is included in the output if the check
-fails.
Here’s an example that passes and an example that fails:
| (check-= 1.0 1.01 0.01 "I work") |
| (check-= 1.0 1.01 0.005 "I fail") |
| (check-true v [message]) → #t |
| v : any |
| message : string? = "" |
| (check-false v [message]) → #t |
| v : any |
| message : string? = "" |
| (check-not-false v [message]) → #t |
| v : any |
| message : string? = "" |
Checks that v is #t, #f, or not
-#f as appropriate. The optional message
-is included in the output if the check fails.
For example, the following checks all fail:
| (check-true 1) |
| (check-false 1) |
| (check-not-false #f) |
| (check-exn exn-predicate thunk [message]) → #t |
| exn-predicate : (-> any (or/c #t #f)) |
| thunk : (-> any) |
| message : string? = "" |
Checks that thunk raises an exception for which
-exn-predicate returns #t. The optional
-message is included in the output if the check
-fails. A common error is to use an expression instead of a
-function of no arguments for thunk. Remember that
-checks are conceptually functions.
Here are two example, one showing a test that succeeds, and one showing a common error:
| (check-not-exn thunk [message]) → #t |
| thunk : (-> any) |
| message : string? = "" |
Checks that thunk does not raise any exceptions.
-The optional message is included in the output if
-the check fails.
| (fail [message]) → #t |
| message : string? = "" |
This checks fails unconditionally. Good for creating test stubs that youintend to fill out later. The optional message is included in the output if the check fails.
| (check-regexp-match regexp string) → #t |
| regexp : regexp? |
| string : string? |
Checks that regexp matches the string.
The following check will succeed:
(check-regexp-match "a+bba" "aaaaaabba")
This check will fail:
(check-regexp-match "a+bba" "aaaabbba")
3.2.1 Augmenting Information on Check Failure
When an check fails it stores information including the name
-of the check, the location and message (if available), the
-expression the check is called with, and the parameters to
-the check. Additional information can be stored by using
-the with-check-info* function, and the
-with-check-info macro.
| (struct check-info (name value)) |
| name : symbol? |
| value : any |
A check-info structure stores information associated
-with the context of execution of an check.
The are several predefined functions that create check
-information structures with predefined names. This avoids
-misspelling errors:
| (make-check-name name) → check-info? |
| name : string? |
| (make-check-params params) → check-info? |
| params : (listof any) |
| (make-check-location loc) → check-info? |
| loc : (list any (or/c number? #f) (or/c number? #f) (or/c number? #f) (or/c number? #f)) |
| (make-check-expression msg) → check-info? |
| msg : any |
| (make-check-message msg) → check-info? |
| msg : string? |
| (make-check-actual param) → check-info? |
| param : any |
| (make-check-expected param) → check-info? |
| param : any |
| (with-check-info* info thunk) → any |
| info : (listof check-info?) |
| thunk : (-> any) |
Stores the given info on the check-info stack for
-the duration (the dynamic extent) of the execution of
-thunk
Example:
When this check fails the message
| time: <current-seconds-at-time-of-running-check> |
will be printed along with the usual information on an
-check failure.
| (with-check-info ((name val) ...) body ...) |
The with-check-info macro stores the given
-information in the check information stack for the duration
-of the execution of the body expressions. Name is
-a quoted symbol and val is any value.
Example:
When this test fails the message
will be displayed along with the usual information on an
-check failure.
3.2.2 Custom Checks
Custom checks can be defined using define-check and
-its variants. To effectively use these macros it is useful
-to understand a few details about a check’s evaluation
-model.
Firstly, a check should be considered a function, even
-though most uses are actually macros. In particular, checks
-always evaluate their arguments exactly once before
-executing any expressions in the body of the checks. Hence
-if you wish to write checks that evalute user defined code
-that code must be wrapped in a thunk (a function of no
-arguments) by the user. The predefined check-exn
-is an example of this type of check.
It is also useful to understand how the check information
-stack operates. The stack is stored in a parameter and the
-with-check-info forms evaluate to calls to
-parameterize. Hence check information has lexical
-scope. For this reason simple checks (see below) cannot
-usefully contain calls to with-check-info to report
-additional information. All checks created using
-define-simple-check or define-check grab
-some information by default: the name of the checks and the
-values of the parameters. Additionally the macro forms of
-checks grab location information and the expressions passed
-as parameters.
| (define-simple-check (name param ...) expr ...) |
The define-simple-check macro constructs a check
-called name that takes the params and an optional
-message as arguments and evaluates the exprs. The
-check fails if the result of the exprs is
-#f. Otherwise the check succeeds. Note that
-simple checks cannot report extra information using
-with-check-info.
Example:
To define a check check-odd?
| (define-simple-check (check-odd? number) |
| (odd? number)) |
We can use these checks in the usual way:
| (check-odd? 3) |
| (check-odd? 2) |
| (define-binary-check (name pred actual expected)) |
| (define-binary-check (name actual expected) expr ...) |
The define-binary-check macro constructs a check
-that tests a binary predicate. It’s benefit over
-define-simple-check is in better reporting on check
-failure. The first form of the macro accepts a binary
-predicate and tests if the predicate holds for the given
-values. The second form tests if expr non-false.
Examples:
Here’s the first form, where we use a predefined predicate
-to construct a binary check:
(define-binary-check (check-char=? char=? actual expected))
In use:
(check-char=? (read-char a-port) #\a)
If the expression is more complicated the second form should
-be used. For example, below we define a binary check that
-tests a number if within 0.01 of the expected value:
| (define-binary-check (check-in-tolerance actual expected) |
| (< (abs (- actual expected)) 0.01)) |
| (define-check (name param ...) expr ...) |
The define-check macro acts in exactly the same way
-as define-simple-check, except the check only fails
-if the macro fail-check is called in the body of
-the check. This allows more flexible checks, and in
-particular more flexible reporting options.
The fail-check macro raises an exn:test:check with
-the contents of the check information stack.
3.2.3 The Check Evaluation Context
The semantics of checks are determined by the parameters
-current-check-around and
-current-check-handler. Other testing form such as
-test-begin and test-suite change the value
-of these parameters.
| (current-check-handler) → (-> any/c any/c) |
| (current-check-handler handler) → void? |
| handler : (-> any/c any/c) |
Parameter containing the function that handles exceptions
-raised by check failures. The default behaviour is to print
-an error message including the exception message and stack
-trace.
| (current-check-around) → (-> thunk any/c) |
| (current-check-around check) → void? |
| check : (-> thunk any/c) |
Parameter containing the function that handles the execution
-of checks. The default value wraps the evaluation of
-thunk in a with-handlers call that calls
-current-check-handler if an exception is raised.
3.3 Compound Testing Forms
3.3.1 Test Cases
As programs increase in complexity the unit of testing
-grows beyond a single check. For example, it may be the case
-that if one check fails it doesn’t make sense to run
-another. To solve this problem compound testing forms can
-be used to group expressions. If any expression in a group
-fails (by raising an exception) the remaining expressions
-will not be evaluated.
A test-begin form groups the exprs into a
-single unit. If any expr fails the following ones
-are not evaluated.
For example, in the following code the world is not
-destroyed as the preceding check fails:
| (test-begin |
| (check-eq? 'a 'b) |
| |
| (destroy-the-world)) |
| (test-case name expr ...) |
Like a test-begin except a name is associated with
-the group of exprs. The name will be reported if
-the test fails.
Here’s the above example rewritten to use test-case
-so the test can be named.
| (test-case |
| "Example test" |
| (check-eq? 'a 'b) |
| |
| (destroy-the-world)) |
True if obj is a test case, and false otherwise
3.3.2 Test Suites
Test cases can themselves be grouped into test suites. A
-test suite can contain both test cases and test suites.
-Unlike a check or test case, a test suite is not immediately
-run. Instead use one of the functions described in
-User Interfaces or Programmatically Running Tests and Inspecting Results.
| (test-suite name [#:before before-thunk] [#:after after-thunk] test ...) |
Constructs a test suite with the given name and tests. The
-tests may be test cases, constructed using
-test-begin or test-case, or other test
-suites.
The before-thunk and after-thunk are
-optional thunks (functions are no argument). They are run
-before and after the tests are run, respectively.
Unlike a check or test case, a test suite is not immediately
-run. Instead use one of the functions described in
-User Interfaces or Programmatically Running Tests and Inspecting Results.
For example, here is a test suite that displays Before
-before any tests are run, and After when the tests have
-finished.
| (test-suite |
| "An example suite" |
| #:before (lambda () (display "Before")) |
| #:after (lambda () (display "After")) |
| (test-case |
| "An example test" |
| (check-eq? 1 1))) |
True if
-obj is a test suite, and false otherwise
3.3.2.1 Utilities for Defining Test Suites
There are some macros that simplify the common cases of
-defining test suites:
| (define-test-suite name test ...) |
The
-define-test-suite form creates a test suite with
-the given name (converted to a string) and tests, and binds
-it to the same name.
For example, this code creates a binding for the name
-example-suite as well as creating a test suite with
-the name "example-suite":
| (define-test-suite example-suite |
| (check = 1 1)) |
| (define/provide-test-suite name test ...) |
This
-for is just like define-test-suite, and in addition
-it provides the test suite.
Finally, there is the test-suite* macro, which
-defines a test suite and test cases using a shorthand
-syntax:
| (test-suite* name (test-case-name test-case-body | | ...) ...) |
|
Defines a test suite with the given name, and
-creates test cases within the suite, with the given names and
-body expressions.
As far I know no-one uses this macro, so it might disappear
-in future versions of SchemeUnit.
3.3.3 Compound Testing Evaluation Context
Just like with checks, there are several parameters that
-control the semantics of compound testing forms.
| (current-test-name) → (or/c string? false/c) |
| (current-test-name name) → void? |
| name : (or/c string? false/c) |
This parameter stores the name of the current test case. A
-value of #f indicates a test case with no name,
-such as one constructed by test-begin.
| (current-test-case-around) → (-> (-> any/c) any/c) |
| (current-test-case-around handler) → void? |
| handler : (-> (-> any/c) any/c) |
This parameter handles evaluation of test cases. The value
-of the parameter is a function that is passed a thunk (a
-function of no arguments). The function, when applied,
-evaluates the expressions within a test case. The default
-value of the current-test-case-around parameters
-evaluates the thunk in a context that catches exceptions and
-prints an appropriate message indicating test case failure.
| (test-suite-test-case-around thunk) → any/c |
| thunk : (-> any/c) |
The current-test-case-around parameter is
-parameterized to this value within the scope of a
-test-suite. This function creates a test case
-structure instead of immediately evaluating the thunk.
| (test-suite-check-around thunk) → any/c |
| thunk : (-> any/c) |
The current-check-around parameter is parameterized
-to this value within the scope of a test-suite.
-This function creates a test case structure instead of
-immediately evaluating a check.
3.4 Test Control Flow
The before, after, and around
-macros allow you to specify code that is always run before,
-after, or around expressions in a test case.
| (before before-expr expr1 expr2 ...) |
Whenever control enters the scope execute the before-expr
-before executing expr-1, and expr-2 ...
| (after expr-1 expr-2 ... after-expr) |
Whenever control exits the scope execute the after-expr
-after executing expr-1, and expr-2 ... The after-expr is
-executed even if control exits via an exception or other means.
| (around before-expr expr-1 expr-2 ... after-expr) |
Whenever control enters the scope execute the
-before-expr before executing expr-1 expr-2 ..., and execute after-expr whenever control
-leaves the scope.
Example:
The test below checks that the file test.dat contains
-the string "foo". The before action writes to this
-file. The after action deletes it.
| (delay-test test1 test2 ...) |
This somewhat curious macro evaluates the given tests in a
-context where current-test-case-around is
-parameterized to test-suite-test-case-around. This
-has been useful in testing SchemeUnit. It might be useful
-for you if you create test cases that create test cases.
3.5 Miscellaneous Utilities
The require/expose macro allows you to access
-bindings that a module does not provide. It is useful for
-testing the private functions of modules.
| (require/expose module (id ...)) |
Requires id from module into the current module. It doesn’t matter if the source module provides the bindings or not; require/expose can still get at them.
Note that require/expose can be a bit fragile,
-especially when mixed with compiled code. Use at your own risk!
This example gets make-failure-test, which is defined in a SchemeUnit test:
(require/expose schemeunit/check-test (make-failure-test))
3.6 User Interfaces
SchemeUnit provides a textual and a graphical user interface
3.6.1 Textual User Interface
The textual UI is in the text-ui module. It is run
-via the run-tests function
| (run-tests test [verbosity]) → natural-number/c |
| test : (or/c test-case? test-suite?) |
| verbosity : (symbols 'quite 'normal 'verbose) = 'normal |
The given test is run and the result of running it
-output to the current-output-port. The output is
-compatable with the (X)Emacs next-error command (as used,
-for example, by (X)Emac’s compile function)
The optional verbosity is one of 'quiet,
-'normal, or 'verbose. Quiet output
-displays only the number of successes, failures, and errors.
-Normal reporting suppresses some extraneous check
-information (such as the expression). Verbose reports all
-information.
run-tests returns the number of unsuccessful tests.
3.6.2 Graphical User Interface
The GUI has not yet been updated to this version of SchemeUnit.
3.7 Programmatically Running Tests and Inspecting Results
SchemeUnit provides an API for running tests, from which
-custom UIs can be created.
3.7.1 Result Types
| (struct (exn:test exn) ()) |
The base structure for SchemeUnit exceptions. You should
-never catch instances of this type, only the subtypes
-documented below.
| (struct (exn:test:check exn:test) (stack)) |
| stack : (listof check-info) |
A exn:test:check is raised when an check fails, and
-contains the contents of the check-info stack at the
-time of failure.
| (struct test-result (test-case-name)) |
| test-case-name : (or/c string #f) |
A test-result is the result of running the test with
-the given name (with #f indicating no name is available).
| (struct (test-failure test-result) (result)) |
| result : any |
Subtype of test-result representing a test failure.
| (struct (test-error test-result) (result)) |
| result : exn |
Subtype of test-result representing a test error.
| (struct (test-success test-result) (result)) |
| result : any |
Subtype of test-result representing a test success.
3.7.2 Functions to Run Tests
| (run-test-case name action) → test-result |
| name : (or/c string #f) |
| action : (-> any) |
Runs the given test case, returning a result representing success, failure, or error.
| (run-test test) → (R = (listof (or/c test-result R))) |
| test : (or/c test-case? test-suite?) |
Runs the given test (test case or test suite) returning a
-tree (list of lists) of results
Example:
| (run-test |
| (test-suite |
| "Dummy" |
| (test-case "Dummy" (check-equal? 1 2)))) |
| (fold-test-results | | result-fn | | | | | | | | seed | | | | | | | | test | | | | | | | | #:run run | | | | | | | | #:fdown fdown | | | | | | | | #:fup fup) | | → | | 'a |
|
| result-fn : ('b 'c ... 'a . -> . 'a) |
| seed : 'a |
| test : (or/c test-case? test-suite?) |
| run : (string (() -> any) . -> . 'b 'c ...) |
| fdown : (string 'a . -> . 'a) |
| fup : (string 'a . -> . 'a) |
Fold result-fn pre-order left-to-right depth-first
-over the results of run. By default run
-is run-test-case and fdown and
-fup just return the seed, so result-fn is
-folded over the test results.
This function is useful for writing custom folds (and hence
-UIs) over test results without you having to take care of
-all the expected setup and teardown. For example,
-fold-test-results will run test suite before and
-after actions for you. However it is still flexible enough,
-via its keyword arguments, to do almost anything that foldts
-can. Hence it should be used in preference to foldts.
result-fn is a function from the results of
-run (defaults to a test-result) and the
-seed to a new seed
Seed is any value
Test is a test-case or test-suite
Run is a function from a test case name (string) and action
-(thunk) to any values.
FDown is a function from a test suite name (string) and the
-seed, to a new seed
FUp is a function from a test suite name (string) and the
-seed, to a new seed.
Examples:
The following code counts the number of successes
| (define (count-successes test) |
| (fold-test-results |
| (lambda (result seed) |
| (if (test-success? result) |
| (add1 seed) |
| seed)) |
| 0 |
| test)) |
The following code returns the symbol 'burp instead
-of running test cases. Note how the result-fn receives the
-value of run.
| (foldts fdown fup fhere seed test) → 'a |
| fdown : (test-suite string thunk thunk 'a -> 'a) |
| fup : (test-suite string thunk thunk 'a 'a -> 'a) |
| fhere : (test-case string thunk 'a -> 'a) |
| seed : 'a |
| test : (or/c test-case? test-suite?) |
Foldts is a nifty tree fold (created by Oleg Kiselyov) that
-folds over a test in a useful way (fold-test-results isn’t
-that useful as you can’t specify actions around test cases).
Fdown is a function of test suite, test suite name, before
-action, after action, and the seed. It is run when a test
-suite is encountered on the way down the tree (pre-order).
Fup is a function of test suite, test suite name, before
-action, after action, the seed at the current level, and the
-seed returned by the children. It is run on the way up the
-tree (post-order).
Fhere is a function of the test case, test case name, the
-test case action, and the seed. (Note that this might change
-in the near future to just the test case. This change would
-be to allow fhere to discriminate subtypes of test-case,
-which in turn would allow test cases that are, for example,
-ignored).
Example:
Here’s the implementation of fold-test-results in terms of
-foldts:
| (define (fold-test-results suite-fn case-fn seed test) |
| (foldts |
| (lambda (suite name before after seed) |
| (before) |
| (suite-fn name seed)) |
| (lambda (suite name before after seed kid-seed) |
| (after) |
| kid-seed) |
| (lambda (case name action seed) |
| (case-fn |
| (run-test-case name action) |
| seed)) |
| seed |
| test)) |
If you’re used to folds you’ll probably be a bit surprised
-that the functions you pass to foldts receive both the
-structure they operate on, and the contents of that
-structure. This is indeed unusual. It is done to allow
-subtypes of test-case and test-suite to be run in customised
-ways. For example, you might define subtypes of test case
-that are ignored (not run), or have their execution time
-recorded, and so on. To do so the functions that run the
-test cases need to know what type the test case has, and
-hence is is necessary to provide this information.
If you’ve made it this far you truly are a master SchemeUnit
-hacker. As a bonus prize we’ll just mention that the code
-in hash-monad.ss and monad.ss might be of interest for
-constructing user interfaces. The API is still in flux, so
-isn’t documented here. However, do look at the
-implementation of run-tests for examples of use.