Chez Scheme: more documentation how tests work
Also, make some adjustments to "Mf-base" so that plain `make` generates a summary file as before and so that output files are linked in a way that lets `make root-experr` and `make patches` find them.
This commit is contained in:
Matthew Flatt
parent
88cb6aae10
commit
e899b519e3
|
|
@ -195,13 +195,11 @@ The makefile supports several targets:
|
|||
|
||||
Runs the build plus runs a set of test programs in various
|
||||
different ways, e.g., with different compiler options. It can take
|
||||
30 minutes or more, depending on the speed of the machine. It
|
||||
produces voluminous output, so it's best to redirect its stdout and
|
||||
stderr to a file.
|
||||
on the order of an hour, depending on the speed of the machine.
|
||||
|
||||
A complete run does *not* imply no errors occurred. To check for
|
||||
errors, look at the file "<workarea>/mats/summary", which should
|
||||
contain one line per test run, something like this:
|
||||
At the end of a complete run, the summary recorded in
|
||||
"<workarea>/mats/summary" is shown. It should contain one line per
|
||||
test configuration, something like this:
|
||||
|
||||
-------- o=0 --------
|
||||
-------- o=3 --------
|
||||
|
|
@ -217,7 +215,7 @@ The makefile supports several targets:
|
|||
-------- o=3 ehc=t eval=interpret --------
|
||||
|
||||
If there is anything else in "<workarea>/mats/summary", something
|
||||
unexpected occurred.
|
||||
unexpected occurred. See "IMPLEMENTATION.md" for more information.
|
||||
|
||||
* `make <machine type>.boot` or `make <machine type>.bootquick`
|
||||
|
||||
|
|
|
|||
|
|
@ -36,6 +36,28 @@ a new platform. In particular, the `pb` (portable bytecode) machine
|
|||
type can run on any supported hardware and operating system, so having
|
||||
`pb` boot files is one way to get started in a new environment.
|
||||
|
||||
# Compiled Files and Boot Files
|
||||
|
||||
A Scheme file conventionally uses the suffix ".ss" and it's compiled
|
||||
form uses the suffix ".so". The format of a compiled file is closely
|
||||
related to the fasl format that is exposed by `fasl-write` and
|
||||
`fasl-read`, but you can't compile Scheme code to some value that is
|
||||
written with `fasl-write`. Instead, `compile-file` and related
|
||||
functions directly generate compiled code in a fasled form that
|
||||
includes needed linking information.
|
||||
|
||||
A boot file, usually with the suffix ".boot", has the same format as a
|
||||
compiled file, but with an extra header that identifies it as a boot
|
||||
file and takes care of some singleton objects, such as `#!base-rtd`
|
||||
and the stub to invoke compiled code.
|
||||
|
||||
The vfasl format is used for the same purposes as the fasl format, but
|
||||
mostly for boot files. It is always platform-specific and its content
|
||||
is very close to the form that the content will take when loaded into
|
||||
memory. It can load especially quickly with streamlined linking and
|
||||
interning of symbols and record types, especially in uncompressed
|
||||
form. The build scripts do not convert boot files to vfasl format.
|
||||
|
||||
# Build System
|
||||
|
||||
Chez Scheme assigns a `machine-type` name to each platform it runs on.
|
||||
|
|
@ -68,6 +90,9 @@ to generate a subdirectory with the appropriate contents to build for
|
|||
that particular machine. This is the script that "configure" runs when
|
||||
configuring for doing the build, but you can also run the "workarea"
|
||||
script on your own, supplying the machine type you'd like to build.
|
||||
The directory where you run "configure" or "workarea" is the "build
|
||||
directory", while the directory named "*machine-type*" created by
|
||||
"workarea" is the "workarea directory".
|
||||
|
||||
Bootstrap from scratch by running the Racket program
|
||||
"rktboot/main.rkt", which should work even with a relatively old
|
||||
|
|
@ -143,32 +168,134 @@ will be compiled as unsafe. While testing and debugging your
|
|||
additions, however, you'll probably want to use `make o=0` in the
|
||||
"*machine-type*/s" workarea space, which compiles in safe mode.
|
||||
|
||||
Tests go in "mats/*...*.ms". In "*machine-type*/mats", you can use
|
||||
`make 7.mo` to build and run `7.ms`. Remove `7.mo` to re-run without
|
||||
changing `7.ms`. Makefile variables like `o` control the way tests
|
||||
are run; for example, use `make o=3 7.mo` to test in unsafe mode.
|
||||
# Writing and Running Tests
|
||||
|
||||
# Compiled Files and Boot Files
|
||||
A group of tests is written in a ".ms" file in the "mats" directory.
|
||||
Within a `mat` form after the name for the group of tests, each test
|
||||
is written as a expression that produces `#t` for success. Use the
|
||||
`error?` form to wrap an expression that is supposed to raise an
|
||||
exception in safe mode, but note that the test doesn't describe the
|
||||
exception specifically, since the expected error message likely
|
||||
depends on the configuration (e.g. safe versus unsafe); more on that
|
||||
below.
|
||||
|
||||
A Scheme file conventionally uses the suffix ".ss" and it's compiled
|
||||
form uses the suffix ".so". The format of a compiled file is closely
|
||||
related to the fasl format that is exposed by `fasl-write` and
|
||||
`fasl-read`, but you can't compile Scheme code to some value that is
|
||||
written with `fasl-write`. Instead, `compile-file` and related
|
||||
functions directly generate compiled code in a fasled form that
|
||||
includes needed linking information.
|
||||
### Running One Set of Tests (no expected-error checking)
|
||||
|
||||
A boot file, usually with the suffix ".boot", has the same format as a
|
||||
compiled file, but with an extra header that identifies it as a boot
|
||||
file and takes care of some singleton objects, such as `#!base-rtd`
|
||||
and the stub to invoke compiled code.
|
||||
Runs tests in a ".ms" file by going to your build's
|
||||
"*machine-type*/mats" directory, then `make` with a ".mo" target. For
|
||||
example, use `make 7.mo` to build and run `7.ms`. Delete `7.mo` to run
|
||||
`7.ms` again. Makefile variables like `o` control the way tests are
|
||||
run; for example, use `make 7.mo o=3` to test in unsafe mode. See the
|
||||
source file "mats/Mf-base" for information about the configuration
|
||||
options. Running tests to make a ".mo" file prints a lot of output, so
|
||||
you'll likely want to redirect stdout and stderr to a file.
|
||||
|
||||
The vfasl format is used for the same purposes as the fasl format, but
|
||||
mostly for boot files. It is always platform-specific and its content
|
||||
is very close to the form that the content will take when loaded into
|
||||
memory. It can load especially quickly with streamlined linking and
|
||||
interning of symbols and record types, especially in uncompressed
|
||||
form. The build scripts do not convert boot files to vfasl format.
|
||||
A test failure is recorded in a ".mo" file as a line that contains
|
||||
`Bug`, `Error`, or `invalid memory`. That's why the target for making
|
||||
a ".mo" file ends by grepping the file. Tests for exceptions produce
|
||||
the output `Expected error`, but there's not currently a way to check
|
||||
that the exception tests of an individual ".ms" file produce the
|
||||
expected error message.
|
||||
|
||||
### Running Tests in One Configuration (with expected-error checking)
|
||||
|
||||
You can make all ".mo" files with just `make` within your build's
|
||||
"*machine-type*/mats". You can provide configuration arguments, too,
|
||||
such as `make o=3` to make all ".mo" files in unsafe mode.
|
||||
|
||||
In this mode, output ".mo" files are written to a subdirectory that is
|
||||
partially configuration-specific, such as "compile-0-f-f-f" for
|
||||
`compile` (as opposed to `interpret`) in safe mode (`0` instead of
|
||||
`3`), without `suppress-primitive-inlining` enabled (first `f`),
|
||||
without cp0 enabled (second `f`), and without
|
||||
`compile-interpret-simple` enabled (last `f`). Note that a set of
|
||||
tests is not run again if an up-to-date ".mo" file is in the output
|
||||
directory, so use `make clean` as needed.
|
||||
|
||||
The combination of all ".mo" error messages (from both expected
|
||||
exceptions and test failures) is compared against a list of expected
|
||||
errors messages for a configuration using `diff`. The `diff` result is
|
||||
written to "report-*config*", where *config* is the name of the
|
||||
configuration. So, an empty "report-*config*" means success.
|
||||
|
||||
The set of expected error messages for a given configuration is
|
||||
generated by starting with either "mats/root-experr-compile-0-f-f-f"
|
||||
or "mats/root-experr-compile-3-f-f-f" (depending on whether the
|
||||
configuration is in unsafe or safe mode) and then applying some number
|
||||
of patches from "mats/patch-*config*". That's why the *config* in
|
||||
"report-*config*" doesn't identify everything about the configuration;
|
||||
it only identifies the combinations that can have different error
|
||||
output.
|
||||
|
||||
If you add a new test that's expected to have error output (usually to
|
||||
check that an exception is correctly raised), then
|
||||
"mats/root-experr-*config*" and/or "mats/patch-*config*" files need to
|
||||
change. Modifying those files by hand is not practical. Instead, the
|
||||
strategy is to make sure that the output diff in "record-*config*" is
|
||||
correct, and then use targets like `make root-experr` and `make
|
||||
patches` to generate new "root-experr-..." and "patch-..." files:
|
||||
|
||||
* Run `make` and then `make root-experr` to generate a new
|
||||
"root-experr-compile-0-f-f-f", then copy the generated file in the
|
||||
workarea to the source "mats" directory. Often, this step is all
|
||||
that is needed to update expected errors, since expected errors
|
||||
tend to happen only in safe mode, and they tend not to change among
|
||||
other configuration options.
|
||||
|
||||
* If you need to update "root-experr-compile-3-f-f-f", use `make
|
||||
root-experr o=3` after running with `make o=3` and then copy the
|
||||
file from the workarea to the "mats" source directory.
|
||||
|
||||
* After running tests for a configuration with `make` plus
|
||||
configuration options, you may be able to recreate the
|
||||
corresponding patch file using `make xpatch-*config*` with the same
|
||||
configuration options. However, some configurations involve layers
|
||||
of patch files, so it's tricky to get this right by running test
|
||||
configurations one at a time, and it's better to run tests for all
|
||||
configurations.
|
||||
|
||||
### Running Tests for All Configurations
|
||||
|
||||
To run tests for all configurations, use `make allx` within your
|
||||
build's "*machine-type*/mats" directory. Add `-j` followed by *N* to
|
||||
run tests using *N* parallel jobs. Using `make allx` implicitly uses
|
||||
`make clean` before it runs tests. You can also use `make test`
|
||||
directly in your build directory, since that's a shortcut for `make
|
||||
allx` in the "*machine-type*/mats" directory.
|
||||
|
||||
To support parallel tests, `make allx` write its output in a
|
||||
collection of "output-*i*-*o*" directories within
|
||||
"*machine-type*/mats", so you can look for "report-*config*" files in
|
||||
those subdirectories. As its last step, `make allx` combines a summary
|
||||
of reports to a `summary` file directly in "*machine-type*/mats", and
|
||||
then it shows that summary as output. As long as that output shows
|
||||
only configurations (i.e., no errors), then all tests passed for all
|
||||
configurations.
|
||||
|
||||
After running `make allx`, if the summary shows only errors that
|
||||
reflect out-of-date expectations from "root-experr-..." or "patch-..."
|
||||
files, you can use the sequence
|
||||
|
||||
```bash
|
||||
make root-experr o=0
|
||||
make root-experr o=3
|
||||
make patches
|
||||
```
|
||||
|
||||
to create new vesions of the files in the workarea directory. Copy
|
||||
changed files to the "mats" source directory; if the only change to a
|
||||
patch file is to the line-number hints, then it's probably not worth
|
||||
keeping the update (as long as the line numbers are not too far off).
|
||||
A `make clean` will delete any "root-experr-..." or "patch-..." files,
|
||||
and links are created on demand in the workarea space to
|
||||
"root-experr-..." or "patch-..." files that are needed to generate
|
||||
expexted-error diffs.
|
||||
|
||||
Despite its name, `make allx` does not run all available tests. Use
|
||||
`make bullyx` to run a different, more stressfull set of tests. The
|
||||
bully tests may cover more configurations than `allx`, so `make
|
||||
patches` after `make bullyx` may pick up additional "patch-..." file
|
||||
changes.
|
||||
|
||||
# Scheme Objects
|
||||
|
||||
|
|
|
|||
|
|
@ -158,7 +158,7 @@ Examples = $(abspath ../examples)
|
|||
MAKEFLAGS += --no-print-directory
|
||||
|
||||
# directory where (most) output for this run will be written
|
||||
outdir=output
|
||||
outdir=.
|
||||
|
||||
conf = $(eval)-$o-$(spi)-$(cp0)-$(cis)
|
||||
objdir=output-$(conf)
|
||||
|
|
@ -214,6 +214,7 @@ $(objdir)/%.mo : %.ms mat.so
|
|||
' (fprintf (console-error-port) "check heap detected errors---grep standard output for !!!\n")'\
|
||||
' (abort))'\
|
||||
| ${Scheme} -q mat.so ${patchfile}
|
||||
! egrep "Bug|Error|invalid memory" $*.mo
|
||||
|
||||
%.so : %.ss
|
||||
echo '(reset-handler abort) (time (compile-file "$*"))' | ${Scheme} -q ${patchfile}
|
||||
|
|
@ -332,8 +333,15 @@ just-reports:
|
|||
allxhelp:
|
||||
$(MAKE) doheader
|
||||
-$(MAKE) all
|
||||
$(MAKE) errors-$(conf) forpatches=.
|
||||
$(MAKE) dosummary
|
||||
|
||||
# To support an eventual `make patches`, link `errors-$(conf)` to allxhelp output
|
||||
# if there's not already a representative for the configuration:
|
||||
forpatches = different-from-outdir
|
||||
$(forpatches)/errors-$(conf): $(outdir)/errors-$(conf)
|
||||
ln -s $(outdir)/errors-$(conf) errors-$(conf)
|
||||
|
||||
config-vars = spi hci ecpf cp0 cis p eval ctb cgr cmg eoc cl rmg
|
||||
full-config-str = $(strip o=$(o) $(foreach var, $(config-vars),$(if $(filter-out $($(var:%=default%)),$($(var))),$(var)=$($(var)))) $(hdrmsg))
|
||||
|
||||
|
|
@ -471,7 +479,7 @@ experr-compile-$o-f-f-f: root-experr-compile-$o-f-f-f
|
|||
|
||||
root-experr: # don't list dependencies!
|
||||
rm -f root-experr-compile-$o-f-f-f
|
||||
cp errors-compile-$o-f-f-f root-experr-compile-$o-f-f-f
|
||||
cp $(outdir)/errors-compile-$o-f-f-f root-experr-compile-$o-f-f-f
|
||||
|
||||
# derive spi=t experr files by patching spi=f experr files
|
||||
# cp first in case patch is empty, since patch produces an empty output
|
||||
|
|
|
|||
Loading…
Reference in New Issue
Block a user