Chez Scheme: more documentation how tests work

Also, make some adjustments to "Mf-base" so that plain `make`
generates a summary file as before and so that output files are linked
in a way that lets `make root-experr` and `make patches` find them.

racket/src/ChezScheme/BUILDING

@@ -195,13 +195,11 @@ The makefile supports several targets:
 
    Runs the build plus runs a set of test programs in various
    different ways, e.g., with different compiler options. It can take
-   30 minutes or more, depending on the speed of the machine. It
+   on the order of an hour, depending on the speed of the machine.
-   produces voluminous output, so it's best to redirect its stdout and
-   stderr to a file.
 
-   A complete run does *not* imply no errors occurred. To check for
+   At the end of a complete run, the summary recorded in
-   errors, look at the file "<workarea>/mats/summary", which should
+   "<workarea>/mats/summary" is shown. It should contain one line per
-   contain one line per test run, something like this:
+   test configuration, something like this:
 
      -------- o=0 --------
      -------- o=3 --------
@@ -217,7 +215,7 @@ The makefile supports several targets:
      -------- o=3 ehc=t eval=interpret --------
 
    If there is anything else in "<workarea>/mats/summary", something
-   unexpected occurred.
+   unexpected occurred. See "IMPLEMENTATION.md" for more information.
 
  * `make <machine type>.boot` or `make <machine type>.bootquick`
 

racket/src/ChezScheme/IMPLEMENTATION.md

@@ -36,6 +36,28 @@ a new platform. In particular, the `pb` (portable bytecode) machine
 type can run on any supported hardware and operating system, so having
 `pb` boot files is one way to get started in a new environment.
 
+# Compiled Files and Boot Files
+
+A Scheme file conventionally uses the suffix ".ss" and it's compiled
+form uses the suffix ".so". The format of a compiled file is closely
+related to the fasl format that is exposed by `fasl-write` and
+`fasl-read`, but you can't compile Scheme code to some value that is
+written with `fasl-write`. Instead, `compile-file` and related
+functions directly generate compiled code in a fasled form that
+includes needed linking information.
+
+A boot file, usually with the suffix ".boot", has the same format as a
+compiled file, but with an extra header that identifies it as a boot
+file and takes care of some singleton objects, such as `#!base-rtd`
+and the stub to invoke compiled code.
+
+The vfasl format is used for the same purposes as the fasl format, but
+mostly for boot files. It is always platform-specific and its content
+is very close to the form that the content will take when loaded into
+memory. It can load especially quickly with streamlined linking and
+interning of symbols and record types, especially in uncompressed
+form. The build scripts do not convert boot files to vfasl format.
+
 # Build System
 
 Chez Scheme assigns a `machine-type` name to each platform it runs on.
@@ -68,6 +90,9 @@ to generate a subdirectory with the appropriate contents to build for
 that particular machine. This is the script that "configure" runs when
 configuring for doing the build, but you can also run the "workarea"
 script on your own, supplying the machine type you'd like to build.
+The directory where you run "configure" or "workarea" is the "build
+directory", while the directory named "*machine-type*" created by
+"workarea" is the "workarea directory".
 
 Bootstrap from scratch by running the Racket program
 "rktboot/main.rkt", which should work even with a relatively old
@@ -143,32 +168,134 @@ will be compiled as unsafe. While testing and debugging your
 additions, however, you'll probably want to use `make o=0` in the
 "*machine-type*/s" workarea space, which compiles in safe mode.
 
-Tests go in "mats/*...*.ms". In "*machine-type*/mats", you can use
+# Writing and Running Tests
-`make 7.mo` to build and run `7.ms`. Remove `7.mo` to re-run without
-changing `7.ms`. Makefile variables like `o` control the way tests
-are run; for example, use `make o=3 7.mo` to test in unsafe mode.
 
-# Compiled Files and Boot Files
+A group of tests is written in a ".ms" file in the "mats" directory.
+Within a `mat` form after the name for the group of tests, each test
+is written as a expression that produces `#t` for success. Use the
+`error?` form to wrap an expression that is supposed to raise an
+exception in safe mode, but note that the test doesn't describe the
+exception specifically, since the expected error message likely
+depends on the configuration (e.g. safe versus unsafe); more on that
+below.
 
-A Scheme file conventionally uses the suffix ".ss" and it's compiled
+### Running One Set of Tests (no expected-error checking)
-form uses the suffix ".so". The format of a compiled file is closely
-related to the fasl format that is exposed by `fasl-write` and
-`fasl-read`, but you can't compile Scheme code to some value that is
-written with `fasl-write`. Instead, `compile-file` and related
-functions directly generate compiled code in a fasled form that
-includes needed linking information.
 
-A boot file, usually with the suffix ".boot", has the same format as a
+Runs tests in a ".ms" file by going to your build's
-compiled file, but with an extra header that identifies it as a boot
+"*machine-type*/mats" directory, then `make` with a ".mo" target. For
-file and takes care of some singleton objects, such as `#!base-rtd`
+example, use `make 7.mo` to build and run `7.ms`. Delete `7.mo` to run
-and the stub to invoke compiled code.
+`7.ms` again. Makefile variables like `o` control the way tests are
+run; for example, use `make 7.mo o=3` to test in unsafe mode. See the
+source file "mats/Mf-base" for information about the configuration
+options. Running tests to make a ".mo" file prints a lot of output, so
+you'll likely want to redirect stdout and stderr to a file.
 
-The vfasl format is used for the same purposes as the fasl format, but
+A test failure is recorded in a ".mo" file as a line that contains
-mostly for boot files. It is always platform-specific and its content
+`Bug`, `Error`, or `invalid memory`. That's why the target for making
-is very close to the form that the content will take when loaded into
+a ".mo" file ends by grepping the file. Tests for exceptions produce
-memory. It can load especially quickly with streamlined linking and
+the output `Expected error`, but there's not currently a way to check
-interning of symbols and record types, especially in uncompressed
+that the exception tests of an individual ".ms" file produce the
-form. The build scripts do not convert boot files to vfasl format.
+expected error message.
+
+### Running Tests in One Configuration (with expected-error checking)
+
+You can make all ".mo" files with just `make` within your build's
+"*machine-type*/mats". You can provide configuration arguments, too,
+such as `make o=3` to make all ".mo" files in unsafe mode.
+
+In this mode, output ".mo" files are written to a subdirectory that is
+partially configuration-specific, such as "compile-0-f-f-f" for
+`compile` (as opposed to `interpret`) in safe mode (`0` instead of
+`3`), without `suppress-primitive-inlining` enabled (first `f`),
+without cp0 enabled (second `f`), and without
+`compile-interpret-simple` enabled (last `f`). Note that a set of
+tests is not run again if an up-to-date ".mo" file is in the output
+directory, so use `make clean` as needed.
+
+The combination of all ".mo" error messages (from both expected
+exceptions and test failures) is compared against a list of expected
+errors messages for a configuration using `diff`. The `diff` result is
+written to "report-*config*", where *config* is the name of the
+configuration. So, an empty "report-*config*" means success.
+
+The set of expected error messages for a given configuration is
+generated by starting with either "mats/root-experr-compile-0-f-f-f"
+or "mats/root-experr-compile-3-f-f-f" (depending on whether the
+configuration is in unsafe or safe mode) and then applying some number
+of patches from "mats/patch-*config*". That's why the *config* in
+"report-*config*" doesn't identify everything about the configuration;
+it only identifies the combinations that can have different error
+output.
+
+If you add a new test that's expected to have error output (usually to
+check that an exception is correctly raised), then
+"mats/root-experr-*config*" and/or "mats/patch-*config*" files need to
+change. Modifying those files by hand is not practical. Instead, the
+strategy is to make sure that the output diff in "record-*config*" is
+correct, and then use targets like `make root-experr` and `make
+patches` to generate new "root-experr-..." and "patch-..." files:
+
+ * Run `make` and then `make root-experr` to generate a new
+   "root-experr-compile-0-f-f-f", then copy the generated file in the
+   workarea to the source "mats" directory. Often, this step is all
+   that is needed to update expected errors, since expected errors
+   tend to happen only in safe mode, and they tend not to change among
+   other configuration options.
+
+ * If you need to update "root-experr-compile-3-f-f-f", use `make
+   root-experr o=3` after running with `make o=3` and then copy the
+   file from the workarea to the "mats" source directory.
+
+ * After running tests for a configuration with `make` plus
+   configuration options, you may be able to recreate the
+   corresponding patch file using `make xpatch-*config*` with the same
+   configuration options. However, some configurations involve layers
+   of patch files, so it's tricky to get this right by running test
+   configurations one at a time, and it's better to run tests for all
+   configurations.
+
+### Running Tests for All Configurations
+
+To run tests for all configurations, use `make allx` within your
+build's "*machine-type*/mats" directory. Add `-j` followed by *N* to
+run tests using *N* parallel jobs. Using `make allx` implicitly uses
+`make clean` before it runs tests. You can also use `make test`
+directly in your build directory, since that's a shortcut for `make
+allx` in the "*machine-type*/mats" directory.
+
+To support parallel tests, `make allx` write its output in a
+collection of "output-*i*-*o*" directories within
+"*machine-type*/mats", so you can look for "report-*config*" files in
+those subdirectories. As its last step, `make allx` combines a summary
+of reports to a `summary` file directly in "*machine-type*/mats", and
+then it shows that summary as output. As long as that output shows
+only configurations (i.e., no errors), then all tests passed for all
+configurations.
+
+After running `make allx`, if the summary shows only errors that
+reflect out-of-date expectations from "root-experr-..." or "patch-..."
+files, you can use the sequence
+
+```bash
+make root-experr o=0
+make root-experr o=3
+make patches
+```
+
+to create new vesions of the files in the workarea directory. Copy
+changed files to the "mats" source directory; if the only change to a
+patch file is to the line-number hints, then it's probably not worth
+keeping the update (as long as the line numbers are not too far off).
+A `make clean` will delete any "root-experr-..." or "patch-..." files,
+and links are created on demand in the workarea space to
+"root-experr-..." or "patch-..." files that are needed to generate
+expexted-error diffs.
+
+Despite its name, `make allx` does not run all available tests. Use
+`make bullyx` to run a different, more stressfull set of tests. The
+bully tests may cover more configurations than `allx`, so `make
+patches` after `make bullyx` may pick up additional "patch-..." file
+changes.
 
 # Scheme Objects
 

racket/src/ChezScheme/mats/Mf-base

@@ -158,7 +158,7 @@ Examples = $(abspath ../examples)
 MAKEFLAGS += --no-print-directory
 
 # directory where (most) output for this run will be written
-outdir=output
+outdir=.
 
 conf = $(eval)-$o-$(spi)-$(cp0)-$(cis)
 objdir=output-$(conf)
@@ -214,6 +214,7 @@ $(objdir)/%.mo : %.ms mat.so
              '  (fprintf (console-error-port) "check heap detected errors---grep standard output for !!!\n")'\
              '  (abort))'\
            | ${Scheme} -q mat.so ${patchfile}
+	! egrep "Bug|Error|invalid memory" $*.mo
 
 %.so : %.ss
 	echo '(reset-handler abort) (time (compile-file "$*"))' | ${Scheme} -q ${patchfile}
@@ -332,8 +333,15 @@ just-reports:
 allxhelp:
 	$(MAKE) doheader
 	-$(MAKE) all
+	$(MAKE) errors-$(conf) forpatches=.
 	$(MAKE) dosummary
 
+# To support an eventual `make patches`, link `errors-$(conf)` to allxhelp output
+# if there's not already a representative for the configuration:
+forpatches = different-from-outdir
+$(forpatches)/errors-$(conf): $(outdir)/errors-$(conf)
+	ln -s $(outdir)/errors-$(conf) errors-$(conf)
+
 config-vars = spi hci ecpf cp0 cis p eval ctb cgr cmg eoc cl rmg
 full-config-str = $(strip o=$(o) $(foreach var, $(config-vars),$(if $(filter-out $($(var:%=default%)),$($(var))),$(var)=$($(var)))) $(hdrmsg))
 
@@ -471,7 +479,7 @@ experr-compile-$o-f-f-f: root-experr-compile-$o-f-f-f
 
 root-experr: # don't list dependencies!
 	rm -f root-experr-compile-$o-f-f-f
-	cp errors-compile-$o-f-f-f root-experr-compile-$o-f-f-f
+	cp $(outdir)/errors-compile-$o-f-f-f root-experr-compile-$o-f-f-f
 
 # derive spi=t experr files by patching spi=f experr files
 # cp first in case patch is empty, since patch produces an empty output