Progress toward making the bytecode compiler deterministic, so that a
fresh `make base` always produces exactly the same bytecode from the
same sources. Most changes involve avoiding hash-table order
dependencies and adjusting scope identity. The namespace used to load
a reader extension is also better defined. Plus many other little
changes.
The identity of a scope that is unmarshaled from a bytecode file now
incorporates the hash of the file, and the relative order of scopes is
preserved in a bytecode file. This combination allows compilation to
start with modules that loaded and compiled in different orders
(including delayed loading of bytecode fragments within one file).
Formerly, a reader extension triggered by `#lang` or `#reader` was
loaded in whatever namespace happens to be current. That's
unpredictable and can pollute a module build at the level of bytecode.
To help make builds deterministic, reader extensions are now loaded in
a root namespace of the current namespace.
Deterministic compilation in general relies on deterministic macros.
The two most common ways for a macro to be non-deterministic are by
using `gensym` (use `generate-temporaries`, instead) and by using an
unsorted hash-table traversal (don't do that).
At this point, bytecode generation is unlikely to be completely
deterministic, since I uncovered non-determinism mostly by iterating
attempts over the base collections. For now, the intent is not to
provide guarantees outside of the compilation of the base collections
--- but "more deterministic" is likely to be useful in the short run,
and we can improve further in the long run.
When repoting an error during compilation, always show the path to the
module being compiled. That path was sometimes available in the error
message anyway, due to source locations for syntax errors, but often
there would be no path due to run-time errors in macros, a lack of
source locations on macro-introduced forms, etc.
The `raco setup` improvements rely on new machinery in `compiler/cm`,
and `raco make` inherits that machinery.
The parallel and non-parallel variants of `raco setup` reported
excpetions in slightly different formats, and now they're consistent.
The initial report of an exception now always shows an evaluation
context, while the summary's repeat of the error omits the context.
For GC purposes, if a "prefix" (a closure frame that caprues
top-level or module-level bindings) may refer to syntax objects
that are not used by any reachable closure, in which case the
syntax object can be dropped. This pruning of syntax objects
uses the infrastructure already in place to prune variables.
Syntax objects were not included in the original pruning
implementation, because they are unlikely to create
finalization cycles in the way that global-variable
references can. A syntax object can retain a namespace's
table of module imports, however, which can be substantial
and worth releasing of a closure is only held, say, for
a low-level finalization action.
Instead of introducing a subtype of `file-dependency` to imply one new
option, add a subtype that has an options table for easier
extensibility. (Thanks to Sam for pointing out that I shouldn't make
this mistake again.)
If module M in package P imports module N from package Q,
and if N has a `lazy-require` for a module in R that is
triggered during the compilation of M, then P doesn't really
depend on R; P depends on Q, and Q depends on R, and P
shoudn't necessarily know anything about Q. At the same time,
a change to the file in R means that M must be recompiled.
So, continue to track the compilation dependency, but mark
it as "indirect" so that the package-dependency checker can
ignore the dependency.
If "sqlite3.dll" is installed as a foreign library but shouldn't
be, then `raco setup` cannot simply deleet the file, because
starting `raco setup` opened the DLL. To avoid that problem,
rename the file to start with "raco-setup-delete-", then attempt to
delete the renamed file; the delete won't work, but the file
will be moved out of the way, and a future `raco setup` can
clean up.
The prefix "raco-setup-delete-" thus becomes special on Windows for
the directories that hold foreign libraries, shared files, and
man pages, because `raco setup` will try to delete any file
that starts with "raco-setup-delete-".
It's all very ugly, but I don't have a better idea for the
problems that I keep hitting.
Restore (but in a hopefully better way) a step that installs native
libraries before trying a full `raco setup`, since the libraries
may be needed for the setup proces --- especially on Windows.
