In `syntax-local-lift-require`, avoid scope adjustments intended
to deal with `require` forms that are compiled in one namespace
and evaluated in another.
When `or` has many subexpressions, the expansion generates a
sequence of deeply nested `let`s, where original and macro-introduced
forms are interleaved in a way that defeats a minimal
child-is-same-as-parent sharing of scope sets. Add a small
cache that's good enough to capture extra sharing and
dramatically lower memory use for an `or` that has 1000
subexpressions.
When an import is shadowed by another import or by a definition, don't
include it in the set of bindings in the resut of
`syntax-local-module-required-identifiers` or in the set that can be
exported by `all-from-out`.
Merge to v6.3
To make the API consistent for MSVC versus MinGW builds, make
a functional formerly required for embedding on 32-bit Windows
always available and required for all Windows variants.
Recent versions of MinGW-W64 use emutls for `__thread` variables,
and that's much slower than Windows-native TLS. Go back to the
inline-assembly implementation of therad-local access.
Make the old-generation marking process incremental
on request, where `(collect-garbage 'incremental)`
makes a request.
Only the marking phase of an old-generation collection
is incremental, so far. In exchange for slower
minor collections and a larger heap, you get a major
collection pause time that is roughly halved. So, this
is a step forward, but not good enough for most purposes
that need incremental collection.
An incremental-mode request sticks until the next
major GC. The idea is that any program that could
benefit from incremental collection will have
some sort of periodic task where it can naturally
request incremental mode. (In particular, that
request belongs in the program, not in some external
flag to the runtime system.) Otherwise, the
system should revert to non-incremental mode, given
that incremental mode is slower overall and can
use much more memory --- usually within a factor of
two, but the factor can be much worse due to
fragmentation.
For a minor GC and pages that contain backpointers,
leave mark bits as they are; instead make a pass to
shift mark bits for new objects to "dead" bits, and
use dead bits for fixup.
This change is intended as a small step toward incremental
collection.
The rule for using generation 1/2is based on the current
memory use versus the maximum size of generation 0. Recent
changes to the GC have caused that size to vary during
a collection, which means that the choice to use generation
1/2 or not can change within a collection.
Partial use of generation 1/2 doesn't inherently cause problems, but
it can cause a generation-1 object to point to a generation-1/2 object
even though the former was allocated after the latter. That's a
problem on if getting generations out of order relative to allocation
order can create problems. As it happens, reset_finalizer_tree()
checks the generation of the finalization record and not the finalized
pointer, because the record is always allocated after the pointer.
Merge to v6.3
Certain datatypes in the runtime system are not supposed
to be hashed, where bits normally reserved for hash codes
are used for other purposes. A bad bytecode file can cause
some of those to be hashed, anyway. Normally, the damage is
isolated to that content of the damaged bytecode, but
certain variable-reference bytecode forms are both shared
and non-hashable. Set a bit that ensures hashing will not
change flags in the shared object.
This problem was exposed by fuzz testing.
When a compiler is run in standards mode, predefined macros that
do not start with "_" are dropped, so use the "_" versions
consistently. Whether or not Racket itself would compile in
standards mode, the Racket headers should be able to work that
way --- at least on Unix platforms.
In Mac OS X 10.11, something about the use of exceptions triggers
a libunwind stack traversal, and that traversal runs into trouble
with Racket's stack mangling for threads. Inserting generated code
in the stack frame sequence causes libunwind to give up and avoids
a crash (e.g., with `-j -l drracket` on startup).
After some expansions, a expression with the syntax property 'inferred-name of
'x is converted to one with ('x . 'x), so it's not useful to get the name of a
procedure. So we simplify the syntax property 'inferred-name to handle
these cases.
When a place message is deserialized by simply adopting the page
containing the message, the adoption can trigger a garbage
collection, but there's still a pointer to a chain of objects
"in flight" in the thread, and a GC can discard the pairs that
form the chain.
Removing all original module context doesn't work, because it
doesn't distinguish between fragments of syntax that had the
"inside-edge" scope without the "outside-edge" scope.
Record the presence of the outside-edge scope by using the
root scope, and convert the root scope to the current namespace's
outside-edge scope on evaluation.
The bug could cause
#lang racket/base
(define x 'outer)
(define-syntax-rule (def-and-use-m given-x)
(begin
(define-syntax-rule (m)
(let ()
(define given-x 'inner)
x))
(m)))
(def-and-use-m x)
to produce 'inner when it should produce 'outer.
Thanks to Brian Mastenbrook for pointing the problem and
providing examples.
Interrupting bytecode unmarshal for syntax objects could leave
half-constructed values in a table that is intended to resolve graph
structure. Clear out work towards a graph construction when
interrupted.
The most common symptom of half-constructed syntax objects was a crash
after a Ctl-C during startup.
Rename fields in a page record and split some of them with `union` to
better document the intent of each field.
This change is intended to have no effect on the GC's behavior. One
tricky case is the line dropped around line 3542 of "newgc.c". That
line reset `scan_boundary` (formerly `previous_size`), which on the
surface is inconsistent with leving objects before the boundary
without `marked` bits set. However, that line is reachable only
when geneation-1 objects are being marked (objects newly moved
there would not be unmarked), in which case `san_boundary` should
already be reset.
Using `--enable-racket=auto` causes a Racket for the current platform
to be built in a "local" subdirectory of the build directory as
support for cross-compilation.
The original idea was to count phantom bytes as "administrative
overhead", but issues discussed in #962 identified problems
with that idea. Finish shifting the accounting to treat
phantom bytes as payload allocation.
The misplacement of `SCHEME_PRIM_SOMETIMES_INLINED` caused the
optimizer to produce different results when the JIT is statically
disabled, for example.
This bug is an old one, in a sense, because travesing fields
in a closure could have moved the prefix with earlier versions
of the collector. It shows up now because we're changing fields
one indirection closer.
Compact fewer blocks by moving them only when other
blocks have room.
Also, fix block protection tracking in the case of a page
count that isn't divisible by 8.
In the common case of a minor GC without a generation 1/2
or a major GC without compaction, a single pass suffices
to both mark and update references.
This change reduces overall GC time by 10%-25% on typical
programs.
The GC supported allocation for an array of objects where
the first one provides a tag, but at this point it was
used only in some corners. Change those corner and simplify
the GC by removing support for arrays of tagged objects.
The main corner to clean up is in the handling of a macro-expansion
observer and inferred names. Move those into the compile-time
environment. It's possible that name inference has been
broken by the changes, but in addition to passing the tests,
the generated bytecode for the base collections is exactly the
same as before the change.
Although a block cache is set up to group most page-protection changes
into a single OS call, allocating new old-generation pages was not
covered. Adjust the block cache to group those.
This change has a small effect on performance, but it seems
better to have a few system calls in place of thousands.
First bug:
When the optimize converts
(let-values ([(X ...) (values M ...)])
....)
to
(let ([X M] ...)
....)
it incorrectly attached a virtual timestamp to each "[X M]" binding
that corresponds to the timestamp after the whole `(values M ...)`.
The solution is to approximate tracking the timestamp for invidual
expressions.
Second bug:
The compiler could reorder a continuation-capturing expression past
an allocation.
The solution is to track allocations with a new virtual clock.
Make `eval-syntax`, `compile-syntax`, and `expand-syntax` more
consistent (with intent and each other) by not installing a fallback
automatically. In particular, a fallback is not installed for a
`module` form, so that different ways of expanding a `module` form
produce consistent results (e.g., for ambiguous bindings).
Putting <PlatformToolset> in the new places makes the projects
work when more than one version of Visual Studio is installed.
Maybe the old place was always the wrong place, or maybe
VS 2010 wanted it in the old place. Either way, sprinkling
the version in more places seems unlikely to hurt.
User-scope package installation matching the version of
Racket being built could affect the collections visible
during `raco setup` for `make base`. In particular, the
presence of `setup/scribble` could cause all built docs
to be discarded.
Also, add the `--no-user-path` flag to `racket` (which
has long been documented as an alias for `-U`).
The table as a tree is traversed to prune empty branches,
but the travseral is needed only toward branches that
have changed. Skipping the traversal can save several
milliseconds on each collection.
Name handling formerly interned symbols along the
way to allocating a plain string, which takes effort
and causes changes to the symbol table, which forces
a minor GC to traverse the whole symbol table. Skip
unnecessary symbol-interning steps.
Refine the changes in 16c198805b so that `(define id ... id ... )` at
the top level compiles more consistently when `id` is an identifier
whose lexical context does not include `#%top`.
When `compile` is used on a top-level definition, do not
create a binding in the current namespace, but arrange for
a suitable binding to be in place for the target namespace.
Closes#1036
This repair adjusts the bug fix of commit 769ad3e98. That older commit
ensured that `sync/enable-break` doesn't both break and accept a
channel message or semaphore wait. But it effectively disables those
actions if the break is continued.
Instead of (partially!) ending the `sync` get out of semaphore
and channel queues so that no event can be selected during
the break, and then get back in line if the break is continued.
When a path is made relative for marshaling to bytecode, record
a list of byte strings in stead of a platform-specific relative
path.
For syntax-object source locations, convert any non-relative path to a
string that shows just the last couple of path elements preceded by
".../". This conversion avoids embedding absolute paths in bytecode,
but at the cost of some information. A more complete and consistent
solution would invove using a module-path index instead of a path, but
that would be a big change at several layers.
Make room in the bytecode format for source locations and 'paren-shape
property values for syntax objects. Saving source locations increases
bytecode size by about 10% on average.
Also, convert the internal representation of syntax properties to
use immutable hash tables, instead of lists.
The `prop:expansion-contexts` property can control the expansion
of a rename transformer in much the same that conditionals on
`(syntax-local-context)` can control the expansion of other
transformers.
All places uses the same accounting bit for objects
that are in the shared space. Each place also flips
the bit value it wants on each accounting, so if two
places are accounting at the same time with opposite
bit values and can reach the same objects, they can
interefere. It's even possible for them to race
through cycles and cause each other to loop forever.
Add a lock to ensure that there's only one bit value
in play for the shared space at any given time. A
place must stall if other places are busy with memory
accounting and an opposite bit value.
While a place message is received by a thread but not yet
deserialized, if the message contains references to objects in the
shared space, and if a "master" GC happens (which crosses all places),
make sure that the references in the still-serialized message are
traversed.
Adjust installation tools to support cross-installation (i.e.,
installation for a platform other than the current one) as triggered
by "system.rktd" in "lib" having different information than the
running Racket executable.
Also, change floating-point handling to be like the MSVC build by
default, where the process is left in double-precision mode and
the mode is changed for exfl operations.
Includes repairs for integer-size mismatches in uses of Windows
threads.
The error message for the guard used an incorrect contract.
Also removed an unused line that allows a box value in the
property. I don't think it was possible to trigger this line
anyway because of the dynamic check.
In a case like
(let-values ([(X ...) (with-continuation-mark M_k M_v
(values M ...))])
....)
where the bytecode compiler cannot convert to a sequence of `let`
bindings, make the JIT implement `values` as delivering argument
results directly to the corresponding variable locations.
Progress toward making the bytecode compiler deterministic, so that a
fresh `make base` always produces exactly the same bytecode from the
same sources. Most changes involve avoiding hash-table order
dependencies and adjusting scope identity. The namespace used to load
a reader extension is also better defined. Plus many other little
changes.
The identity of a scope that is unmarshaled from a bytecode file now
incorporates the hash of the file, and the relative order of scopes is
preserved in a bytecode file. This combination allows compilation to
start with modules that loaded and compiled in different orders
(including delayed loading of bytecode fragments within one file).
Formerly, a reader extension triggered by `#lang` or `#reader` was
loaded in whatever namespace happens to be current. That's
unpredictable and can pollute a module build at the level of bytecode.
To help make builds deterministic, reader extensions are now loaded in
a root namespace of the current namespace.
Deterministic compilation in general relies on deterministic macros.
The two most common ways for a macro to be non-deterministic are by
using `gensym` (use `generate-temporaries`, instead) and by using an
unsorted hash-table traversal (don't do that).
At this point, bytecode generation is unlikely to be completely
deterministic, since I uncovered non-determinism mostly by iterating
attempts over the base collections. For now, the intent is not to
provide guarantees outside of the compilation of the base collections
--- but "more deterministic" is likely to be useful in the short run,
and we can improve further in the long run.
Specialize a
(call-with-immediate-continuation-mark _key (lambda (_arg) _body) _def-val)
call to an internal
(with-immediate-continuation-mark [_arg (#%immediate _key _def_val)] _body)
form, which avoids a closure allocation and more.
This optimization is useful for contracts, which use
`call-with-immediate-continuation-mark` to avoid redundant
contract checks.
On OS X, it seems that access() can sometimes fail with EPERM
when checking for execute permission on a file without it.
I've previously seen this result when running as the superuser,
but that's apparently not the only possibility; a long path
may also be relevant.
Re-linking in a new namespace doesn't need the namespace of
compilation.
A "namespac.rktl" test exposed this problem, where the "transfer a
definition of a macro-introduced variable" test could fail if a GC
occurred between compilation in one namespace and evaluation in
another.
Although `eval-syntax` is not supposed to add the current namespace's
"outer edge" scope, it must add the "inner edge" scope to be consistent
with adding the inner edge to every intermediate expansion (as in
other definition contexts).
In addition, `eval`, `eval-syntax`, `expand`, and `expand-syntax`
did not cooperate properly with `local-expand` on the inner edge.
Some failure paths were missing an update before calling failure
code, and the new failure paths need to unconditionally update the
runstack pointer (because the common stub doesn't know whether the
calling context needs an update).
Genereating a use-site scope, instead of a macro-introduction scope,
prevents the scope's presense from triggering a #f result from
`syntax-original?`.
This change mostly reverts 1465ff25fc, which turned out to be a hassle
because it created more cyclic structure.
A simpler strategy is to allow a phase-specific scope to be detached
(perhaps temporarily, due to on-demand loading of bytecode) from its
group; when that's possible, the scope is not reachable from a place
where it can be moved to other syntax objects, so it's ok to be
detached. Debugging output needs to handle that gracefully, though.
Also, in case of broken bytecode, fix up a detached scope if it
does end up in an unexpected place.
Formerly, compiling a definition in one namespace and evaluating it in
another would cause the definition to take place in the original
namespace --- unless the compiled code is marshaled to a byte string
and back. Adjust the "linking" process to redirect the variable
definition and any references to the new namespace. (This is a change
relative to the compiler with the old macro expander.)
Also, repair a compiled `require` form along similar lines. (This is
*not* a change relative to the compiler with the old macro expander;
the mismatch is part of the motivation for changing `define`
handling.)
Add the current definition context's scope to any expression that is
produced by macro expansion before trying to expand again, in case the
expansion needs to refer to a definition introduced by a previous
expansion.
Previously, the scope was added before any expansion and after any
expansion, but that misses intermediate points.
The old expander had this bug, too (some of the new tests fail there),
but it showed up less often and was sometimes considered correct, for
various reasons.
I had tried to simplify the "generation 0" allocation function to
always use `GEN0_PAGE_SIZE`, but "generation 0" is also used for place
messages, in which case a much smaller size should be used.
The "place-in-channel-fnl.rkt" test exposed this problem.
A recent GC change (included with the set-of-scopes expander)
allows the GCs marking procedure to recur directly to a limited
depth, instead of always pushing pointers onto a stack. Direct
recursion is not cmopatible with ephemeron-resolution process,
so switch to no-recur mode.
This problem was uncovered by an existing test.
The combination of splitting a `letrec` and optimizing
the resulting `(let ([x <proc>]) x)` to just `<proc>`
used a bad coordinate shift, which made property testing
incorrect, etc.
For reasons that are not clear, the new expander triggered
the problem through an existing test.
The `eval-syntax` function (which is used by other functions, such as
loading a module) should not install fallback-binding scopes from
the current namespace.
When `(let ([x ...]) (let ([y x]) ... y ... y ...))` turns into
`(let ([x ...]) ... x ... x ...)`, make sure that `x` is not
still marked as single-use. Incorrect marking as single-use could
cause the optimizer to inline too much, for example.
Thanks to Gustavo for tracking down the problem.
Previously all the predicates recognized only non-#f things, so ´not´ can be
added to the list of disjoint predicates. But many of the parts of the code
relied on the non-#f property and had to be modified.
In (if (eq? x <pred?-expr>) <tbranch> <fbranch>) infer that the type of
x is pred? in the tbranch.
Also, reduce (eq? x y) => #f when the types are different.
The optimizer reduces the variables with a known type to #t in a Boolean context.
But some predicates imply that the variable has a definite values, so they can be
reduced in a non-Boolean context too.
For example, in (lambda (x) (if (null? x) x 0))) reduce the last x ==> null.
This fixes the bug twice:
* Don't reduce mutable variables with a type to #t in a Boolean context.
* Don't record the type of mutable variables when a predicate is
checked in a test condition.
While reducing some ignored constructors, the optimizer may wrap the arguments
<expr> in (values <expr>) to ensure that it's a single value non-cm expression.
This avoids the unnecessary nesting of (values (values <expr>)).
Similarly, add the cases for begin and begin0 to single_valued_noncm_expression
While `#:in-original-place? #t` provides one way to serialize
foreign calls, it acts as a single lock and requires expensive
context switches. Using an explicit lock can be more efficient
for serializing calls across different places.
For example, running "plot.scrbl" takes 70 seconds on my machine
in the original place and using `#:lock-name` in any place,
while it took 162 seconds in a non-main place with Cairo+Pango
serialization via `#:in-original-place? #t`.
Internally, the named lock combines compare-and-swap with a
place channel. That strategy gives good performance in the case
of no contention, and it cooperates properly with the Racket
scheduler where there is contention.
The optimizer was able to use the type information gained outside
the let's to reduce expressions inside the lets. For example, in
(lambda (z) (car z) (let ([o (random)]) (pair? z)))
it reduces (pair? z) ==> #t.
This enable the propagation in the other direction so in
(lambda (z) (let ([o (random)]) (car z)) (pair? z))
it reduces (pair? z) ==> #t too.
Using `(thread-resume t1 t2)` would not prevent a GC of t1, but it
would create an intermediate record to make the link from t1 to t2,
and that intermediate record would leak due to a missing level of
indirection in a table-cleanup traveral. The leak not only accumulated
memory, it also caused ever slower traversals of the table in an
attempt to clean up.
(Since the leak is small and the leaking object is not directly
accessible, I don't have a good idea on how to test this repair
automatically, but see the program in the PR.)
Closes PR 15099.
Modern OS configurations likely use an even larger buffer size, and
making it small can have substantial negative performance effects
(e.g., with PostgreSQL over TCP).
When AC_PROG_CC picks GCC, move its selection of CFLAGS
into CPPFLAGS, so that preprocessing will have the same
optimization and debugging flags as compilation.
Arguably, AC_PROG_CC plus AC_PROG_CPP should do that
soemhow, but it's understandable that the autoconf
implementers didn't cover the possibility of
preprocessing that changes with the optimization level.
Closes#945
When `local-require` is used in a non-phase-0 position and it is
`expand`ed (as opposed to compiled directly), then the generated
`#%require` form had the wrong binding phase.
Merge to v6.2
In many use cases the length of the vector is fixed and know,
so we are sure that make-vector will not raise an error and
we can recognize these expressions as omittable and drop
them when the result is ignored.
The result of some procedures is a vector, but they are not omittable
because they may rise an error. With the recent changes of the
predicate reduction these cases are correctly handled.
The optimizer checks the type of the argument of some unary procedures and
uses the gathered information to replace them by the unsafe version, reduce
predicates and detect type errors. This extends the checks to more procedures
that have no unsafe version and procedures that have more than one argument.
Use the given readtable more consistently to parse
delimiters in the top-level form. This change particularly
addresses problems with trying to restore the original
`(` when parsing a hash table, but allowing nested
forms to still use a different `(` mapping.
When determing whether expressions can be reordered, a reference to a
module-defined variable was considered unreorderable when it is
known to have a value and no further mutation, but the value isn't
constant across all runs.
