When upgrading native-library versions, a still-relevant
patch got lost. The patch corrects a problem with empty
glyphs getting treated as missing glyphs.
Closesracket/pict#42
The pattern
(ephemeron-value
(hash-ref! intern key
(lambda ()
(make-ephemeron key (wrap key)))))
is wrong, because a GC might happen between the time that the
epehemeron is found in the table (or the time that the key was just
added to the table) and the time that `ephemeron-value` is called to
extract the value. If the key is not otherwise accessible, the value
may no longer be in the ephemeron.
If `make cs` is run without specifying a SCHEME_SRC, then make sure
that `configure` and `make` are re-run in the Chez Scheme checkout,
in case it was updated.
A collection can only invoke certain callabcks (e.g., for DrRacket's
GC icon) when the collection is performed in the main thread. Also,
delay posting GC logging events to receivers that cannot work at
interrupt time.
This reverts commit 41fd4f3a5e.
The problems this change was intended to solve can be solved in other
ways, without loosening guarantees about expansion order. See the
discussion in #2154 for more details.
closes#2154
The procsses-based build was technically broken with the addition of a
"prefetch" thread (some time back) to improve parallelism, because the
`write`-based implementation of messages did not protect again
interleaving by different threads. The problem turns out to be easier
to expose when running with RacketCS.
Formerly, `--scope-dir` would include only the specified directory in
the search path for already installed packages, etc., which means that
it would only work right as a kind of installation scope that is a
step beyond "installation" on the "user"-to-"installation" spectrum.
The `'pkgs-search-dirs` confiugration entry, meanwhile, provides more
control over search ordering in installation scope. Make `--scope-dir`
work more consistently with that search-path configration.
This change also makes "instllation"-scope operations use the search
path more consistently, since some actions used to use the whole
search list while others pruned any prefix before the main
installation directory in the search list.
Add a way to declare an Objective-C method call as blocking in the
sense of the `#:blocking?` argument to `_cprocedure`.
Usingf `with-blocking-tell` should allow the Cocoa backend for
`racket/gui` to wait for events in the main place without blocking
other places.
Fix various problems with the implementation of places, and let
`processor-count` return the actual number of processors. A parallel
build via `raco setup` seems to work but not scale well.
Instead of defining `rktio_fd_t` to be independent of a `rktio_t`,
which isnt quite true, introduce `rktio_fd_detach` and
`rktio_fd_attach` functions to make a transfer explicit, such as when
a file descriptor goes through a place channel.
This adjustment avoids a corner case in cleaning up a file descriptor
from an abandoned channel, where the finalizer might run in a Chez
Scheme thread that is not associated with any place.
When a single-use function is inlined late enough in the optimization
process, and when the body has the only use of some variable computed
by an expression that can be move in place of the use in the inlined
function (but not in the non-inlined function), then it's a problem if
the function binding isn't pruned away early enough. Make sure the
binding to the function is marked as unused after the function is
inlined.
This bug was exposed by a recent change to the "dssl2" package.
Implement place channels and messages, and change `place-enabled?` to
claim that places are enabled, but `processor-count` still reports 1.
The implementation of place channels has an interesting use of
ephemerons --- that is, a use that isn't just solving a key-in-value
problem. Using epehemerons solves the problem of forgetting a place
channel and any thread blocked on the read end when there are no
producers on the write end. Along similar lines, when only the write
end is retained (i.e., no readers), the channel data is forgotten and
writes become a no-op. The read end holds a "read key" and references
the channel data through an ephemeron keyed by a "write key"; the
write end similarly holds a "write key" and uses an ephemeron keyed by
the "read key". This use of an epehemeron implements a reachability
"and": retain the place-channel data only if the read end *and* the
write end are both reachable. (Minor point: a read end also holds onto
the "write key" anytime the channel already has data.)
The Rumble layer provides a primitive `fork-place` operation and
`start-place` hook to tie in place initialization from other layers
(especially "io"), but the rest can live in the "thread" layer,
especially to handle place-channel synchronization.
When a stand-alone executable created by `raco exe` needs to load
modules that start with a `#lang` line, there have been various
obstacles to adding the right run-time support via `++lib`. The
`++lang` flag addresses those problems and makes it easy to indicate
that enough should be embedded to support loading modules with a
specified language.
There are problems in the way that various handlers interact for the
"lang/reader.rkt" versus `(submod "." reader)` search path that
converts a language name to a reader. To accomodate the search in a
standalone executable (that does not provide access to collections in
general), the module name resolver must refrain from raising an
exception for a non-existent submodule path that refers to a
non-existent collection.
There's no place-channel communication yet --- just enough of a
conversion to thread-load storage to make places possible.
In contrast to traditional Racket, where the expander linklet is
instantiated once per place, the flattened expander linklet is
instantiated only once in RacketCS (because it's inlined into a Chez
Scheme library). The expander therefore needs to keep per-place state
separate, and the same for the thread, io, and regexp laters.
In the expander/thread/io/regexp source, place-local state is put in
an unsafe place-local cell. For traditional Racket, a place-local cell
is just a box. For RacketCS, the thread through expander layers are
compiled in a way that maps each cell to a fixed index in a vector
that is stored in a virtual register, so the value is roughly two
pointer indirections away (thread context -> virtual register array ->
place-local vector). Multiple Chez Scheme threads in a place, such as
threads to run futures, share the same place-local vector.
Although `place-enabled?` reports #f, `dynamic-place` from `'#%place`
can create a place as a Chez Scheme thread and load a module there.
An extra scope is needed to separate the bindings of a `letrec` from
the `letrec` body, in case a macro moves right-hand-side expressions
to the body.
Michael Ballantyne and William Hatch reported this problem and its
solution in December 2016, but I forgot to add the repair.
Relevant to #2237
The repair in 7176fc4253 did not make the no-discard decision stick
well enough for some cases. Robby found this bug using the Redex model
and random testing, too.
Using a parameter for the current expansion context means that if a
macro spawns a thread, the thread thinks that it's in an expansion
context. Switching to a raw continuation mark avoids that problem.
Along the way, bring Racket and RacketCS more in line by making both
have an internal notion of "root" prompt tag that can be used to get
all continuation marks independent of any prompts. That's not structly
necessary, since a continuation mark could be combined with a distinct
tag to make the mark always accessible, but it's simpler and more
lightweight to use a root prompt tag.
This improvement removes only a few places where a variable use is
considered possibly too early in the RacketCS implementation, but
the improvement is significant for uses of `input-port?` in "io".
Allow a single argument to comparison functions like `<`, and
support the same arities as the generic version for fixnum and
flonum operations like `fx+` or `fl+`.
RacketCS weak `equal?`-based hash tables didn't retain flonums, and
hashing of hash tables was not properly insensitive to the order of
keys.
Racket, meanwhile, didn't limit work consistently for different kinds
of hash tables, and it didn't keep a counter value odd as intended
(but the counter never gets large enough to appear to be a mapped
pointer, anyway).
Racket did not check for a break when escaping from a break-disabled
context to a break-enabled context. RacketCS didn't check in other
cases, either. Fix those various cases.
Commit 1afcbee381 effctively dropped a conditional case that was
marked as "shouldn't happen" --- but it does happen and makes sense.
Adjust the replacement `delete-directory/files` call to accomodate the
case.
Relevant to #2198 and #2236
Part of this change restores a `++direct` that was lost in 98ae91e0ba
for "racket/src/thread" to make the atomicity state a virtual
register. Also make `display` on a byte string more directly call
`write-bytes`. That change restores a 5-10% speed improvement for
`racketcs -cl racket/base`.
Change the expander-performance macro so that it's a very low cost if
not enabled on startup. An extra JIT specialization reduces the cost
further, since the enabled state is known by JIT time.
When a child place terminates, the parent's count of the child's
memory use needs to be updated. Until this repair, the
`current-memory-use` function has been reporting an incorrectly large
number after a place terminates.
Fix liveness for "simple" arguments to inlined functions. Fix
handling of non-authrntic structure access and mutation to
allow the possibility of a GC.
Also, while we're at it, make the functions produced by curry cooperate
better with other parts of Racket. Namely, make the information reported
by procedure-arity and procedure-keywords accurate, and give procedures
more useful dynamic names.
The mask encoding of an arity is often easier to test and manipulate,
and masked-based functions are sometimes faster than functions that
used the old arity representation (while always being at least as
fast).
Attempting to assign an arity like `(expt 2 100)` to `(lambda x x)`
won't work anymore; it will raise an out-of-memory exception, because
the arity is represented internally as a mask. The arities that cannot
be represented aren't sensible arities, anyway.
The main performance improvement is in calling a function returned by
`procedure-{reduce-arity,rename}` when the arity is not a single
integer. Calls to functions with > 29 arguments can be worse, but
that seems like a much rarer case.
Forcing JIT code generation through an environment variable is useful
to get a sense of how much machine code is generated for a program.
Setting `PLT_LINKLET_TIMES` causes the overall memory used by
JIT-generated code (including adminstrative overhead) to be printed on
exit.
In pass 1, syntax class defns (without #:attributes decl) need to parse
patterns to determine exported attributes. But to allow forward refs,
stxclass references are not resolved until pass 2.
Previously, this was done by just reparsing patterns in pass 2. But that
means pattern expanders get expanded twice, and their expansions might
not agree (eg generate-temporaries). So instead, parse in pass 1, insert
"fixup" patterns to delay stxclass resolution, and resolve fixups in pass 2.
Complication: a fixup pattern is assumed S, but can change to H in pass 2.
The actual problem is that build-chaperone-contract-property
exported to the user defaults #:exercise and #:generate
to false. This commit changes the default fallback value
in the case where #:generate is not a procedure instead
of changing build-chaperone-contract-property directly
to stay consistent with the how contract-struct-exercise
currently does it.
The default values updated in commit ffc5720b5 do
not work for very subtle reasons. In build-contract
in racket/contract/private/prop, the default values
should not accept an extra ctc argument since ctc
is already handled by make-flat-contract. The
default gen procedure should also be (λ (fuel) #f)
instead of (λ (ctc) (λ () #f)) since the latter
would generate false when the generation should
have failed. In build-property, the default procedure
(λ (ctc) (λ (fuel) #f)) is correct and should not
be changed to (λ (ctc) (λ () #f)).
Improve the `hash-ref` error message when the failure result does not
accept zero arguments. (This only changes what the error messages says.)
Example:
```(hash-ref #hash() 'a add1)```
Old message:
```
; add1: arity mismatch;
; the expected number of arguments does not match the given number
; expected: 1
; given: 0
```
New message:
```
; hash-ref: contract violation
; expected: (-> any)
; given: #<procedure:add1>
; argument position: 3rd
```
When a `define` that shadows a `require` appears before the `require`,
then the `require` may fail to other, non-shadowed bindings from the
same `require` spec.
Thanks to Matthias for reporting the problem.
The new argument to `hash-iterate-value` and most other
`...-hash-iterate-...` functions determines a result to be returned in
place of raising a "bad index" exception.
For most kinds of hash tables, a "bad index" exception will only
happen when the provided index is wrong or when a hash table is
mutated during an iteration. Mutation during iteration is generally a
bad idea, but in the case of a weak hash table, a potential background
mutation by the garbage collector is difficult to suppress or ignore.
Adding an option to control bad-index behavior makes it easier to
write loops that defend against uncooperative tables, including loops
where a hash-table key disappears asynchronously.
Racket's printer was already using this functionality internally, so
the change to `hash-iterate-value` and company mostly exposes existing
functionality.
The `in-hash` form and related sequence constructors similarly support
a bad-index alternate value so iterations can handle that case
explicitly. They do not use the new bad-index support implicitly to
skip missing entries, because that idea does not play well with the
iteration API. A hash-table index can go bad after `in-hash` has
selected the index and determined that it should be used for the next
iteration, and a sequence can't take back that decision.
The `ffi/unsafe/collect-callback` library exposes functionality
formerly only available via Racket's C interface, but implement
it for both Racket and RacketCs.
Make `map` inline again, which involves optimizing away
(variable-reference-from-unsafe? (#%variable-reference))
early enough. Fix post-schemify optimization for `procedure?
by adding both forms of an import name to the `imports` table.
Fix a problem with inlining operations passed to an inlined
function (as reflected by the addition of `find-known+import`).
At phase 1 and higher, the expander tentatively allows an unbound
identifier so that, for example, `define-for-syntax` can define a
helper function syntactically after a compile-time expression that
uses the helper. While unbound references eventually trigger an error,
the reordering can be consuing, as in the example
#lang racket
(define-syntax (f stx)
(syntax-parse stx
[(_ oops) #'ok]))
which complains about `_` when the real problem is that `syntax-parse`
isn't imported.
To provide better errors, `raise-syntax-error` now implicitly extends
an error message to include a list of previously encountered unbound
identifiersin the current compilation unit. That list will be
non-empty only at phase >= 1. With that change, the error message for the
above example is
bad.rkt:5:5: _: wildcard not allowed as an expression
after encountering unbound identifier (which is possibly the real problem):
syntax-parse
in: (_ oops)
....
Closes#2167
The expander's output normally uses a distinct symbolic name for every
distinct binding within a linklet. That property is useful for
consumers like schemify, but it's counterproductive for minimizing the
diff in changes to "startup.inc", since the traditional Racket
compiler doesn't need that guarantee. Use `--local-rename` to generate
"startup.inc", which should make future diffs smaller and more
composable after changes to the expander.
* Fix fasl bug in Racket 7.0 beta
The following program causes racket to error in Racket 7.
(parameterize ([current-write-relative-directory (build-path "/" "a")])
(s-exp->fasl (build-path "/")))
This bug appears to have been introduced in Racket 7, and not in
Racket 6.x.
* Fix another bug where 'same was put through path-element->bytes
* "/" => (car (root-paths-list))
This is for windows where simply "/" is not a complete path.
* Add similar tests to serialize library.
* Better error message when relative-directory is a bad pair
Before it would give an internal list-tail error, now it returns
a proper bad argument error.
* Better tests, and improved common case
Now that `ffi/unsafe/alloc` deallocations are triggered by a place
exit, it's more likely that an ffi-call lock can be contended during a
place's termination. When that happens, the place cannot cooperate as
usual. Accomodate this rare situation by spinning.
The repair in 49a90ba75e reorders two lines in a way that, in
retrospect, seems worrying. I can't construct an example that goes
wrong, so maybe it's fine, but it seems possible (now or with some
future change) that attempting to visit available modules could lead
to the same attempt in the same thread and therefore a loop.
This commit changes the repair to just always take the lock instead of
fixing up the attempted shortcut. There's a tiny extra cost to always
taking the lock, but that extra cost seems like a better choice
overall.
If interrupts are disabled to prevent thread swaps, then don't react
to `end-atomic` by performing a deferred swap. That might happen
with logger callbacks after a GC where a deferred action was
overlooked due to a rare race condition.
Now that Chez Scheme supports libraries in boot files, make
"racket.so" work when it is converted to a boot file. Loading it as a
boot file can save around 20ms on every major GC, since the
"racket.so" code will be in the static generation.
For now, however, `racketcs` is not set up to use "racket.so" in boot
form.
* Relative paths should still be readable.
The resent PR to enable relative serialization resulted in serialzed
objects that weren't actually readable (containing literal path
elements). This PR converts them to bytes.
* Move from serialize to relative path.
Also change path->bytes to path-element->bytes
* Path elements can also be 'up and 'same.
Also merge in the relevant code from racket/fasl.
Use an association list instead of an eq hashtable. This choice is
compatible with assumptions in traditional Racket (i.e., that the
number of mark keys per continuation frame will be small) and cuts
about 1/4 of the time in a benchmark like
(define f
((contract (-> (-> integer? integer?))
(λ () values)
'pos 'neg)))
(time
(for ([x (in-range 1000000)])
(f x)))
* Add allow the binder in prop:serialize to be a procedure.
This procedure is evaluated at serialize time, and is useful if
the deserializer is not known during object-type creation time,
but is during serialize time.
* Add docs+tests.
* Add a history note.
* Add option to create relative paths for 'serialize'
Serialize would previously always create complete byte-string paths.
This adds an optional parameter to serialize (#:relative-to) to enable
relative path creation.
Now when deserialize finds a relative path, it resolves it with
respect to `current-load-relative-directory`.
* Moved fasl's path<->relative-path-elements functions
I moved it into the private/relative-path module, so that serialize
can make use of it.
* Update serialize to use relative-path.
* Add tests.
* And update docs.
When expanding a `(module* _name #f ...)` submodule, accumulate all
module scopes on the `#f` and use the `#f` for a reexpansion.
Attempting to start each time from the enclosing module's scope loses
scopes that were generated from previous expansions. One consequence
of losind a scope is that an original definition may appear to be
macro-introduced, and the defined variable may become inaccessible
in the module's namespace.
Generating the code for a `_fun` type takes hundreds ot thousands
of times as long as in the traditional Racket VM, to cache results
to reduce the cost.
Further correct the implementation of `variable-reference-constant?`
on bindings to primitive variable.
This repair affects method-call ctype caching in `ffi/unsafe/objc`.
Add some logging there to make problems easier to detect. Also,
add and improve linklet-evel performance logging for comparing
the traditional Racket VM to Racket-on-Chez.
When setting up the namespaces that imitate primitive instances,
the "constant" annotation wasn't set. The v6.x expander gets this
wrong, too, for different reasons.
Make schemify inline structure accessors and mutators across linklet
boundaries --- or, in JIT mode, across function boundaries --- by
replacing an accessor or mutator with a `#%record?` test and
`unsafe-struct*-{ref,set!}` operation.
When linklets are compiled in JIT mode and a called procedure is to be
compiled on demand, consult a cache of compiled fragments (by default,
"jit.sqlite" in the addons directory) and either use an existing
compiled fragment or add to the cache after compiling.
Results for this initial implementation suggests that the idea is
workable. With the cache, starting a JIT-mode program a second time is
almost as fast as non-JIT mode (i.e., directly loading machine code).
Some refinements are needed: limiting the size of the JIT-fragment
cache, better contention handling, and better inlining of structure
operations in JIT mode (which may be useful to cross-linklet
optimization in non-JIT mode, too).
To avoid recording absolute paths from a build environment in bytecode
files, the bytecode writer converts paths to relative form based on
`current-write-relative-directory`. For paths that cannot be made
relative in that way and that are in source locations in syntax
objects, the printer in v6.x converted those paths to strings that
drop most of the path.
The v7 expander serializes syntax objects as part of `compile` instead
of `write`, so it can't truncate paths in the traditional way. To help
out the expander, the core `write` function for compiled code now
allows `srcloc` values --- as long as the source field is a path,
string, byte string, symbol, or #f. (Constraining the source field
avoids various problems, including problems that could be created by
cyclic values.) As the core `write` for compiled code prints a path,
it truncates a source path in the traditional way.
The expander doesn't constrain source locations in syntax objects to
have path, string, etc., source values. It can serialize syntax
objects with non-path source values at `compile` time, so there's no
loss of functionality.
The end result is to fix abolute paths that were getting stored in the
bytecode for compiled packages, since that's no good for installing
packages in built form (which happens, for example, during a
distribution build).
Although SHA-1 hashing functions are available from `openssl`
libraries, a fast crytopgraphic hash is useful for many purposes below
the layer where the OpenSSL library has been opened. And SHA-1 is
reasonably easy to add to rktio.
Meanwhile, provide an equally convenient SHA-2 function to discourage
bad security practices (i.e., using SHA-1 where SHA-2 should be
preferred).
Applying jitify to a linklet now generates fragments of code that
are not nested. The drawback of this approach is that calling
a nested function needs an extra indirection, and the closure
has an extra slot. The advantage is that the fragments can be
separately compiled and fasled, which could enable a cache of
compiled fragments.
Use a `(let ([<name> ....]) <name>)` wrapper to communicate
an 'inferred-name property from correlated objects to
Chez Scheme. This stategy relies on a Chez Scheme patch to
make the wrapper work consistently.
The improvements reported in 74012f8c57 were actually due to a broken
experiment that dropped source locations on application forms, instead
of preserving them in marshaled code. Adjust the expansion pipeline
to do that earlier and intentionally.
The xify pas doesn't help all that much after all, but it's still more
comfortable to be independent of local-variable names.
The xify pass replaces local variable names with `x0`, `x1`, etc.
Using a minimal set of symbols makes the fasled form smaller
and typically take only 60-70% as long to read.
The places test suite included some tests that create lots of places
and don't wait for them, which can lead to an overload of places that
exhausts resources such as file descriptors. Improve the tests, and
also improve a failure behavior from a crash to an error message.
The `place-kill` function sends a message to another place to
terminate, but it didn't wait for that message to take
effect before returning. Worse, it put the place object in a
state that claimed that the place had terminated.
When the first subexpression is complex and the second
is a literal character, the generated JIT code swaps the
argument order, but compilation didn't swap the test for
whether one or the other is a literal character to skip
a run-time test.
When an embedding application calls `scheme_basic_env` a
second time, it's supposed to reset the main namespace, but
the new expander wasn't reset correctly.
* Fix handling of single-percision infinities and nan
* Document that non-`hash-eq?` hash tables are accepted by `jsexpr?`.
* Document that the value of `json-null` is recognized using `eq?`
* Use `case` instead of `assoc`.
* Use contracts
Delay reporting of potential problems until an actual problem
is detected. Correct a mismatch between original and renamed
symbols to restore detection of problems.
* When you delete a file in Windows, then the name doesn't go away
until the file is closed in all processes (and background tasks like
search indexing may open files behind your back). Worse, attempting
to create a new file with the same name reports a permission error,
not a file-exists error; there's seems to be no way to tell whether
a permission error was really a file-exists error.
This creates trouble for `make-temporary-file` when files are
created, deleted, and created again quickly enough and when
something like a search indexer runs in the background (which is the
usual Windows configuration). In practice, that kind of collision
happens often for `raco setup` on my machine.
To compensate, make `make-temporary-file` try up to 32 times on a
permission error. A collision that many times seems extremely
unlikely, and it seems ok to delay an actual permission error.
Windows provides a GetTempFileName function from "kernel.dll" that
must be able to deal with this somehow --- perhaps because it's in
the kernel --- but it doesn't solve the problem for making temporary
directories, hence the 32-tries approach for now.
* When a deleted file's name persists because the file is open in some
process, then a directory containing the file cannot be deleted.
This creates trouble for `delete-directory/files`, since
`delete-file` on a directory's content doesn't necessarily make the
directory empty. In practice, this happens often for package
upgrades on my machine, where the package system wants to delete a
short-lived working space that the indexer is trying to scan.
To compenstate, change `delete-directory/files` to delete a file by
first moving it to the temporary directory with a fresh name, and
then delete the file there. It may take a while for a file to
disappear from the temporary directory, but meanwhile it's not in
the way of the original enclosing directory.
* When a file is open by any process, it prevents renaming any
ancestor directory of the file.
This creates trouble for the package system, which installs a
package by unpacking it in a temporary place and then moving it by
renaming. The package system also removes a package by renaming it
to a subdirectory of a ".trash" directory. If a background indexer
has a package file open, the move fails. In practice, a move fails
often on my machine when I'm attempting to upgrade many packages.
To compensate, make the package system fall back to copy + delete
if moving fails with a permission error.
Bind variables in a way that allows `local-expand` (with an empty stop
list) to replace a reference to the binding with one that has the same
scopes as the binding.
This repair was motivated by tests in the "rex" package. The
new test added here failed before by finding 'new both times,
but in the "rex" case, the mixup led to the same variable
being imported and exported at the linklet level.
struct-out was putting the super-struct's accessors into two parts of a
struct-info: the accessor list and the mutator list
this commit puts the accessors only in the accessor list and the
mutators in the mutator list
Using a frame pointer for the ABI of internal helper functions
should make the stack friendlier to tools like `perf`. There
may be a small performance cost, though.
Alexis's repair, and as she notes, forcing a `post-expansion` context
value in the core `#%module-begin` expander may allow a simplification
in "definition-context.rkt". But it's not immediately obvious, so save
that potential improvement for later.
Relevant to #2118
When `local-expand` receives one or more internal definition contexts,
it would forget about any current post-expansion scopes. That's
particularly a problem in a 'module-begin expansion context, where the
post-expansion scope ensures that any bindings are suitably
phase-specific.
Closes#2115
This is similar to the recent change of functions with optional
values. Using unsafe-undefined instead of a gensym makes it easier
to avoid the check of the missing argument.
When the separator is a string, these function construct a regexp
that is cached to make repeated calls faster. But when the string
is mutated it is necessary to recalculate the regexp.
Commit 32b256886e adds shifts in one place where it shoouldn't;
the "determinsitic-zo" test exposed the problem.
Also, avoid adding shifts that will have no effect, which avoids
accumulating useless shifts in some top-level contexts.
