#lang scribble/doc @(require scribble/manual scribble/bnf "common.ss" scribble/eval (for-label racket/base racket/include racket/contract racket/future compiler/cm compiler/cm-accomplice setup/parallel-build)) @(define cm-eval (make-base-eval)) @(interaction-eval #:eval cm-eval (require compiler/cm)) @title[#:tag "make"]{@exec{raco make}: Compiling Source to Bytecode} The @exec{raco make} command accept filenames for Racket modules to be compiled to bytecode format. Modules are re-compiled only if the source Racket file is newer than the bytecode file and has a different SHA-1 hash, or if any imported module is recompiled or has a different SHA-1 hash for its compiled form plus dependencies. @; ---------------------------------------------------------------------- @section{Bytecode Files} A file @filepath{@nonterm{name}.@nonterm{ext}} is compiled to bytecode that is saved as @filepath{compiled/@nonterm{name}_@nonterm{ext}.zo} relative to the file. As a result, the bytecode file is normally used automatically when @filepath{@nonterm{name}.@nonterm{ext}} is required as a module, since the underlying @racket[load/use-compiled] operation detects such a bytecode file. For example, in a directory that contains the following files: @itemize[ @item{@filepath{a.rkt}: @racketmod[ racket (require "b.rkt" "c.rkt") (+ b c) ]} @item{@filepath{b.rkt}: @racketmod[ racket (provide b) (define b 1) ]} @item{@filepath{c.rkt}: @racketmod[ racket (provide c) (define c 1) ]}] then @commandline{raco make a.rkt} triggers the creation of @filepath{compiled/a_rkt.zo}, @filepath{compiled/b_rkt.zo}, and @filepath{compiled/c_rkt.zo}. A subsequent @commandline{racket a.rkt} loads bytecode from the generated @filepath{.zo} files, paying attention to the @filepath{.rkt} sources only to confirm that each @filepath{.zo} file has a later timestamp. In contrast, @commandline{racket b.rkt c.rkt} would create only @filepath{compiled/b_rkt.zo} and @filepath{compiled/c_rkt.zo}, since neither @filepath{b.rkt} nor @filepath{c.rkt} imports @filepath{a.rkt}. @; ---------------------------------------------------------------------- @section{Dependency Files} In addition to a bytecode file, @exec{raco make} creates a file @filepath{compiled/@nonterm{name}_@nonterm{ext}.dep} that records dependencies of the compiled module on other module files and the source file's SHA-1 hash. Using this dependency information, a re-compilation request via @exec{raco make} can consult both the source file's timestamp/hash and the timestamps/hashes for the bytecode of imported modules. Furthermore, imported modules are themselves compiled as necessary, including updating the bytecode and dependency files for the imported modules, transitively. Continuing the @exec{raco make a.rkt} example from the previous section, the @exec{raco make} command creates @filepath{compiled/a_rkt.dep}, @filepath{compiled/b_rkt.dep}, and @filepath{compiled/c_rkt.dep} at the same time as the @filepath{.zo} files. The @filepath{compiled/a_rkt.dep} file records the dependency of @filepath{a.rkt} on @filepath{b.rkt}, @filepath{c.rkt} and the @racketmodname[racket] library. If the @filepath{b.rkt} file is modified (so that its timestamp and SHA-1 hash changes), then running @commandline{raco make a.rkt} again rebuilds @filepath{compiled/a_rkt.zo} and @filepath{compiled/b_rkt.zo}. For module files that are within library collections, @exec{raco setup} uses the same @filepath{.zo} and @filepath{.dep} conventions and files as @exec{raco make}, so the two tools can be used together. @; ---------------------------------------------------------------------- @section{API for Making Bytecode} @defmodule[compiler/cm]{The @racketmodname[compiler/cm] module implements the compilation and dependency management used by @exec{raco make} and @exec{raco setup}.} @defproc[(make-compilation-manager-load/use-compiled-handler [delete-zos-when-rkt-file-does-not-exist? any/c #f]) (path? (or/c symbol? false/c) . -> . any)]{ Returns a procedure suitable as a value for the @racket[current-load/use-compiled] parameter. The returned procedure passes it arguments on to the @racket[current-load/use-compiled] procedure that is installed when @racket[make-compilation-manager-load/use-compiled-handler] is called, but first it automatically compiles a source file to a @filepath{.zo} file if @itemize[ @item{the file is expected to contain a module (i.e., the second argument to the handler is a symbol);} @item{the value of each of @racket[(current-eval)], @racket[(current-load)], and @racket[(namespace-module-registry (current-namespace))] is the same as when @racket[make-compilation-manager-load/use-compiled-handler] was called;} @item{the value of @racket[use-compiled-file-paths] contains the first path that was present when @racket[make-compilation-manager-load/use-compiled-handler] was called;} @item{the value of @racket[current-load/use-compiled] is the result of this procedure; and} @item{one of the following holds: @itemize[ @item{the source file is newer than the @filepath{.zo} file in the first sub-directory listed in @racket[use-compiled-file-paths] (at the time that @racket[make-compilation-manager-load/use-compiled-handler] was called), and either no @filepath{.dep} file exists or it records a source-file SHA-1 hash that differs from the current version and source-file SHA-1 hash;} @item{no @filepath{.dep} file exists next to the @filepath{.zo} file;} @item{the version recorded in the @filepath{.dep} file does not match the result of @racket[(version)];} @item{one of the files listed in the @filepath{.dep} file has a @filepath{.zo} timestamp newer than the target @filepath{.zo}, and the combined hashes of the dependencies recorded in the @filepath{.dep} file does not match the combined hash recorded in the @filepath{.dep} file.} ]} ] If SHA-1 hashes override a timestamp-based decision to recompile the file, then the target @filepath{.zo} file's timestamp is updated to the current time. After the handler procedure compiles a @filepath{.zo} file, it creates a corresponding @filepath{.dep} file that lists the current version and the identification of every file that is directly @racket[require]d by the module in the compiled file. Additional dependencies can be installed during compilation via @racketmodname[compiler/cm-accomplice]. The @filepath{.dep} file also records the SHA-1 hash of the module's source, and it records a combined SHA-1 hash of all of the dependencies that includes their recursive dependencies. The handler caches timestamps when it checks @filepath{.dep} files, and the cache is maintained across calls to the same handler. The cache is not consulted to compare the immediate source file to its @filepath{.zo} file, which means that the caching behavior is consistent with the caching of the default module name resolver (see @racket[current-module-name-resolver]). If @racket[use-compiled-file-paths] contains an empty list when @racket[make-compilation-manager-load/use-compiled-handler] is called, then @racket[exn:fail:contract] exception is raised. If the @racket[delete-zos-when-rkt-file-does-not-exist?] argument is a true value, then the returned handler will delete @filepath{.zo} files when there is no corresponding original source file. @emph{Do not} install the result of @racket[make-compilation-manager-load/use-compiled-handler] when the current namespace contains already-loaded versions of modules that may need to be recompiled---unless the already-loaded modules are never referenced by not-yet-loaded modules. References to already-loaded modules may produce compiled files with inconsistent timestamps and/or @filepath{.dep} files with incorrect information.} @defproc[(managed-compile-zo [file path-string?] [read-src-syntax (any/c input-port? . -> . syntax?) read-syntax]) void?]{ Compiles the given module source file to a @filepath{.zo}, installing a compilation-manager handler while the file is compiled (so that required modules are also compiled), and creating a @filepath{.dep} file to record the timestamps of immediate files used to compile the source (i.e., files @racket[require]d in the source). If @racket[file] is compiled from source, then @racket[read-src-syntax] is used in the same way as @racket[read-syntax] to read the source module. The normal @racket[read-syntax] is used for any required files, however.} @defboolparam[trust-existing-zos trust?]{ A parameter that is intended for use by @exec{setup-plt} when installing with pre-built @filepath{.zo} files. It causes a compilation-manager @racket[load/use-compiled] handler to ``touch'' out-of-date @filepath{.zo} files instead of re-compiling from source.} @defproc[(make-caching-managed-compile-zo [read-src-syntax (any/c input-port? . -> . syntax?)]) (path-string? . -> . void?)]{ Returns a procedure that behaves like @racket[managed-compile-zo] (providing the same @racket[read-src-syntax] each time), but a cache of timestamp information is preserved across calls to the procedure.} @defparam[manager-compile-notify-handler notify (path? . -> . any)]{ A parameter for a procedure of one argument that is called whenever a compilation starts. The argument to the procedure is the file's path.} @defparam[manager-trace-handler notify (string? . -> . any)]{ A parameter for a procedure of one argument that is called to report compilation-manager actions, such as checking a file. The argument to the procedure is a string.} @defparam[manager-skip-file-handler proc (-> path? (or/c (cons/c number? promise?) #f))]{ A parameter whose value is called for each file that is loaded and needs recompilation. If the procedure returns a pair, then the file is skipped (i.e., not compiled); the number in the pair is used as the timestamp for the file's bytecode, and the promise may be @racket[force]d to obtain a string that is used as hash of the compiled file plus its dependencies. If the procedure returns @racket[#f], then the file is compiled as usual. The default is @racket[(lambda (x) #f)].} @defproc[(file-stamp-in-collection [p path?]) (or/c (cons/c number? promise?) #f)]{ Calls @racket[file-stamp-in-paths] with @racket[p] and @racket[(current-library-collection-paths)].} @defproc[(file-stamp-in-paths [p path?] [paths (listof path?)]) (or/c (cons/c number? promise?) #f)]{ Returns the file-modification date and @racket[delay]ed hash of @racket[p]or its bytecode form (i.e., @filepath{.zo} file), whichever exists and is newer, if @racket[p] is an extension of any path in @racket[paths] (i.e., exists in the directory, a subdirectory, etc.). Otherwise, the result is @racket[#f]. This function is intended for use with @racket[manager-skip-file-handler].} @defproc[(get-file-sha1 [p path?]) (or/c string? #f)]{ Computes a SHA-1 hash for the file @racket[p]; the result is @racket[#f] if @racket[p] cannot be opened.} @defproc[(get-compiled-file-sha1 [p path?]) (or/c string? #f)]{ Computes a SHA-1 hash for the bytecode file @racket[p], appending any dependency-describing hash available from a @filepath{.dep} file when available (i.e., the suffix on @racket[p] is replaced by @filepath{.dep} to locate dependency information). The result is @racket[#f] if @racket[p] cannot be opened.} @defproc[(with-compile-output [p path-string?] [proc ([port input-port?] [tmp-path path?] . -> . any)]) any]{ Opens a temporary path for writing and calls @racket[proc] passing the resulting @racket[port] and @racket[tmp-path]. Once @racket[proc] returns, @racket[with-compile-output] renames @racket[tmp-path] to @racket[p] and arranges to delete @racket[temp-path] if there's an exception. Breaks are managed so that the @racket[port] is reliably closed and the @racket[tmp-path] file is reliably deleted if there's a break. The result of @racket[proc] is the result of the @racket[with-compile-output] call. Windows prevents programs from overwriting files that are open. As a result, @racket[with-compile-output] calls to @racket[rename-file-or-directory] will fail if the destination file argument is an open file. Windows, however, does allow you to rename an open file. To avoid overwriting open files windows, @racket[with-compile-output] creates a second temporary file @racket[tmp-path2], renames @racket[p] to @racket[tmp-path2], renames @racket[tmp-path] to @racket[p], and finally deletes @racket[tmp-path2].} @defparam[parallel-lock-client proc ([command (or/c 'lock 'unlock)] [zo-path bytes?] . -> . boolean?)]{ Holds the parallel compilation lock client, which prevents compilation races between parallel builders. The @racket[proc] function takes a command argument of either @racket['lock] or @racket['unlock]. The @racket[zo-path] argument specifies the path of the zo for which compilation should be locked. When the @racket[proc] @racket['lock] command returns @racket[#t], the current builder has obtained the lock for @racket[zo-path]. Once compilation of @racket[zo-path] is complete, the builder process must release the lock by calling @racket[proc] @racket['unlock] with the exact same @racket[zo-path]. When the @racket[proc] @racket['lock] command returns @racket[#f], another parallel builder obtained the lock first and has already compiled the zo. The parallel builder should continue without compiling @racket[zo-path]. @examples[ #:eval cm-eval (let* ([lc (parallel-lock-client)] [zo-name #"collects/racket/compiled/draw_rkt.zo"] [locked? (and lc (lc 'lock zo-name))] [ok-to-compile? (or (not lc) locked?)]) (dynamic-wind (lambda () (void)) (lambda () (when ok-to-compile? (printf "Do compile here ...\n"))) (lambda () (when locked? (lc 'unlock zo-name))))) ] } @; ---------------------------------------------------------------------- @section[#:tag "api:parallel-build"]{API for Parallel-Build} @defmodule[setup/parallel-build]{ The @schememodname[setup/parallel-build] library provides the parallel compilation to bytecode functionality of @exec{rack setup} and @exec{raco make}.} @; ---------------------------------------------------------------------- @defproc[(parallel-compile-files [list-of-files (listof path?)] [#:worker-count worker-count non-negative-integer? (processor-count)] [#:handler handler (->i ([handler-type symbol?] [path path-string?] [msg string?] [out string?] [err string?]) void?) void]) void?]{ The @racket[parallel-compile] utility function is used by @exec{raco make} to compile a list of paths in parallel. The optional keyword argument @racket[#:worker-count] specifies the number of compile workers to spawn during parallel compilation. The callback, @racket[handler], is called with the symbol @racket['done] as the @racket[_handler-type] argument for each successfully compiled file, @racket['output] when a successful compilation produces stdout/stderr output, @racket['error] when a compilation error has occured, or @racket['fatal-error] when a unrecoverable error occurs. The other arguments give more information for each status update. @racketblock[ (parallel-compile-files source-files #:worker-count 4 #:handler (lambda (type work msg out err) (match type ['done (when (verbose) (printf " Made ~a\n" work))] ['output (printf " Output from: ~a\n~a~a" work out err)] [else (printf " Error compiling ~a\n~a\n~a~a" work msg out err)])))] } @defproc[(parallel-compile [worker-count non-negative-integer?] [setup-fprintf (->* ([stage string?] [format string?]) () #:rest (listof any/c) void)] [append-error (cc? [prefix string?] [exn (or/c exn? null?)] [out string?] [err srtring?] [message string?] . -> . void?)] [collects-tree (listof any/c)]) (void)]{ The @racket[parallel-compile] internal utility function is used by @exec{rack setup} to compile collects in parallel. The @racket[worker-count] argument specifies the number of compile workers to spawn during parallel compilation. The @racket[setup-fprintf] and @racket[append-error] functions are internal callback mechanisms that @exec{rack setup} uses to communicate intermediate compilation results. The @racket[collects-tree] argument is a compound datastructure containing an in-memory tree representation of the collects directory. } @section{Compilation Manager Hook for Syntax Transformers} @defmodule[compiler/cm-accomplice] @defproc[(register-external-file [file (and path? complete-path?)]) void?]{ Logs a message (see @racket[log-message]) at level @racket['info]. The message data is a @racketidfont{file-dependency} prefab structure type with one field whose value is @racket[file]. A compilation manager implemented by @racketmodname[compiler/cm] looks for such messages to register an external dependency. The compilation manager records (in a @filepath{.dep} file) the path as contributing to the implementation of the module currently being compiled. Afterward, if the registered file is modified, the compilation manager will know to recompile the module. The @racket[include] macro, for example, calls this procedure with the path of an included file as it expands an @racket[include] form.} @; ---------------------------------------------------------------------- @section[#:tag "zo"]{Compiling to Raw Bytecode} The @DFlag{no-deps} mode for @exec{raco make} is an improverished form of the compilation, because it does not track import dependencies. It does, however, support compilation of non-module source in an namespace that initially imports @racketmodname[scheme]. Outside of a module, top-level @racket[define-syntaxes], @racket[module], @racket[#%require], @racket[define-values-for-syntax], and @racket[begin] expressions are handled specially by @exec{raco make --no-deps}: the compile-time portion of the expression is evaluated, because it might affect later expressions. For example, when compiling the file containing @racketblock[ (require racket/class) (define f (class object% (super-new))) ] the @racket[class] form from the @racketmodname[racket/class] library must be bound in the compilation namespace at compile time. Thus, the @racket[require] expression is both compiled (to appear in the output code) and evaluated (for further computation). Many definition forms expand to @racket[define-syntaxes]. For example, @racket[define-signature] expands to @racket[define-syntaxes]. In @DFlag{no-deps} mode, @exec{raco make --no-deps} detects @racket[define-syntaxes] and other expressions after expansion, so top-level @racket[define-signature] expressions affect the compilation of later expressions, as a programmer would expect. In contrast, a @racket[load] or @racket[eval] expression in a source file is compiled---but @emph{not evaluated!}---as the source file is compiled. Even if the @racket[load] expression loads syntax or signature definitions, these will not be loaded as the file is compiled. The same is true of application expressions that affect the reader, such as @racket[(read-case-sensitive #t)]. The @Flag{p} or @DFlag{prefix} flag for @exec{raco make} takes a file and loads it before compiling the source files specified on the command line. In general, a better solution is to put all code to compile into a module and use @exec{raco make} in its default mode. @(close-eval cm-eval)