raco setup: --tidy mode (as needed by `raco pkg remove') and more

When a collection disappears, then documentation and metadata
cross-references need fixing up based on everything that remains
available, even though other setup actions are confined to
collections that are specifically relevant to the packages. The
new `--tidy' mode takes care of that.

Package installation now also provides `--avoid-main' when working
with packages in a scope other than installation scope, and `raco
setup' now better respects `--avoid-main' to avoid creating
executables or re-running documentation.

Also, revise the `raco' documentation to better orient it toward the
package manager.

collects/compiler/compiler-unit.rkt

@@ -130,7 +130,7 @@
                                                [len (bytes-length skip-path)])
                                            (and ((bytes-length b) . > . len)
                                                 (bytes=? (subbytes b 0 len) skip-path)))
-                                         (list -inf.0 "")))])
+                                         (cons -inf.0 "")))])
                     (let* ([sses (append
                                   ;; Find all .rkt/.ss/.scm files:
                                   (filter extract-base-filename/ss (directory-list))

collects/pkg/main.rkt

@@ -19,7 +19,9 @@
                                (if (list? s) s (list s)))
                              (append setup-collects
                                      (if installation? '("scribblings/main") null)
-                                     '("scribblings/main/user")))))))
+                                     '("scribblings/main/user"))))
+     #:tidy? #t
+     #:avoid-main? (not installation?))))
 
 (define ((pkg-error cmd) . args)
   (apply raise-user-error

collects/scribblings/raco/command.scrbl

@@ -5,9 +5,11 @@
 @title[#:tag "command"]{Adding a @exec{raco} Command}
 
 The set of commands supported by @exec{raco} can be extended by
-installed collections and @|PLaneT| packages. A command is added by
+installed packages, @|PLaneT| packages, and other collections. A
-defining @indexed-racket[raco-commands] in the @filepath{info.rkt}
+command is added by defining @indexed-racket[raco-commands] in the
-library of a collection or package (see @secref["info.rkt"]).
+@filepath{info.rkt} library of a collection (see @secref["info.rkt"]),
+and then @exec{raco setup} (as called directly or as part of a package
+or @|PLaneT| installation) must index the @filepath{info.rkt} file.
 
 The value bound to @racket[raco-commands] must be a list of command
 specifications, where each specification is a list of four values:
@@ -34,7 +36,7 @@ the command name used to load the command. When @exec{raco help} is
 used on a command, the command is launched with an initial
 @DFlag{help} argument in @racket[current-command-line-arguments].
 
-The @racket[description-string] is a short string used to describe the
+The @racket[_description-string] is a short string used to describe the
 command in response to @exec{raco help}. The description should not be
 capitalized or end with a period.
 
@@ -47,7 +49,7 @@ more prominent. The @exec{pack} tool, which is currently ranked as the
 least-prominent of the frequently used commands, has a value of
 @racket[10].
 
-As an example, the @filepath{info.rkt} of the "compiler" collection
+As an example, the @filepath{info.rkt} of the @filepath{compiler} collection
 might contain the
 
 @racketblock[

collects/scribblings/raco/link.scrbl

@@ -12,6 +12,13 @@ The @exec{raco link} command inspects and modifies a @tech[#:doc
 reference-doc]{collection links file} to display, add, or remove
 mappings from collection names to filesystem directories.
 
+Managing links directly is somewhat discouraged. Instead, use the
+package manager (see @other-manual['(lib
+"pkg/scribblings/pkg.scrbl")]), which installs and manages links
+(i.e., it builds on @exec{raco link}) in a way that more gracefully
+leads to sharing collections with others. Nevertheless, @exec{raco
+link} is available for direct use.
+
 For example, the command
 
 @commandline{raco link maze}

collects/scribblings/raco/make.scrbl

@@ -4,6 +4,7 @@
                      racket/include
                      racket/contract
                      racket/future
+                     racket/promise
                      compiler/cm
                      compiler/cm-accomplice
                      setup/parallel-build))

collects/scribblings/raco/plt.scrbl

@@ -6,18 +6,22 @@
 
 @title[#:tag "plt"]{@exec{raco pack}: Packing Library Collections}
 
-@margin-note{Before creating a @filepath{.plt} archive to distribute,
+The @exec{raco pack} command creates an archive of files and
-consider instead posting your package on
+directories. Formerly, such archives were used directly to distribute
-@link["http://planet.racket-lang.org/"]{@|PLaneT|}.}
+library files to Racket users, but the package manager (see
+@other-manual['(lib "pkg/scribblings/pkg.scrbl")]) is now the
+preferred mechanism for distribution.
 
-The @exec{raco pack} command creates an archive for distributing
+A packed archive usually has the suffix @as-index{@filepath{.plt}}.
-library files to Racket users. A distribution archive usually has the
+The @exec{raco pkg} command recognizes a @filepath{.plt} archive for
-suffix @as-index{@filepath{.plt}}, which DrRacket recognizes as an
+installation as a package. The @exec{raco setup} command (see
-archive to provide automatic unpacking facilities. The @exec{raco
+@secref["setup"]) also supports @filepath{.plt} unpacking and
-setup} command (see @secref["setup"]) also supports @filepath{.plt}
+installation when using the @Flag{A} flag, but such installations do
-unpacking with installation, while the @exec{raco unpack} command (see
+not benefit from the more general management facilities of @exec{raco
-@secref["unpack"]) unpacks an archive locally without attempting to
+pkg}, while the @exec{raco unpack} command (see @secref["unpack"])
-install it.
+unpacks an archive locally without attempting to install it. DrRacket
+recognizes the @filepath{.plt} and currently treats such an archive in
+the same way as @exec{raco setup -A}.
 
 An archive contains the following elements:
 

collects/scribblings/raco/setup.scrbl

@@ -46,85 +46,191 @@
 
 @title[#:tag "setup" #:style 'toc]{@exec{raco setup}: Installation Management}
 
-The @exec{raco setup} command finds, compiles, configures,
+The @exec{raco setup} command builds bytecode, documentation,
-and installs documentation for all collections in a Racket
+executables, and metadata indexes for all installed collections.
-installation. It can also install single @filepath{.plt} files.
+
+The collections that are built by @exec{raco setup} can be part of the
+original Racket distribution, installed via the package manager (see
+@other-manual['(lib "pkg/scribblings/pkg.scrbl")]), installed via
+@|PLaneT| (see @other-manual['(lib "planet/planet.scrbl")]), linked
+via @exec{raco link}, in a directory that is listed in the
+@envvar{PLTCOLLECTS} environment variable, or placed into one of the
+default collection directories.
+
+The @exec{raco setup} tool itself does not directly support the
+installation of collections, except through the now-discouraged
+@Flag{A} flag (see @secref["raco-setup-A"]). The @exec{raco setup} command is
+used by installation tools such as the package manager or @|PLaneT|.
+Programmers who modify installed collections may find it useful to run
+@exec{raco setup} as an alternative to un-installing and re-installing
+a set of collections.
 
 @local-table-of-contents[]
 
-@; ------------------------------------------------------------------------
 @; ------------------------------------------------------------------------
 
 @section[#:tag "running"]{Running @exec{raco setup}}
 
-The @exec{raco setup} command performs two main services:
+With no command-line arguments, @exec{raco setup} finds all of the
+current collections---see @secref[#:doc ref-src]{collects}---and
+compiles libraries in each collection.  (Directories that are named
+@filepath{.git} or @filepath{.svn} are not treated as collections.)
+
+To restrict @exec{raco setup} to a set of collections, provide the
+collection names as arguments. For example, @exec{raco setup
+scribblings/raco} would only compile and render the documentation for
+@exec{raco}, which is implemented in a @filepath{scribblings/raco}
+collection.
+
+An optional @filepath{info.rkt} within the collection can indicate
+specifically how the collection's files are to be compiled and other
+actions to take in setting up a collection, such as creating
+executables or building documentation. See @secref["setup-info"] for
+more information.
+
+The @exec{raco setup} command accepts the following command-line
+flags:
 
 @itemize[
 
- @item{@bold{Compiling and setting up all or some collections:} 
+ @item{@DFlag{workers} @nonterm{n} or @Flag{j} @nonterm{n} --- use up
-   When @exec{raco setup} is run without any arguments, it
+   to @nonterm{n} parallel processes.  By default, @exec{raco setup}
-   finds all of the current collections---see @secref[#:doc
+   uses @racket[(processor-count)] processors, which typically uses
-   ref-src]{collects}---and compiles libraries in each collection.
+   all of the machine's processing cores.}
-   (Directories that are named @filepath{.git} or @filepath{.svn} are 
-   not treated as collections.)
 
-   To restrict @exec{raco setup} to a set of collections, provide the
+ @item{@DFlag{only} --- restrict setup to specified collections and
-   collection names as arguments. For example, @exec{raco setup
+   @|PLaneT| packages, even if none are specified. This mode is the
-   scribblings/raco} would only compile and render the documentation
+   default if any collection is specified as a command-line argument,
-   for @exec{raco}, which is implemented in a
+   through the @Flag{l} flag, or through the @Flag{P} flag.}
-   @filepath{scribblings/raco} collection.
 
-   An optional @filepath{info.rkt} within the collection can indicate
+ @item{@Flag{P} @nonterm{owner} @nonterm{package-name} @nonterm{maj}
-   specifically how the collection's files are to be compiled and
+  @nonterm{min} --- constrain setup actions to the specified @|PLaneT|
-   other actions to take in setting up a collection, such as creating
+  package, in addition to any other specified @|PLaneT| packages or
-   executables or building documentation. See @secref["setup-info"]
+  @nonterm{collections}.}
-   for more information.
 
-   The @DFlag{clean} (or @Flag{c}) flag to @exec{raco setup} causes it to
+ @item{@DFlag{tidy} --- remove metadata cache information and
-   delete existing @filepath{.zo} files, thus ensuring a clean build
+  documentation for non-existent collections (to clean up after removal)
-   from the source files. The exact set of deleted files can be
+  even when setup actions are otherwise confined to specified collections.}
-   controlled by @filepath{info.rkt}; see
-   @elemref["clean"]{@racket[clean]} for more information.
 
-   The @DFlag{workers} (or @Flag{j}) flag to @exec{raco setup} takes
+ @item{@DFlag{clean} or @Flag{c} --- delete existing @filepath{.zo}
-   an argument @racket[_n] to make compilation use up to @racket[_n]
+   files, thus ensuring a clean build from the source files. The exact
-   parallel processes.  The default value of @racket[_n] is
+   set of deleted files can be controlled by @filepath{info.rkt}; see
-   @racket[(processor-count)], which typically uses all the machine's
+   @elemref["clean"]{@racket[clean]} for more information.}
-   processing cores.
 
-   The @DFlag{mode} @nonterm{mode} flag causes @exec{raco setup} to use a
+ @item{@DFlag{no-zo} or @Flag{n} --- refrain from compiling source
-   @filepath{.zo} compiler other than the default compiler, and to put
+   files to @filepath{.zo} files.}
-   the resulting @filepath{.zo} files in a subdirectory (of the usual
-   place) named by @nonterm{mode}. The compiler is obtained by using
-   @nonterm{mode} as a collection name, finding a
-   @filepath{zo-compile.rkt} module in that collection, and extracting
-   its @racket[zo-compile] export. The @racket[zo-compile] export
-   should be a function like @racket[compile]; see the
-   @filepath{errortrace} collection for an example.
 
-   When building @exec{racket}, flags can be provided to @exec{raco
+ @item{@DFlag{trust-zos} --- fix timestamps on @filepath{.zo} files on
-   setup} as run by @exec{make install} by setting the
+   the assumption that they are already up-to-date.}
-   @as-index{@envvar{PLT_SETUP_OPTIONS}} environment variable. For
-   example, the following command line uses a single process to build
-   collections during an install:
 
-   @commandline{env PLT_SETUP_OPTIONS="-j 1" make install}}
+ @item{@DFlag{no-launcher} or @Flag{x} --- refrain from creating
+   executables (as specified in @filepath{info.rkt} files; see
+   @secref["setup-info"]).}
 
- @item{@bold{Unpacking @filepath{.plt} files:} A
+ @item{@DFlag{no-install} or @Flag{i} --- refrain from running
-   @filepath{.plt} file is a platform-independent distribution archive
+   pre-install actions (as specified in @filepath{info.rkt} files; see
-   for software based on Racket. When one or more file names are
+   @secref["setup-info"]).}
-   provided as the command line arguments to @exec{raco setup} with the @Flag{A} flag, the files
-   contained in the @filepath{.plt} archive are unpacked (according to
-   specifications embedded in the @filepath{.plt} file) and only
-   collections specified by the @filepath{.plt} file are compiled and
-   setup.}]
 
-Run @exec{raco help setup} to see a list of all options accepted by
+ @item{@DFlag{no-post-install} or @Flag{I} --- refrain from running
-the @exec{raco setup} command.
+   post-install actions (as specified in @filepath{info.rkt} files; see
+   @secref["setup-info"]).}
+
+ @item{@DFlag{no-info-domain} or @Flag{d} --- refrain from building
+   a cache of metadata information from @filepath{info.rkt}
+   files. This cache is needed by other tools. For example,
+   @exec{raco} itself uses the cache to locate plug-in tools.}
+
+ @item{@DFlag{no-docs} or @Flag{D} --- refrain from building
+   documentation.}
+
+ @item{@DFlag{doc-pdf} @nonterm{dir} --- in addition to building HTML
+   documentation, render documentation to PDF and place files in
+   @nonterm{dir}.}
+
+ @item{@DFlag{no-user} or @Flag{U} --- refrain from any user-specific
+  (as opposed to installation-specific) setup actions.}
+
+ @item{@DFlag{no-planet} --- refrain from any setup actions for
+  @|PLaneT| actions; this flags is implied by @DFlag{no-user}.}
+
+ @item{@DFlag{avoid-main} --- refrain from any setup actions that
+  affect the installation, as opposed to user-specific actions.}
+
+ @item{@DFlag{mode} @nonterm{mode} --- use a @filepath{.zo} compiler
+   other than the default compiler, and put the resulting
+   @filepath{.zo} files in a subdirectory (of the usual place) named
+   by @nonterm{mode}. The compiler is obtained by using @nonterm{mode}
+   as a collection name, finding a @filepath{zo-compile.rkt} module in
+   that collection, and extracting its @racket[zo-compile] export. The
+   @racket[zo-compile] export should be a function like
+   @racket[compile]; see the @filepath{errortrace} collection for an
+   example.}
+
+ @item{@DFlag{verbose} or @Flag{v} --- More verboase output about
+   @exec{raco setup} actions.}
+
+ @item{@DFlag{make-verbose} or @Flag{m} --- More verboase output about
+   dependency checks.}
+
+ @item{@DFlag{compiler-verbose} or @Flag{r} --- Even more verbose
+   output about dependency checks and compilation.}
+
+ @item{@DFlag{pause} or @Flag{p} --- Pause for user input if any
+  errors are reported (so that a user has time to inspect output that
+  might otherwise disappear when the @exec{raco setup} process ends).}
+
+ @item{@Flag{l} @nonterm{collection} @racket[...] --- constrain setup
+  actions to the specified @nonterm{collection}s (i.e., the same as
+  providing @nonterm{collections}s without a flag, but with no
+  possibility that a @nonterm{collection} is interpreted as a flag.}
+
+ @item{@Flag{A} @nonterm{archive} @racket[...] --- Install each
+  @nonterm{archive}; see @secref["raco-setup-A"].}
+
+ @item{@DFlag{force} --- for use with @Flag{A}, treat version
+  mismatches for archives as mere warnings.}
+
+ @item{@DFlag{all-users} or @Flag{a} --- for use with @Flag{A},
+  install archive into the installation instead of a user-specific
+  location.}
+
+
+]
+
+When building @exec{racket}, flags can be provided to @exec{raco
+setup} as run by @exec{make install} by setting the
+@as-index{@envvar{PLT_SETUP_OPTIONS}} environment variable. For
+example, the following command line uses a single process to build
+collections during an install:
+
+   @commandline{env PLT_SETUP_OPTIONS="-j 1" make install}
 
 @; ------------------------------------------------------------------------
 
-@subsection[#:tag "setup-info"]{Controlling @exec{raco setup} with @filepath{info.rkt} Files}
+@section[#:tag "raco-setup-A"]{Installing @filepath{.plt} Archives}
+
+A @filepath{.plt} file is a platform-independent distribution archive
+for software based on Racket. A typical @filepath{.plt} file can be
+installed as a package using @exec{raco pkg} (see @other-manual['(lib
+"pkg/scribblings/pkg.scrbl")]), in which case @exec{raco pkg} supplies
+facilities for uninstalling the package and managing dependencies.
+
+An older approach is to supply a @filepath{.plt} file to @exec{raco
+setup} with the @Flag{A} flag, the files contained in the
+@filepath{.plt} archive are unpacked (according to specifications
+embedded in the @filepath{.plt} file) and only collections specified
+by the @filepath{.plt} file are compiled and setup. Archives processed
+in this way can include arbitrary code that is executed at install
+time, in addition to any actions triggered by the normal
+collection-setup part of @exec{raco setup}.
+
+Finally, the @exec{raco unpack} (see @secref["unpack"]) command can
+list the content of a @filepath{.plt} archive or unpack the archive
+without installing it as a package or collection.
+
+@; ------------------------------------------------------------------------
+
+@section[#:tag "setup-info"]{Controlling @exec{raco setup} with @filepath{info.rkt} Files}
 
 To compile a collection's files to bytecode, @exec{raco setup} uses the
 @racket[compile-collection-zos] procedure. That procedure, in turn,
@@ -410,7 +516,7 @@ Optional @filepath{info.rkt} fields trigger additional actions by
 
 @; ------------------------------------------------------------------------
 
-@section[#:tag "setup-plt-plt"]{API for Installation}
+@section[#:tag "setup-plt-plt"]{API for Setup}
 
 @defmodule[setup/setup]
 
@@ -425,6 +531,8 @@ Optional @filepath{info.rkt} fields trigger additional actions by
                 [#:make-user? make-user? any/c #t]
                 [#:make-docs? make-docs? any/c #t]
                 [#:clean? clean? any/c #f]
+                [#:tidy? tidy? any/c #f]
+                [#:avoid-main? avoid-main? any/c #f]
                 [#:jobs jobs exact-nonnegative-integer? #f]
                 [#:get-target-dir get-target-dir (or/c #f (-> path-string?)) #f])
           void?]{
@@ -449,6 +557,13 @@ Runs @exec{raco setup} with various options:
 
  @item{@racket[clean?] --- if true, enables cleaning mode instead of setup mode}
 
+ @item{@racket[tidy?] --- if true, enables global tidying of
+       documentation and metadata indexes even when @racket[collections]
+       or @racket[planet-specs] is non-@racket[#f]}
+
+ @item{@racket[avoid-main?] --- if true, avoids setup actions that affect
+       the main installation, as opposed to user directories}
+
  @item{@racket[jobs] --- if not @racket[#f], determines the maximum number of parallel
        tasks used for setup}
 

collects/scribblings/raco/unpack.scrbl

@@ -9,7 +9,9 @@
 
 The @exec{raco unpack} command unpacks a @filepath{.plt} archive (see
 @secref["plt"]) to the current directory without attempting to install
-any collections. Use @exec{raco setup -A} (see @secref["setup"]) to
+any collections. Use @exec{raco pkg} (see @other-manual['(lib
+"pkg/scribblings/pkg.scrbl")]) to install a @filepath{.plt} archive as
+a package, or use @exec{raco setup -A} (see @secref["setup"]) to
 unpack and install collections from a @filepath{.plt} archive.
 
 Command-line flags:

collects/setup/collects.rkt

@@ -3,5 +3,5 @@
 (provide (struct-out cc))
 
 (define-struct cc
-  (collection path name info omit-root info-root info-path info-path-mode shadowing-policy)
+  (collection path name info omit-root info-root info-path info-path-mode shadowing-policy main?)
   #:inspector #f)

collects/setup/option-sig.rkt

@@ -11,6 +11,7 @@
      compiler-verbose
      clean
      compile-mode
+     make-only
      make-zo
      make-info-domain
      make-launchers
@@ -18,6 +19,7 @@
      make-user
      make-planet
      avoid-main-installation
+     make-tidy
      call-install
      call-post-install
      pause-on-errors

collects/setup/option-unit.rkt

@@ -36,6 +36,7 @@
   (define-flag-param compiler-verbose #f)
   (define-flag-param clean #f)
   (define-flag-param compile-mode #f)
+  (define-flag-param make-only #f)
   (define-flag-param make-zo #t)
   (define-flag-param make-launchers #t)
   (define-flag-param make-info-domain #t)
@@ -43,6 +44,7 @@
   (define-flag-param make-user #t)
   (define-flag-param make-planet #t)
   (define-flag-param avoid-main-installation #f)
+  (define-flag-param make-tidy #f)
   (define-flag-param call-install #t)
   (define-flag-param call-post-install #t)
   (define-flag-param pause-on-errors #f)

collects/setup/scribble.rkt

@@ -118,6 +118,8 @@
          auto-start-doc?    ; if #t, expands `only-dir' with [user-]start to
                                         ;  catch new docs
          make-user?         ; are we making user stuff?
+         tidy?              ; clean up, even beyond `only-dirs'
+         avoid-main?        ; avoid main collection, even for `tidy?'
          with-record-error  ; catch & record exceptions
          setup-printf)
   (unless (doc-db-available?)
@@ -191,7 +193,9 @@
       (filter-user-docs (append-map (get-docs main-dirs) infos recs) make-user?)))
   (define-values (main-docs user-docs) (partition doc-under-main? docs))
 
-  (unless only-dirs
+  (when (and (or (not only-dirs) tidy?)
+             (not avoid-main?)
+             (not latex-dest))
     ;; Check for extra document directories that we should remove
     ;; in the main installation:
     (log-setup-info "checking installation document directories")
@@ -300,28 +304,36 @@
                  (hash-set! out-path->info-cache path i)
                  i)))))
 
-  (define (make-loop first? iter)
+  (define (tidy-database)
-    (let ([infos (filter-not info-failed? infos)]
+    (when (and (or (not only-dirs) tidy?)
-          [src->info (make-hash)]
+               (not latex-dest))
-          [out-path->info-cache (make-hash)]
+      (log-setup-info "tidying database")
-          [main-db (find-doc-db-path latex-dest #f)]
-          [user-db (find-doc-db-path latex-dest #t)])
-      (unless only-dirs
-        (log-setup-info "cleaning database")
       (define files (make-hash))
+      (define tidy-docs (if tidy?
+                            docs
+                            (map info-doc infos)))
       (define (get-files! main?)
-          (for ([i (in-list infos)]
+        (for ([doc (in-list tidy-docs)]
-                #:when (eq? main? (main-doc? (info-doc i))))
+              #:when (eq? main? (main-doc? doc)))
-            (define doc (info-doc i))
           (hash-set! files (sxref-path latex-dest doc "in.sxref") #t)
           (for ([c (in-range (add1 (doc-out-count doc)))])
             (hash-set! files (sxref-path latex-dest doc (format "out~a.sxref" c)) #t))))
+      (unless avoid-main?
         (get-files! #t)
-        (doc-db-clean-files main-db files)
+        (doc-db-clean-files main-db files))
       (when (and (file-exists? user-db)
                  (not (equal? main-db user-db)))
         (get-files! #f)
-          (doc-db-clean-files user-db files)))
+        (doc-db-clean-files user-db files))))
+
+  (define main-db (find-doc-db-path latex-dest #f))
+  (define user-db (find-doc-db-path latex-dest #t))
+
+  (define (make-loop first? iter)
+    (let ([infos (filter-not info-failed? infos)]
+          [src->info (make-hash)]
+          [out-path->info-cache (make-hash)])
+      (tidy-database)
       ;; Check for duplicate definitions
       (when first?
         (log-setup-info "checking for duplicates")
@@ -532,6 +544,10 @@
                        (add1 count)))
           (make-loop #f (add1 iter))))))
 
+  (unless infos
+    ;; since we won't use `make-loop':
+    (tidy-database))
+
   (when infos
     (make-loop #t 0)
     ;; cache info to disk

collects/setup/setup-cmdline.rkt

@@ -33,6 +33,18 @@
      #:program long-name
      #:argv argv
      #:once-each
+     [("-j" "--workers") workers "Use <#> parallel-workers"
+      (add-flags `((parallel-workers ,(string->number workers))))]
+     [("--only") "Set up only specified <collection>s"
+      (add-flags '((make-only #t)))]
+     #:multi
+     [("-P") owner package-name maj min
+      "Setup specified PLaneT packages only"
+      (set! x-specific-planet-packages (cons (list owner package-name maj min)
+                                             x-specific-planet-packages))]
+     #:once-each
+     [("--tidy") "Clear references to removed, even if not a specified <collection>"
+      (add-flags '((make-tidy #t)))]
      [("-c" "--clean") "Delete existing compiled files; implies -nxi"
       (add-flags '((clean #t)
                    (make-zo #f)
@@ -40,10 +52,10 @@
                    (make-launchers #f)
                    (make-info-domain #f)
                    (make-docs #f)))]
-     [("-j" "--workers") workers "Use <#> parallel-workers"
-      (add-flags `((parallel-workers ,(string->number workers))))]
      [("-n" "--no-zo") "Do not produce .zo files"
       (add-flags '((make-zo #f)))]
+     [("--trust-zos") "Trust existing .zos (use only with prepackaged .zos)"
+      (add-flags '((trust-existing-zos #t)))]
      [("-x" "--no-launcher") "Do not produce launcher programs"
       (add-flags '((make-launchers #f)))]
      [("-i" "--no-install") "Do not call collection-specific pre-installers"
@@ -54,12 +66,16 @@
       (add-flags '((make-info-domain #f)))]
      [("-D" "--no-docs") "Do not compile .scrbl files and do not build documentation"
       (add-flags '((make-docs #f)))]
+     [("--doc-pdf") dir "Build doc PDFs, write to <dir>"
+      (add-flags `((doc-pdf-dest ,dir)))]
      [("-U" "--no-user") "Do not setup user-specific collections (implies --no-planet)"
       (add-flags '((make-user #f) (make-planet #f)))]
      [("--no-planet") "Do not setup PLaneT packages"
       (add-flags '((make-planet #f)))]
      [("--avoid-main") "Do not make main-installation files"
       (add-flags '((avoid-main-installation #t)))]
+     [("--mode") mode "Select a compilation mode"
+      (add-flags `((compile-mode ,mode)))]
      [("-v" "--verbose") "See names of compiled files and info printfs"
       (add-flags '((verbose #t)))]
      [("-m" "--make-verbose") "See make and compiler usual messages"
@@ -67,18 +83,8 @@
      [("-r" "--compile-verbose") "See make and compiler verbose messages"
       (add-flags '((make-verbose #t)
                    (compiler-verbose #t)))]
-     [("--trust-zos") "Trust existing .zos (use only with prepackaged .zos)"
-      (add-flags '((trust-existing-zos #t)))]
      [("-p" "--pause") "Pause at the end if there are any errors"
       (add-flags '((pause-on-errors #t)))]
-     [("--force") "Treat version mismatches for archives as mere warnings"
-      (add-flags '((force-unpacks #t)))]
-     [("-a" "--all-users") "Install archives to main (not user-specific) installation"
-      (add-flags '((all-users #t)))]
-     [("--mode") mode "Select a compilation mode"
-      (add-flags `((compile-mode ,mode)))]
-     [("--doc-pdf") dir "Build doc PDFs, write to <dir>"
-      (add-flags `((doc-pdf-dest ,dir)))]
      [("-l") => (lambda (flag . collections)
                   (check-collections short-name collections)
                   (cons 'collections (map list collections)))
@@ -86,11 +92,11 @@
      [("-A") => (λ (flag . archives)
                   (cons 'archives archives))
              '("Unpack and install <archive>s" "archive")]
-     #:multi
+     [("--force") "Treat version mismatches for archives as mere warnings"
-     [("-P") owner package-name maj min
+      (add-flags '((force-unpacks #t)))]
-      "Setup specified PLaneT packages only"
+     [("-a" "--all-users") "Install archives to main (not user-specific) installation"
-      (set! x-specific-planet-packages (cons (list owner package-name maj min)
+      (add-flags '((all-users #t)))]
-                                             x-specific-planet-packages))]
+     
      #:handlers
      (lambda (collections/archives . rest)
        (let ([pre-archives (if (and (pair? collections/archives)

collects/setup/setup-unit.rkt

@@ -165,13 +165,17 @@
     (if (make-planet) (specific-planet-dirs) null))
 
   (define no-specific-collections?
-    (and (null? x-specific-collections) (null? x-specific-planet-dirs)))
+    (and (null? x-specific-collections)
+         (null? x-specific-planet-dirs)
+         (not (make-only))))
 
   ;; ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
   ;;              Find Collections                 ;;
   ;; ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
 
-  (define (make-cc* collection path omit-root info-root info-path info-path-mode shadowing-policy)
+  (define (make-cc* collection path omit-root info-root 
+                    info-path info-path-mode shadowing-policy 
+                    main?)
     (define info
       (or (with-handlers ([exn:fail? (warning-handler #f)]) (getinfo path))
           (lambda (flag mk-default) (mk-default))))
@@ -197,7 +201,8 @@
                   info
                   omit-root
                   info-root info-path info-path-mode
-                  shadowing-policy)))
+                  shadowing-policy
+                  main?)))
 
   (define ((warning-handler v) exn)
     (setup-printf "WARNING" "~a" (exn->string exn))
@@ -212,7 +217,8 @@
                           #:omit-root [omit-root #f]
                           #:info-root [given-info-root #f]
                           #:info-path [info-path #f]
-                          #:info-path-mode [info-path-mode 'relative])
+                          #:info-path-mode [info-path-mode 'relative]
+                          #:main? [main? #f])
     (define info-root
       (or given-info-root
           (ormap (lambda (p)
@@ -243,7 +249,8 @@
                 info-path-mode
                 ;; by convention, all collections have "version" 1 0. This
                 ;; forces them to conflict with each other.
-                (list (cons 'lib (map path->string collection-p)) 1 0)))
+                (list (cons 'lib (map path->string collection-p)) 1 0)
+                main?))
     (when new-cc
       (hash-update! collection-ccs-table
                     collection-p
@@ -281,7 +288,8 @@
                    #f ; don't need info-root; absolute paths in cache.rktd will be ok
                    (get-planet-cache-path)
                    'abs
-                   (list `(planet ,owner ,pkg-file ,@extra-path) maj min))))
+                   (list `(planet ,owner ,pkg-file ,@extra-path) maj min)
+                   #f)))
 
   ;; planet-cc->sub-cc : cc (listof bytes [encoded path]) -> cc
   ;; builds a compilation job for the given subdirectory of the given cc this
@@ -309,14 +317,16 @@
         #:unless (skip-collection-directory? collection)
         #:when (directory-exists? (build-path cp collection)))
     (collection-cc! (list collection)
-                    #:path (build-path cp collection)))
+                    #:path (build-path cp collection)
+                    #:main? (equal? cp (find-collects-dir))))
   (let ([main-collects (find-collects-dir)])
     (define (cc! col #:path path)
       (collection-cc! col
                       #:path path
                       #:info-root main-collects
                       #:info-path-mode 'abs-in-relative
-                      #:omit-root 'dir))
+                      #:omit-root 'dir
+                      #:main? #t))
     (for ([c+p (in-list (links #:user? #f #:with-path? #t))])
       (cc! (list (string->path (car c+p)))
            #:path (cdr c+p)))
@@ -371,23 +381,7 @@
             (let loop ([l collections-to-compile])
               (append-map (lambda (cc) (cons cc (loop (get-subs cc)))) l))))
 
-  (define (collection-tree-map collections-to-compile
+  (define (collection-tree-map collections-to-compile)
-                               #:skip-path [orig-skip-path (and (avoid-main-installation)
-                                                                (find-collects-dir))])
-    (define skip-path
-      (and orig-skip-path
-           (path->bytes (simplify-path (if (string? orig-skip-path)
-                                         (string->path orig-skip-path)
-                                         orig-skip-path)
-                                       #f))))
-    (define (skip-path? path)
-      (and skip-path
-           (let ([b (path->bytes (simplify-path path #f))]
-                 [len (bytes-length skip-path)])
-             (and ((bytes-length b) . > . len)
-                  (bytes=? (subbytes b 0 len) skip-path)))
-           path))
-
     (define (build-collection-tree cc)
       (define (make-child-cc parent-cc name)
         (collection-cc! (append (cc-collection parent-cc) (list name))
@@ -405,9 +399,7 @@
         (if (eq? 'all omit)
             (values null null)
             (partition (lambda (x) (directory-exists? (build-path ccp x)))
-                       (filter (lambda (p)
+                       (filter (lambda (p) (not (member p omit)))
-                                 (not (or (member p omit)
-                                          (skip-path? p))))
                                (directory-list ccp)))))
       (define children-ccs
         (map build-collection-tree
@@ -449,7 +441,8 @@
 
   (define (check-against-all given-ccs nothing-else-to-do?)
     (when (and (null? given-ccs)
-               nothing-else-to-do?)
+               nothing-else-to-do?
+               (not (make-tidy)))
       (setup-printf #f "nothing to do")
       (exit 1))
     (define (cc->name cc)
@@ -507,14 +500,19 @@
       null))
 
   (define top-level-plt-collects
+    ((if (avoid-main-installation)
+         (lambda (l) (filter (lambda (cc) (not (cc-main? cc))) l))
+         values)
      (if no-specific-collections?
          all-collections
          (check-against-all
           (append-map
            (lambda (c)
+             (define sc (map (lambda (s) (if (path? s) (path->string s) s))
+                             c))
              (define elems
                (append-map (lambda (s) (map string->path (regexp-split #rx"/" s)))
-                        c))
+                           sc))
              (define ccs (collection->ccs elems))
              (when (null? ccs)
                ;; let `collection-path' complain about the name, if that's the problem:
@@ -524,10 +522,10 @@
                ;; spell the name
                (setup-printf "WARNING"
                              "nothing to compile in a given collection path: \"~a\""
-                          (string-join c "/")))
+                             (string-join sc "/")))
              ccs)
            x-specific-collections)
-       (null? planet-collects))))
+          (null? planet-collects)))))
 
   (define planet-dirs-to-compile
     (sort-collections
@@ -818,8 +816,10 @@
     (setup-printf #f "--- compiling collections ---")
     (if ((parallel-workers) . > . 1)
       (begin
+        (when (or no-specific-collections?
+                  (member "racket" x-specific-collections))
           (for/fold ([gcs 0]) ([cc (in-list (collection->ccs (list (string->path "racket"))))])
-          (compile-cc cc 0))
+            (compile-cc cc 0)))
         (managed-compile-zo (collection-file-path "parallel-build-worker.rkt" "setup"))
         (with-specified-mode
           (lambda ()
@@ -854,32 +854,26 @@
     (define ht (make-hash))
     (define ht-orig (make-hash))
     (define roots (make-hash))
-    (for ([cc ccs-to-compile])
+    (define (get-info-ht info-root info-path info-path-mode)
-      (define-values [path->info-relative info-relative->path]
+      (define-values (path->info-relative info-relative->path)
         (apply values
                (hash-ref roots
-                         (cc-info-root cc)
+                         info-root
                          (lambda ()
                            (define-values [p-> ->p]
-                             (if (cc-info-root cc)
+                             (if info-root
-                                 (make-relativize (lambda () (cc-info-root cc))
+                                 (make-relativize (lambda () info-root)
                                                   'info
                                                   'path->info-relative
                                                   'info-relative->path)
                                  (values #f #f)))
-                           (hash-set! roots (cc-info-root cc) (list p-> ->p))
+                           (hash-set! roots info-root (list p-> ->p))
                            (list p-> ->p)))))
-      (define domain
+      (hash-ref ht info-path
-        (with-handlers ([exn:fail? (lambda (x) (lambda () null))])
-          (dynamic-require (build-path (cc-path cc) "info.rkt")
-                           '#%info-domain)))
-      ;; Check whether we have a table for this cc's info-domain cache:
-      (define t
-        (hash-ref ht (cc-info-path cc)
                 (lambda ()
                   ;; No table for this root, yet. Build one.
                   (define l
-              (let ([p (cc-info-path cc)])
+                    (let ([p info-path])
                       (if (file-exists? p)
                           (with-handlers ([exn:fail? (warning-handler null)])
                             (with-input-from-file p read))
@@ -898,7 +892,7 @@
                          (define p (if (bytes? a) (bytes->path a) a))
                          ;; Check that the path is suitably absolute or relative:
                          (define dir
-                     (case (cc-info-path-mode cc)
+                           (case info-path-mode
                              [(relative abs-in-relative)
                               (or (and (list? p)
                                        (info-relative->path p))
@@ -934,13 +928,26 @@
                            (set! all-ok? #f)])))
                   ;; Record the table loaded for this collection root in the
                   ;; all-roots table:
-            (hash-set! ht (cc-info-path cc) t)
+                  (hash-set! ht info-path t)
                   ;; If anything in the "cache.rktd" file was bad, then claim
                   ;; that the old table was empty, so that we definitely write
                   ;; the new table.
-            (hash-set! ht-orig (cc-info-path cc)
+                  (hash-set! ht-orig info-path
                              (and all-ok? (hash-copy t)))
                   t)))
+    ;; process all collections:
+    (for ([cc ccs-to-compile])
+      (define domain
+        (with-handlers ([exn:fail? (lambda (x) (lambda () null))])
+          (dynamic-require (build-path (cc-path cc) "info.rkt")
+                           '#%info-domain)))
+      ;; Get the table for this cc's info-domain cache:
+      (define t (get-info-ht (cc-info-root cc)
+                             (cc-info-path cc)
+                             (cc-info-path-mode cc)))
+      (define-values (path->info-relative info-relative->path)
+        ;; Look up value that was forced by by `get-info-ht':
+        (apply values (hash-ref roots (cc-info-root cc))))
       ;; Add this collection's info to the table, replacing any information
       ;; already there, if the collection has an "info.ss" file:
       (when (or (file-exists? (build-path (cc-path cc) "info.rkt"))
@@ -954,6 +961,19 @@
                        ;; Use absolute path:
                        (path->bytes (cc-path cc)))
                    (cons (domain) (cc-shadowing-policy cc)))))
+    ;; In "tidy" mode, make sure we check each "cache.rktd":
+    (when (make-tidy)
+      (for ([c (in-list (current-library-collection-paths))])
+        (when (and (directory-exists? c)
+                   (not (and (avoid-main-installation)
+                             (equal? c (find-collects-dir)))))
+          (define info-path (build-path c "info-domain" "compiled" "cache.rktd"))
+          (when (file-exists? info-path)
+            (get-info-ht c info-path 'relative))))
+      (when (make-user)
+        (define info-path (get-planet-cache-path))
+        (when (file-exists? info-path)
+          (get-info-ht #f info-path 'abs))))
     ;; Write out each collection-root-specific table to a "cache.rktd" file:
     (hash-for-each ht
       (lambda (info-path ht)
@@ -998,6 +1018,7 @@
               name-str
               (if no-specific-collections? #f (map cc-path ccs-to-compile))
               latex-dest auto-start-doc? (make-user)
+              (make-tidy) (avoid-main-installation)
               (lambda (what go alt) (record-error what "Building docs" go alt))
               setup-printf))
 

collects/setup/setup.rkt

@@ -22,6 +22,8 @@
                #:make-docs? [make-docs? #t]
                #:make-user? [make-user? #t]
                #:clean? [clean? #f]
+               #:tidy? [tidy? #f]
+               #:avoid-main? [avoid-main? #f]
                #:jobs [parallel #f])
   (define-unit set-options@
     (import setup-option^ compiler^)
@@ -43,12 +45,21 @@
     (when collections
       (specific-collections collections))
 
+    (when (or planet-specs collections)
+      (make-only #t))
+
     (unless make-user?
       (make-user #f))
 
     (unless make-docs?
       (make-docs #f))
 
+    (when tidy?
+      (make-tidy #t))
+
+    (when avoid-main?
+      (avoid-main-installation #t))
+    
     (when clean?
       (clean #t)
       (make-zo #f)