raco pkg: add --binary-lib package stripping to remove docs

The `--binary-lib` mode is like `--binary`, but it also omits
documentation to form a "binary library package".

The `--binary-lib` flag and "X-lib" vs. "X-doc" approaches solve the
same problem with different trade-offs:

 * When a package is split into "-lib" and "-doc" packages, then it's
   easier install non-documentation parts, and it's possible to
   install them from source. A programmer has to work more to split
   the packages, however, and the library and its implementation must
   have separate sources (i.e., no or restricted in-source
   documentation).

 * When a package is just "X", then users can install a no-source,
   no-documentation version by specifying `--binary-lib`, but only
   when the package is available from some catalog and provider in
   built form (such as from a distribution site or a package-build
   service).

In the long run, I think that relying on `--binary-lib will be best
and typical for most packages. The "X-lib" plus "X-doc" approach
that's common in the current distribution's packages, meanwhile, will
likely stick around for basic packages that are commonly useful in
constrained settings (including the setting of a package-build
service).
This commit is contained in:
Matthew Flatt 2014-08-15 10:33:20 +01:00
parent 53290089b5
commit 05523a0b42
12 changed files with 143 additions and 64 deletions

View File

@ -143,7 +143,7 @@ dependency.}
[#:checksum checksum (or/c #f string?) #f]
[#:in-place? in-place? boolean? #f]
[#:namespace namespace namespace? (make-base-namespace)]
[#:strip strip (or/c #f 'source 'binary) #f]
[#:strip strip (or/c #f 'source 'binary 'binary-lib) #f]
[#:use-cache? use-cache? boolean? #f]
[#:quiet? quiet? boolean? #t])
(values string? path? (or/c #f string?) boolean? (listof module-path?))]{
@ -190,7 +190,7 @@ The package lock must be held (allowing writes if @racket[set?] is true); see
@defproc[(pkg-create [format (or/c 'zip 'tgz 'plt 'MANIFEST)]
[dir path-string?]
[#:source source (or/c 'dir 'name)]
[#:mode mode (or/c 'as-is 'source 'binary 'built)]
[#:mode mode (or/c 'as-is 'source 'binary 'binary-lib 'built)]
[#:dest dest-dir (or/c (and/c path-string? complete-path?) #f)]
[#:quiet? quiet? boolean? #f]
[#:from-command-line? from-command-line? boolean? #f])
@ -215,7 +215,7 @@ is true, error messages may suggest specific command-line flags for
[#:use-cache? use-cache? boolean? #t]
[#:quiet? boolean? quiet? #f]
[#:from-command-line? from-command-line? boolean? #f]
[#:strip strip (or/c #f 'source 'binary) #f]
[#:strip strip (or/c #f 'source 'binary 'binary-lib) #f]
[#:link-dirs? link-dirs? boolean? #f])
(or/c 'skip
#f
@ -255,7 +255,7 @@ The package lock must be held; see @racket[with-pkg-lock].}
[#:use-cache? use-cache? quiet? #t]
[#:quiet? boolean? quiet? #f]
[#:from-command-line? from-command-line? boolean? #f]
[#:strip strip (or/c #f 'source 'binary) #f]
[#:strip strip (or/c #f 'source 'binary 'binary-lib) #f]
[#:link-dirs? link-dirs? boolean? #f])
(or/c 'skip
#f
@ -318,7 +318,7 @@ The package lock must be held to allow reads; see
[#:strict-doc-conflicts? strict-doc-conflicts? boolean? #f]
[#:quiet? boolean? quiet? #f]
[#:from-command-line? from-command-line? boolean? #f]
[#:strip strip (or/c #f 'source 'binary) #f])
[#:strip strip (or/c #f 'source 'binary 'binary-lib) #f])
(or/c 'skip
#f
(listof (or/c path-string?

View File

@ -22,6 +22,9 @@ faster. A drawback of a binary package is that it is version-specific
and the source code may be less immediately accessible to other
programmers.
A @deftech{binary library package} is like a @tech{binary package},
but it further omits documentation.
A @deftech{built package} combines source and compiled elements. A
@tech{built package} can be installed more quickly than source, as
long as it is installed for a suitable Racket version, but the source
@ -29,20 +32,21 @@ remains available as a back-up for other programmers to consult or to
re-build for a different Racket version.
A package is not specifically tagged as a @tech{source package},
@tech{binary package}, or @tech{built package}. The different kinds of
@tech{binary package}, @tech{binary library package}, or @tech{built
package}. The different kinds of
packages are just conventions based on the content of the package. All
forms of packages can be mixed in an installation, and a package can
be updated from any form to any other form. Furthermore,
@exec{raco pkg install} and @exec{raco pkg update} support
@DFlag{source} and @DFlag{binary} flags, which can be used to convert
a @tech{built package} to a @tech{source package} or @tech{binary
package} on installation.
@DFlag{source}, @DFlag{binary}, @DFlag{binary-lib} flags, which can be used to convert
a @tech{built package} to a @tech{source package}, @tech{binary
package}, or @tech{binary library package}, respectively, on installation.
Programmers normally supply only @tech{source packages}, while a
package catalog service may convert each source package to a
@tech{binary package} or @tech{built package}. Alternatively,
programmers can create @tech{binary packages} or @tech{built packages}
by using the @command-ref{create} subcommand with @DFlag{binary} or
@tech{binary package}, @tech{binary library package}, or @tech{built package}. Alternatively,
programmers can create @tech{binary packages}, @tech{binary library package}, or @tech{built packages}
by using the @command-ref{create} subcommand with @DFlag{binary}, @DFlag{binary-lib}, or
@DFlag{built}. As a convenience, the @command-ref{create} subcommand
can also create a @tech{source package} from an installed package or
repository checkout, dropping repository elements (such as a
@ -50,7 +54,7 @@ repository checkout, dropping repository elements (such as a
@command-ref{create} by default bundles a package directory as-is,
with no filtering at all.
Creating a @tech{source package}, @tech{binary package}, or
Creating a @tech{source package}, @tech{binary package}, @tech{binary library package}, or
@tech{built package} from a directory or package installation prunes
the following files and directories:
@ -68,10 +72,11 @@ the following files and directories:
Any of the above can be suppressed, however, by a
@racket[source-keep-files] (for @tech{source package} and @tech{built
package} bundling) or @racket[binary-keep-files] (for @tech{binary
package} and @tech{built package} bundling) definition in an
package} bundling), @racket[binary-keep-files] (for @tech{binary
package}, @tech{binary library package} @tech{built package} bundling),
or @racket[binary-lib-keep-files] (for @tech{binary library package} bundling) definition in an
@filepath{info.rkt} in the package or any subdirectory. A
@racket[binary-keep-files] or @racket[binary-keep-files] definition
@racket[source-keep-files], @racket[binary-keep-files], or @racket[binary-lib-keep-files] definition
should bind the name to a list of paths relative to the
@filepath{info.rkt} file.
@ -150,12 +155,33 @@ following files (when they are not pruned):
]
Creating a @tech{binary library package} prunes the following
additional files and directories and makes additional adjustments
compared to a @tech{binary package}:
@itemlist[
@item{directories/files named @filepath{doc} are removed;}
@item{directories/files named in an @filepath{info.rkt} file's
@racket[binary-lib-omit-files] definition are removed; and}
@item{each @filepath{info.rkt} is adjusted to remove any
@racket[scribblings] definition.}
]
Any of the removals can be suppressed through
@racketidfont{binary-lib-keep-files}, in addition to suppressions
through @racketidfont{binary-keep-files}.
Creating a @tech{built package} removes any file or directory that
would be removed for a @tech{source package} and @tech{binary
would be removed for a @tech{source package} @emph{and} @tech{binary
package}, and it performs the @filepath{.html} file updating of a
@tech{binary package}.
Finally, creating @tech{binary package} or @tech{built package}
Finally, creating a @tech{binary package}, @tech{binary library package},
or @tech{built package}
``unmoves'' files that were installed via @racket[move-foreign-libs],
@racket[move-shared-files], or @racket[move-man-pages] definitions in
an @filepath{info.rkt} file, retrieving them if they are not present
@ -169,14 +195,14 @@ support for copying a package-style directory to a given destination
with the same file/directory omissions and updates as
@command-ref{create}.}
@defproc[(generate-stripped-directory [mode (or/c 'source 'binary 'built)]
@defproc[(generate-stripped-directory [mode (or/c 'source 'binary 'binary-lib 'built)]
[src-dir path-string?]
[dest-dir path-string?])
void?]{
Copies @racket[src-dir] to @racket[dest-dir] with file/directory
omissions and updates corresponding the creation of a @tech{source
package}, @tech{binary package}, or @tech{built package} as indicated
package}, @tech{binary package}, @tech{binary library package}, or @tech{built package} as indicated
by @racket[mode].}

View File

@ -6,5 +6,6 @@ MANIFEST
pkg-test1b*
/src-pkgs/
/built-pkgs/
/binary-lib-pkgs/
pkg-add-a/doc
pkg-add-base/doc

View File

@ -9,4 +9,5 @@
(define update-implies '("pkg-y"))
(define binary-omit-files '("nobin-top.txt"))
(define binary-lib-omit-files '("nobinlib-top.txt"))
(define source-omit-files '("nosrc-top.txt"))

View File

@ -3,7 +3,9 @@
(define scribblings '(("x.scrbl")))
(define binary-omit-files '("nobin.txt"))
(define binary-lib-omit-files '("nobinlib.txt"))
(define source-omit-files '("nosrc.txt"))
(define binary-keep-files '("keep.scrbl"))
(define binary-lib-keep-files '("keep2/doc"))
(define source-keep-files '("keep/doc"))

View File

@ -0,0 +1 @@
This file is kept in a binary-library install.

View File

@ -33,6 +33,7 @@
(make-directory* "test-pkgs/src-pkgs")
(make-directory* "test-pkgs/built-pkgs")
(make-directory* "test-pkgs/binary-lib-pkgs")
$ "raco pkg create --from-install --source --dest test-pkgs/src-pkgs pkg-x"
$ "raco pkg create --from-install --source --dest test-pkgs/src-pkgs pkg-y"
$ "raco pkg create --from-install --source --dest test-pkgs/src-pkgs pkg-z"
@ -41,6 +42,10 @@
$ "raco pkg create --from-install --binary --dest test-pkgs pkg-y"
$ "raco pkg create --from-install --binary --dest test-pkgs pkg-z"
$ "raco pkg create --from-install --binary-lib --dest test-pkgs/binary-lib-pkgs pkg-x"
$ "raco pkg create --from-install --binary-lib --dest test-pkgs/binary-lib-pkgs pkg-y"
$ "raco pkg create --from-install --binary-lib --dest test-pkgs/binary-lib-pkgs pkg-z"
$ "raco pkg create --from-install --built --dest test-pkgs/built-pkgs pkg-x"
$ "raco pkg create --from-install --built --dest test-pkgs/built-pkgs pkg-y"
$ "raco pkg create --from-install --built --dest test-pkgs/built-pkgs pkg-z"
@ -80,7 +85,9 @@
(unless (if (file-exists? f)
(file-exists? new-f)
(directory-exists? new-f))
(unless (regexp-match? #rx#"nosrc" (path->bytes f))
(unless (or (regexp-match? #rx#"nosrc" f)
(and (regexp-match? #rx#"keep2" f)
(regexp-match? #rx#"doc" f)))
(error 'diff "missing ~s" new-f)))))))
(unpack "x")
(unpack "y")
@ -90,19 +97,28 @@
(shelly-case
"check content of binary package"
(make-directory* tmp-dir)
(define (unpack-bin name)
(let ([z (path->complete-path (format "test-pkgs/pkg-~a.zip" name))])
(define (unpack-bin name lib?)
(let ([z (path->complete-path (format "test-pkgs/~apkg-~a.zip"
(if lib? "binary-lib-pkgs/" "")
name))])
(define d (build-path tmp-dir name))
(delete-directory/files d #:must-exist? #f)
(make-directory* d)
(parameterize ([current-directory d])
(unzip z)
(for ([f (in-directory)])
(when (or (and (regexp-match? #rx#"(?:[.](?:rkt|scrbl|dep)|_scrbl[.]zo)$" (path->bytes f))
(not (regexp-match? #rx#"(?:info_rkt[.]dep)$" (path->bytes f))))
(regexp-match? #rx#"nobin" (path->bytes f)))
(unless (regexp-match? #rx#"(?:info[.]rkt|keep.scrbl)$" (path->bytes f))
(error 'binary "extra ~s" f)))
(when (regexp-match? #rx#"(?:info[.]rkt|info_rkt.zo)$" (path->bytes f))
(when (or (and (regexp-match? #rx#"(?:[.](?:rkt|scrbl|dep)|_scrbl[.]zo)$" f)
(not (regexp-match? #rx#"(?:info_rkt[.]dep)$" f)))
(and (regexp-match? #rx#"nobin" f)
(or lib?
(not (regexp-match? #rx#"nobinlib" f)))))
(unless (regexp-match? #rx#"(?:info[.]rkt|keep.scrbl)$" f)
(error (if lib? 'binary-lib 'binary) "extra ~s" f)))
(when lib?
(when (and (regexp-match? #rx#"doc$" f)
(not (regexp-match? #rx#"keep2" f)))
(error 'binary-lib "extra ~s" f)))
(when (regexp-match? #rx#"(?:info[.]rkt|info_rkt.zo)$" f)
(parameterize ([current-namespace (make-base-namespace)])
(define i (dynamic-require f '#%info-lookup))
(for ([key '(implies update-implies)])
@ -112,9 +128,12 @@
(for ([s (in-list implies)])
(unless (member s deps)
(error 'binary "bad `~a'" key))))))))))
(unpack-bin "x")
(unpack-bin "y")
(unpack-bin "z")
(unpack-bin "x" #f)
(unpack-bin "y" #f)
(unpack-bin "z" #f)
(unpack-bin "x" #t)
(unpack-bin "y" #t)
(unpack-bin "z" #t)
(delete-directory/files tmp-dir))
(shelly-case
@ -140,17 +159,17 @@
(when (file-exists? f)
(unless (or (file-exists? (build-path sd f))
(file-exists? (build-path bd f)))
(unless (regexp-match? #rx#"(?:[.](?:dep)|_scrbl[.]zo)$" (path->bytes f))
(unless (regexp-match? #rx#"(?:[.](?:dep)|_scrbl[.]zo)$" f)
(error 'built "extra ~s" f))))
(when (regexp-match? #rx#"[.]zo$" (path->bytes f))
(when (regexp-match? #rx#"[.]zo$" f)
(unless (file-exists? (path-replace-suffix f #".dep"))
(error 'build "dep missing for ~s" f)))
(when (regexp-match? #rx#"[.](rkt|scrbl|ss)$" (path->bytes f))
(when (regexp-match? #rx#"[.](rkt|scrbl|ss)$" f)
(let-values ([(base name dir?) (split-path f)])
(unless (file-exists? (build-path (if (eq? base 'relative) 'same base)
"compiled"
(path-add-suffix name #".zo")))
(unless (regexp-match? #rx#"^(?:info.rkt|x/keep.scrbl)$" (path->bytes f))
(unless (regexp-match? #rx#"^(?:info.rkt|x/keep.scrbl)$" f)
(error 'build "compiled file missing for ~s" f)))))))
(parameterize ([current-directory sd])
(for ([f (in-directory)])

View File

@ -3692,7 +3692,7 @@
path-string?)
(#:source (or/c 'dir 'name)
#:pkg-name (or/c #f string?)
#:mode (or/c 'as-is 'source 'binary 'built)
#:mode (or/c 'as-is 'source 'binary 'binary-lib 'built)
#:quiet? boolean?
#:from-command-line? boolean?
#:dest (or/c (and/c path-string? complete-path?) #f))
@ -3710,7 +3710,7 @@
#:ignore-checksums? boolean?
#:strict-doc-conflicts? boolean?
#:use-cache? boolean?
#:strip (or/c #f 'source 'binary)
#:strip (or/c #f 'source 'binary 'binary-lib)
#:link-dirs? boolean?)
(or/c #f 'skip (listof (or/c path-string? (non-empty-listof path-string?)))))]
[pkg-remove
@ -3739,7 +3739,7 @@
#:skip-installed? boolean?
#:quiet? boolean?
#:from-command-line? boolean?
#:strip (or/c #f 'source 'binary)
#:strip (or/c #f 'source 'binary 'binary-lib)
#:link-dirs? boolean?)
(or/c #f 'skip (listof (or/c path-string? (non-empty-listof path-string?)))))]
[pkg-migrate
@ -3752,7 +3752,7 @@
#:use-cache? boolean?
#:quiet? boolean?
#:from-command-line? boolean?
#:strip (or/c #f 'source 'binary))
#:strip (or/c #f 'source 'binary 'binary-lib))
(or/c #f 'skip (listof (or/c path-string? (non-empty-listof path-string?)))))]
[pkg-catalog-show
(->* ((listof string?))
@ -3789,7 +3789,7 @@
[pkg-stage (->* (pkg-desc?)
(#:namespace namespace?
#:in-place? boolean?
#:strip (or/c #f 'source 'binary)
#:strip (or/c #f 'source 'binary 'binary-lib)
#:use-cache? boolean?
#:quiet? boolean?)
(values string?

View File

@ -193,7 +193,9 @@
#:skip-installed? skip-installed
#:update-deps? update-deps
#:update-implies? (not ignore-implies)
#:strip (or (and source 'source) (and binary 'binary))
#:strip (or (and source 'source)
(and binary 'binary)
(and binary-lib 'binary-lib))
#:link-dirs? link-dirs?
(for/list ([p (in-list sources)])
(pkg-desc p a-type* name checksum #f))))))
@ -251,7 +253,9 @@
#:use-cache? (not no-cache)
#:update-deps? (or update-deps auto)
#:update-implies? (not ignore-implies)
#:strip (or (and source 'source) (and binary 'binary))
#:strip (or (and source 'source)
(and binary 'binary)
(and binary-lib 'binary-lib))
#:link-dirs? link-dirs?))))
(setup "updated" no-setup #f setup-collects jobs)))]
;; ----------------------------------------
@ -329,6 +333,7 @@
#:once-any
[#:bool source () ("Strip built elements of the package before installing")]
[#:bool binary () ("Strip source elements of the package before installing")]
[#:bool binary-lib () ("Strip source elements and documentation before installing")]
#:once-any
scope-flags ...
#:once-each
@ -352,7 +357,9 @@
#:ignore-checksums? ignore-checksums
#:strict-doc-conflicts? strict-doc-conflicts
#:use-cache? (not no-cache)
#:strip (or (and source 'source) (and binary 'binary))))))
#:strip (or (and source 'source)
(and binary 'binary)
(and binary-lib 'binary-lib))))))
(setup "migrated" no-setup #f setup-collects jobs)))]
;; ----------------------------------------
[create
@ -369,6 +376,7 @@
[#:bool as-is () "Bundle the directory/package as-is (the default)"]
[#:bool source () "Bundle sources only"]
[#:bool binary () "Bundle bytecode and rendered documentation without sources"]
[#:bool binary-lib () "Bundle bytecode without sources or documentation"]
[#:bool built () "Bundle sources, bytecode and rendered documentation"]
#:once-each
[(#:str dest-dir #f) dest () "Create output files in <dest-dir>"]
@ -385,6 +393,7 @@
#:mode (cond
[source 'source]
[binary 'binary]
[binary-lib 'binary-lib]
[built 'built]
[else 'as-is])))]
;; ----------------------------------------
@ -524,9 +533,10 @@
[#:bool static-link () ("Link in place, promising collections do not change")]
[#:bool copy () ("Treat directory sources the same as other sources")]
[#:bool source () ("Strip packages' built elements before installing; implies --copy")]
[#:bool binary () ("Strip packages' source elements before installing; implies --copy")])
[#:bool binary () ("Strip packages' source elements before installing; implies --copy")]
[#:bool binary-lib () ("Strip source & documentation before installing; implies --copy")])
#:install-copy-defns
[(define link-dirs? (not (or copy source binary)))
[(define link-dirs? (not (or copy source binary binary-lib)))
(define a-type (or (and link 'link)
(and static-link 'static-link)
(and (eq? type 'dir) link-dirs? 'link)

View File

@ -37,6 +37,9 @@
(case mode
[(source) (get-paths 'source-omit-files)]
[(binary) (get-paths 'binary-omit-files)]
[(binary-lib)
(union (get-paths 'binary-omit-files)
(get-paths 'binary-lib-omit-files))]
[(built)
(intersect (get-paths 'source-omit-files)
(get-paths 'binary-omit-files))]))
@ -44,6 +47,9 @@
(case mode
[(source) (get-paths 'source-keep-files)]
[(binary) (get-paths 'binary-keep-files)]
[(binary-lib)
(union (get-paths 'binary-keep-files)
(get-paths 'binary-lib-keep-files))]
[(built)
(union (get-paths 'source-keep-files)
(get-paths 'binary-keep-files))]))
@ -74,7 +80,7 @@
(case mode
[(source)
(regexp-match? #rx#"^(?:compiled|doc)$" bstr)]
[(binary)
[(binary binary-lib)
(or (regexp-match? #rx#"^(?:tests|scribblings|.*(?:[.]scrbl|[.]dep|_scrbl[.]zo))$"
bstr)
(and (regexp-match? #rx"[.](?:ss|rkt)$" bstr)
@ -82,6 +88,10 @@
(file-exists? (let-values ([(base name dir?) (split-path (get-p))])
(build-path base "compiled" (path-add-suffix name #".zo")))))
(immediate-doc/css-or-doc/js?)
(case mode
[(binary-lib)
(regexp-match? #rx#"^(?:doc)$" bstr)]
[else #f])
;; drop these, because they're recreated on fixup:
(equal? #"info_rkt.zo" bstr)
(equal? #"info_rkt.dep" bstr))]
@ -109,17 +119,22 @@
[else #f]))
(define (fixup new-p path src-base level)
(unless (eq? mode 'source)
(define bstr (path->bytes path))
(cond
[(regexp-match? #rx"[.]html$" bstr)
(fixup-html new-p)]
[(and (eq? mode 'binary)
(equal? #"info.rkt" bstr))
(fixup-info new-p src-base level)]
[(and (eq? mode 'binary)
(regexp-match? #rx"[.]zo$" bstr))
(fixup-zo new-p)])))
(case mode
[(binary binary-lib built)
(define bstr (path->bytes path))
(cond
[(regexp-match? #rx"[.]html$" bstr)
(fixup-html new-p)]
[else
(case mode
[(binary binary-lib)
(cond
[(equal? #"info.rkt" bstr)
(fixup-info new-p src-base level mode)]
[(regexp-match? #rx"[.]zo$" bstr)
(fixup-zo new-p)])]
[else (void)])])]
[else (void)]))
(define (explore base ; containing directory relative to `dir`, 'base at start
paths ; paths in `base'
@ -184,10 +199,10 @@
(explore 'same (directory-list dir) drops keeps #f level)
(case mode
[(binary built) (unmove-files dir dest-dir drop-keep-ns)]
[(binary binary-lib built) (unmove-files dir dest-dir drop-keep-ns)]
[else (void)])
(case mode
[(binary) (assume-virtual dest-dir (eq? level 'collection))]
[(binary binary-lib) (assume-virtual dest-dir (eq? level 'collection))]
[else (void)]))
(define (fixup-html new-p)
@ -262,8 +277,8 @@
#:exists 'truncate/replace
(lambda (out) (write-bytes new-bstr out)))))
;; Used in binary mode:
(define (fixup-info new-p src-base level)
;; Used in binary[-lib] mode:
(define (fixup-info new-p src-base level mode)
(define dir (let-values ([(base name dir?) (split-path new-p)])
base))
;; check format:
@ -284,7 +299,7 @@
(#%module-begin
(define assume-virtual-sources #t)
. ,(filter values
(map (fixup-info-definition get-info) defns)))))
(map (fixup-info-definition get-info mode) defns)))))
(define new-content
(match content
[`(module info ,info-lib (#%module-begin . ,defns))
@ -307,7 +322,7 @@
(unless (eq? level 'package)
(managed-compile-zo new-p)))))
(define ((fixup-info-definition get-info) defn)
(define ((fixup-info-definition get-info mode) defn)
(match defn
[`(define build-deps . ,v) #f]
[`(define update-implies . ,v) #f]
@ -318,6 +333,10 @@
`(define move-shared-files . ,v)]
[`(define copy-man-pages . ,v)
`(define move-man-pages . ,v)]
[`(define scribblings . ,v)
(case mode
[(binary-lib) #f]
[else defn])]
[_ defn]))
(define (unmove-files dir dest-dir metadata-ns)