suzanne.soy/distro-build
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
dea33cdfa1
distro-build/distro-build-client/doc.txt
Sam Tobin-Hochstadt 7e93502014 Remove extra directory.
2014-11-29 14:21:41 -05:00

565 lines
22 KiB
Plaintext
Raw Blame History

Site Configuration Modules
==========================
A build farm is normally run via the `installers', `site', or
`snapshot-site' target of the Racket repository's top-level
makefile. Each of those targets uses `installers', which expects a
`CONFIG=...' argument to specify a configuration module file (or uses
"build/site.rkt" as the default).
A site configuration module starts `#lang distro-build/config' and
uses keywords to specify various options for the configuration. This
format is described is detail further below; for now, it's enough to
know that there are various options, each of which is associated with
a keyword.
The machine where `make installers' is run is the server machine.
The server machine first prepares packages for installation on
clients. The site configuration's top-level entry is consulted for
a `#:pkgs' and/or `#:doc-search' option, which overrides any `PKGS'
and/or `DOC_SEARCH' configuration from the makefile.
The site configuration file otherwise describes and configures
client machines hierarchically, where configuration options
propagate down the hierarchy when they are not overridden more
locally.
Each client is normally built by running commands via `ssh', where the
client's host, `#:host' (with and optional `#:port' and/or `#:user')
indicate the `ssh' target. Each client machine must be set up with a
public-key authentication, because a direct `ssh' is expected to work
without a password prompt. An exception is when the host is
"localhost" and user is #f, in which case a shell is used directly
instead of `ssh'. When `ssh` is used, -R is also used to create a
tunnel back to the server, and the client by default uses that tunnel
for all communication, and the server by default accepts only
connections via "localhost".
On the client machine, all work is performed at a specified
directory as specified by `#:dir'. The directory defaults to
"build/plt" (Unix, Mac OS X) or "build\\plt" (Windows), except when
the host is "localhost" and the client is #f, in which case the
current directory (i.e., the server's directory) is used.
Normally, the client directory is a git clone:
- If the directory exists already on a client machine (and the
machine is not configured for "clean" mode), then if the directory
contains a ".git" subdirectory, it is assumed to be a git clone
and updated with `git pull'. The `git pull' operation can be
disabled by specifying `#:pull?' as #f, and it defaults to #f
in the case that `#:dir' is not specified, the host is
"localhost", and the user is #f.
- If the directory does not exist, a git repository is
cloned. The repository can be specified with `#:repo'. By
default, the server is used as the source git repository (so
that the server and client are in sync), which means that the
server's directory must be a git clone.
Note that neither `ssh' nor `git' turn out to be needed when the host
is "localhost", the user is #f, and the directory is not specified
(which corresponds to the defaults in all cases).
If a build fails for a machine, building continues on other
machines. Success for a given machine means that its installer
ends up in "build/installers" (and failure for a machine means no
installer) as recorded in the "table.rktd" file.
To use the `site' makefile target, the configuration file must at
least provide a `#:dist-base-url' value, which is a URL at which the
site will be made available. To use the `snapshot-site' makefile
target, then `#:site-dest' will need to be specified, normally as a
path that ends with the value produced by `(current-stamp)'.
Hint: When developing a configuration file, use an empty set of
packages to a configuration that works as quickly as possible. Then,
change the list of packages to the ones that you actually want in the
installers.
Machine Requirements
--------------------
Each Unix or Mac OS X client needs the following available:
* SSH server with public-key authentication (except "localhost")
* git (unless the working directory is ready)
* gcc, make, etc.
Each Windows client needs the following:
* SSH server with public-key authentication, providing either a
Windows command line (like freeSSHd) or bash with access to
cmd.exe (like Cygwin's opensshd)
* git (unless the working directory is ready)
* Microsoft Visual Studio (version >= 9.0, <= 12.0), installed
in the default folder:
C:\Program Files\Microsoft Visual Studio <vers>
or
C:\Program Files (x86)\Microsoft Visual Studio <vers>
* Nullsoft Scriptable Install System (NSIS) verstion 2.x, installed
in the default folder:
C:\Program Files\NSIS\makensis.exe
or
C:\Program Files (x86)\NSIS\makensis.exe
or installed so that `makensis' in your PATH.
Site Configuration
-------------------
A site configuration module is normally written in the
`distro-build/config' language. The configuration describes individual
machines, and groups them with `parallel' or `sequential' to indicate
whether the machine's builds should run sequentially or in parallel.
Options specified at `parallel' or `sequential' are propagated to each
machine in the group.
Site-configuration keywords (where <string*> means no spaces, etc.):
#:host <string*> --- defaults to "localhost"
#:name <string> --- defaults to host; this string is recorded as a
description of the installer and can be used in a generated table of
installer links; see also "Names and Download Pages" below
#:port <integer> --- SSH port for the client; defaults to 22
#:user <string*/false> --- SSH user for the client; defaults to #f,
which means the current user
#:dir <path-string> --- defaults to "build/plt" or "build\\plt", or
to the current directory if the host is "localhost" and the user
is #f
#:server <string*> --- the address of the server as accessed by the
client; when ssh remote tunneling works, then "localhost" should
work to reach the server; defaults to the `SERVER' makefile
variable, which in turn defaults to "localhost"
#:server-port <integer> --- the port of the server as accessed by
the client, and also the port started on clients to tunnel back to
the server; defaults to the `SERVER_PORT' makefile variable, which
in turn defaults to 9440
#:server-hosts <list-of-string*> --- addresses that determine the
interfaces on which the server listens; an empty list means all of
the server's interfaces, while `(list "localhost")' listens only
on the loopback device; defaults to the `SERVER_HOSTS` makefile
variable split on commas, which in turn defaults to `(list
"localhost")'
#:repo <string> --- the git repository for Racket; defaults to
"http://<server>:<server-port>/.git"
#:pkgs '(<string*> ...) --- packages to install; defaults to the
`PKGS' makefile variable
#:dist-base-url <string> --- a URL that is used to construct a
default for `#:doc-search' and `#:dist-catalogs', where the
constructed values are consistent with converting a build server's
content into a download site; since URLs are constructed via
relative paths, this URL normally should end with a slash
#:doc-search <string> --- URL to install as the configuration for
remote documentation searches in generated installers; "" is
replaced with the PLT default; defaults to #:dist-base-url (if
present) extended with "doc/local-redirect/index.html", or the
`DOC_SEARCH' makefile variable
#:install-name <string> --- string used as the name of the
installation for package operations in the `user' package scope,
where "" keeps the name as the Racket version; the default is
"snapshot" if the value of `#:release?' is false, "" otherwise.
#:build-stamp <string> --- a string representing a build stamp,
recorded in installes; the default is from the `BUILD_STAMP'
makefile variable or generated if the value of `#:release?' is
false, "" otherwise.
#:dist-name <string> --- the distribution name; defaults to the
`DIST_NAME' makefile variable
#:dist-base <string*> --- the distribution's installater name prefix;
defaults to the `DIST_BASE' makefile variable
#:dist-dir <string*> --- the distribution's installation directory;
defaults to the `DIST_DIR' makefile variable
#:dist-suffix <string*> --- a suffix for the installer's name,
usually used for an OS variant; defaults to the `DIST_SUFFIX'
makefile variable
#:dist-catalogs '(<string> ...) --- catalog URLs to install as the
initial catalog configuration in generated installed, where "" is
replaced with the PLT default catalogs; defaults to
`#:dist-base-url' (if present) extended with "catalogs" in a list
followed by ""
#:readme <string-or-procedure> --- the content of a "README" file
to include in installers, or a function that takes a hash table
for a configuration and returns a string; the default is the
`make-readme' function from `distro-build/readme' (see below)
#:max-vm <real> --- max number of VMs allowed to run with this
machine, counting the machine; defaults to 1
#:vbox <string> --- Virtual Box machine name (as shown, for example,
in the Virtual Box GUI); if provided, the virtual machine is
started and stopped on the server as needed
#:platform <symbol> --- 'unix, 'macosx, 'windows, or 'windows/bash
(which means 'windows though an SSH server providing `bash', such
as Cygwin's); defaults to `(system-type)'
#:configure '(<string> ...) --- arguments to `configure'
#:bits <integer> --- 32 or 64, affects Visual Studio mode
#:vc <string*> --- provided to "vcvarsall/bat" to select the Visual
Studio build mode; the default is "x86" or "x86_amd64", depending
on `#:bits'
#:sign-identity <string> --- provides an identity to be passed to
`codesign` for code signing on Mac OS X (for all executables in a
distribution), where an empty string disables signing; the default
is ""
#:j <integer> --- parallelism for `make' on Unix and Mac OS X and
for `raco setup' on all platforms; defaults to 1
#:timeout <number> --- numbers of seconds to wait before declaring
failure; defaults to 30 minutes
#:clean? <boolean> --- if true, then the build process on the client
machine starts by removing <dir>; set this to #f for a shared repo
checkout; the default is determined by the `CLEAN_MODE' makefile
variable, unless `#:host' is "localhost", `#:user' is #f, and
`#:dir' is not specified, in which case the default is #f
#:pull? <boolean> --- if true, then the build process on the client
machine starts by a `git pull' in `#:dir'; set to #f, for example,
for a repo checkout that is shared with server; the default is #t,
unless `#:host' is "localhost", `#:user' is #f, and `#:dir' is not
specified, in which case the default is #f
#:release? <boolean> --- if true, then create release-mode
installers; the default is determined by the `RELEASE_MODE'
makefile variable
#:source? <boolean> --- determines the default value for
`#:source-runtime?' and `#:source-pkgs'
#:source-runtime? <boolean> --- if true, then create an archive that
contains the run-time system in source form (possibly with built
packages), instead of a platform-specific installer; a #t value
works best when used with a Unix client machine, since Unix
clients typically have no native-library packages; the default is
the value of `#:source?'
#:source-pkgs? <boolean> --- if true, then packages are included in
the installer/archive only in source form; a true value works best
when the `#:source-runtime?' value is also #t; the default is the
value of `#:source?'
#:versionless? <boolean> --- if true, avoids including the Racket
version number in an installer's name or in the installation path;
the default is determined by the `VERSIONLESS_MODE' makefile
variable
#:mac-pkg? <boolean> --- if true, creates a ".pkg" for Mac OS X (in
single-file format) instead of a ".dmg"; the default is #f
#:pause-before <nonnegative-number> --- a pause in seconds to
wait before starting a machine, which may help a virtual machine
avoid confusion from being stopped and started too quickly; the
default is 0
#:pause-after <nonnegative-number> --- a pause in seconds to
wait after stopping a machine; the default is 0
#:custom <hash-table> --- a hash table mapping arbitrary keywords to
arbitrary values; when a value for `#:custom' is overriden in a
nested configuration, the new table is merged with the overriden
one; use such a table for additional configuration entries other
than the built-in ones, where additional entires may be useful to
a `#:readme' procedure
Top keywords (recognized only in the configuration top-level):
#:site-dest <path-string> --- destination for completed build, used
by the `site' and `snapshot-site' makefile targets; the default is
"build/site"
#:pdf-doc? <boolean> --- whether to build PDF documentation when
assembling a site; the default is #f
#:email-to <listof-of-string> --- a list of addresses to receive
e-mail reporting build results; mail is sent via `sendmail'
unless `#:smtp-...' configuration is supplied
#:email-from <string> --- address used as the sender of e-mailed
reports; the first string in `#:email-to' is used by default
#:smtp-server <string*>
#:smtp-port <string*>
#:smtp-connect <'plain, 'ssl, or 'tls>
#:smtp-user <string-or-#f>
#:smtp-password <string-or-#f>
--- configuration for sending e-mail through SMTP instead of
`sendmail'; the `#:smtp-port' default (25, 465, or 587) is picked
based on `#:smtp-connect', which in turn defaults to 'plain;
supply non-#f `#:smtp-user' and `#:smtp-password' when
authentication is required by the server
#:site-help <hash-table> --- hash table of extra "help" information
for entries on a web page created by the `site' and
`snapshot-site' makefile targets; the hash keys are strings for
row labels in the download table (after splitting on "|" and
removing "{...}"), and the values are X-expressions for the help
content
#:site-title <string> --- title for the main page generated
by the `site' or `snapshot-site' makefile target; the default
is "Racket Downloads"
#:max-snapshots <number> --- number of snapshots to keep, used by
the `snapshot-site' makefile target
#:plt-web-style? <boolean> --- indicates whether `plt-web` should
be used to generate a site or snapshot page; the default is #t
More precisely, the `distro-build/config' language is like
`racket/base' except that the module body must have exactly one
expression (plus any number of definitions, etc.) that produces a
site-configuration value. The value is exported as `site-config'
from the module. Any module can act as a site-configuration module
a long as it exports `site-config' as a site-configuration value.
The `distro-build/config' language also adds the following functions
to `racket/base':
(machine <opt-kw> <opt-val> ... ...) -> site-config?
Produces a site configuration based on the given keyword-based
options. The support keyword arguments are described above.
(sequential <opt-kw> <opt-val> ... ... config ...)
-> site-config?
config : site-config?
Produces a site configuration that runs each `config'
sequentially. The support keyword arguments are described above.
(parallel <opt-kw> <opt-val> ... ... config ...)
-> site-config?
config : site-config?
Produces a site configuration that runs each `config' in
parallel. The support keyword arguments are described above.
(site-config? v) -> boolean?
(site-config-tag config) -> (or/c 'machine 'sequential 'parallel)
config : site-config?
(site-config-options config) -> (hash/c keyword? any/c)
config : site-config?
(site-config-content config) -> (listof site-config?)
config : site-config?
Site configuation inspection
(current-mode) -> string?
(current-mode s) -> void?
s : string?
A parameter whose value is the user's requested mode for this
configuration, normally as provided via the makefile's
`CONFIG_MODE' variable. The default mode is "default". The
interpretation of modes is completely up to the
site configuration file.
(current-stamp) -> string?
Returns a string to identify the current build, normally a
combination of the date and a git commit hash.
READMEs
-------
The `distro-build/readme' library provides functions for constructing
a README file's content. Each function takes a hash table mapping
configuration keywords to values.
(make-readme config) -> string
config : hash?
Produces basic "README" content, using information about the
distribution and the Racket license. The content is constructed
using `config' keywords such as `#:name', `#:platform',
`#:dist-name', and `#:dist-catalogs', and sometimes `current-stamp'.
(make-macosx-notes config) -> string
config : hash?
Produces "README" content to tell Mac OS X users how to install a
distribution folder. This function is used by `make-readme' when
`#:platform' in `config' is 'macosx.
Names and Download Pages
------------------------
The `#:name' for an installer is used in an HTML table of download
links by the `site' or `snapshot-site' targets. The names are first
sorted. Then, for the purposes of building the table, a "|" separated
by any number of spaces within a name is treated as a hierarchical
delimiter, while anything within "{" and "}" in a hierarchical level
is stripped from the displayed name along with surrounding spaces (so
that it can affect sorting without being displayed). Anything after ";
" within a "|"-separated part is rendered as a detail part of the
label (e.g., in a smaller font).
For example, the names
"Racket | {2} Linux | 32-bit"
"Racket | {2} Linux | 64-bit; built on Ubuntu"
"Racket | {1} Windows | 32-bit"
"Racket | {1} Windows | 64-bit"
"Racket | {3} Source"
are shown (actually or conceptually) as
Racket
Windows
[32-bit] <build on Ubuntu>
[64-bit]
Linux
[32-bit]
[64-bit]
[Source]
where the square-bracketed entries are hyperlinks and the
angle-bracketed pieces are details.
Examples
--------
** Single Installer **
The simplest possible configuration file is
#lang distro-build/config
(machine)
In fact, this configuration file is created automatically as
"build/site.rkt" (if the file does not exist already) and used as the
default configuration. With this configuration,
make installers
creates an installer in "build/installers" for the platform that is
used to create the installer.
** Installer Web Page ***
To make a web page that serves both a minimal installer and packages,
create a "site.rkt" file with
#lang distro-build/config
(sequential
;; The packages that will be available:
#:pkgs '("main-distribution")
;; FIXME: the URL where the installer and packages will be:
#:dist-base-url "http://my-server.domain/snapshot/"
(machine
;; FIXME: the way the installer is described on the web page:
#:name "Minimal Racket | My Platform"
;; The packages in this installer:
#:pkgs '()))
then
make site CONFIG=site.rkt
creates a "build/site" directory that you can move to your web server's
"snapshot" directory, so that "build/site/index.html" is the main
page, and so on.
** Accumulated Shapshots Web Page **
To make a web site that provides some number (5, by default) of
snapshots, use `(current-stamp)' when constructing the
`#:dist-base-url' value. Also, use `(current-stamp)' as the directory
for assembling the "site":
#lang distro-build/config
(sequential
;; The packages that will be available:
#:pkgs '("gui-lib")
;; FIXME: the URL where the installer and packages will be:
#:dist-base-url (string-append "http://my-server.domain/snapshots/"
(current-stamp) "/")
;; The local directory where a snapshot is written
#:site-dest (build-path "build/site" (current-stamp))
(machine
;; FIXME: the way the installer is described on the web page:
#:name "Minimal Racket | My Platform"
;; The packages in this installer:
#:pkgs '()))
Then,
make snapshot-site CONFIG=site.rkt
creates a "build/site" directory that you can move to your web
server's "snapshots" directory, so that "build/site/index.html" is the
main page that initially points to "build/site/<stamp>/index.html",
and so on. To make a newer snapshot, update the git repository, leave
"build/site" in place, and run
make snapshot-site CONFIG=site.rkt
again. The new installers will go into a new <stamp> subdirectory, and
the main "index.html" file will be rewritten to point to them.
** Multiple Platforms **
A configuration module that drives multiple clients to build
installers might look like this:
#lang distro-build/config
(sequential
#:pkgs '("drracket")
#:server-hosts '() ; Insecure? See below.
(machine
#:desc "Linux (32-bit, Precise Pangolin)"
#:name "Ubuntu 32"
#:vbox "Ubuntu 12.04"
#:host "192.168.56.102")
(machine
#:desc "Windows (64-bit)"
#:name "Windows 64"
#:host "10.0.0.7"
#:server "10.0.0.1"
#:dir "c:\\Users\\racket\\build\\plt"
#:platform 'windows
#:bits 64))
The configuration describes using the hosts "192.168.56.1" and
"10.0.0.7" for Linux and Windows builds, respectively, which are run
one at a time.
The Linux machine runs in VirtualBox on the server machine (in a
virtual machine named "Ubuntu 12.04"). It contacts the server still as
"localhost", and that works because the ssh connection to the Linux
machine creates a tunnel (at the same port as the server's, wjich
defaults to 9440).
The Windows machine uses freeSSHd (not a `bash'-based SSH server like
Cygwin) and communicates back to the server as "10.0.0.1" instead of
using an SSH tunnel. To make that work, `#:server-hosts' is specified
as the empty list to make the server listen on all interfaces (instead
of just "localhost") --- which is possibly less secure than the
default restriction that allows build-server connections only via
"localhost".
With this configuration file in "site.rkt",
make installers CONFIG=site.rkt
produces two installers, both in "build/installers", and a hash table
in "table.rktd" that maps "Linux (32-bit, Precise Pangolin)" to the
Linux installer and "Windows (64-bit)" to the Windows installer.
Reference in New Issue View Git Blame Copy Permalink