racket/INSTALL.txt

Quick Instructions
==================

On Unix or Mac OS X, `make' (or `make in-place') creates a build in
the "racket" directory --- but for Mac OS X, see the notes below on
`git submodule'.

On Windows, `nmake win32-in-place' creates a build in the "racket"
directory, but see the notes below on `git submodule'.

An in-place build includes (via links) packages that are in the "pkgs"
directory.


Building Racket
===============

The "racket" directory contains a stripped down version of Racket ---
enough to run `raco pkg' to install everything else.

The "pkgs" directory contains the packages that are included in a
Racket distribution, plus some additional packages (such as tests).

On Unix and Mac OS X, you can build Racket plus the included packages
with `make in-place' (or just `make'). On Windows, you can use `nmake
win32-in-place', as long as you're using Microsoft Visual Studio 9.0
(2008). The resulting build is in the "racket" subdirectory. If you
want more control over the process, see below.

On Mac OS X and Windows, you'll need native-library packages in the
"native-pkgs" directory. In the git repostory, "native-pkgs" is a git
submodule, so you'll need
   git submodule init
   git submodule update
to get it set up.

To install a subset of the packages in "pkgs", supply `PKGS' value to
`make'. For example,

  make PKGS="gui-lib readline-lib"

links only the "gui-lib" and "readline-lib" packages and their
dependencies. The default value of `PKGS' reaches all packages in
"pkgs" among its dependencies. See "Linking Packages for Development
Mode", below, for more information on package links and using `again'
or `LINK_MODE' to use remembered `PKGS' values from previous `make
in-place's.

Using `make' (or `make in-place') sets the installation's name to
"development", unless the installation has been previously configured
(i.e., unless the "racket/etc/config.rktd" file exists). The
installation name affects, for example, the directory where
user-specific documentation is installed. Using `make' also sets the
default package scope is set to `installation', which means that
package are installed by default into the installation's space instead
of user-specific space. The name and/or default-scope configuration
can be changed through `raco pkg config'.

Note that `make -j <n>' controls parallelism for the makefile part of
a build, but not for the `raco setup' part. To control both the
makefile and the `raco setup' part, use

 make CPUS=<n>

which recurs with `make -j <n> JOB_OPTIONS="-j <n>"'.


Building Racket Pieces
======================

Instead of just using `make in-place', you can take more control over
the build.

Building Base Racket
--------------------

Instead of using the top-level makefile, you can go into "racket/src"
and follow the "README" there, which gives you more configuration
options.

If you don't want any special configuration and you just want the base
build, you can use `make base' (or `nmake win32-base') with the
top-level makefile.

(The Racket base does not require additional native libraries to run,
but under Windows, encoding-conversion, extflonum, and SSL
functionality is hobbled until native libraries from the
`racket-win32-i386' or `racket-win32-x86_64' package are installed.)

On Unix or Mac OS X, you can use `make in-place' and still set
`configure' options by supplying `CONFIGURE_ARGS_qq'. (The `_qq'
suffix on the variable name is a convention that indicates that
single- and double-quote marks are allowed in the value.)

On all platforms, `JOB_OPTIONS' as a makefile variable and
`PLT_SETUP_OPTIONS' as an environment variable are passed on to the
`raco setup' that is used to build base libraries.

Installing Packages
-------------------

In the near future, after you've built the base, you can install
packages via a package-catalog server (ignoring the content of
"pkgs"). That catalog server is not ready, yet.

Linking Packages for Development Mode
-------------------------------------

Aside from the issue of native-library packages, using the
"main-distribution" package corresponds to a build that is like the
main Racket distribution. Furthermore, if you install all of those
packages with `raco pkg install -i --link ...', then you can edit
libraries or update via `git pull' plus `raco setup' (as opposed to
updating or reinstalling packages).

The `pkg-links' target of the makefile links (or re-links) packages
from "pkgs" into the "racket" build. (The `in-place' target of the
makefile uses `pkg-links'.) By default, `pkg-links' starts with the
"main-distribution" and "plt-services" packages and links all of their
dependencies, which includes all of the packages in "pkgs". Specify a
subset of the packages with `PKGS="...."' as an argument to `make
pkg-links'. Make the `pkg-links' target whenever the set of
dependencies can change.

Packages are linked in a database (at "racket/lib/devel-pkgs") that is
added to the installation's search paths. As a result, the links
affect only the build in the "racket" directory, and they are not
mixed with any links that you specifically install in installation
scope. Packages are linked with the `--no-setup' flag (effectively),
which means that a `raco setup' is needed after installing links.

Native-library packages provide (on Mac OS X and Windows) pre-built
native libraries, such as Cairo. They are available through a git
submodule "native-pkgs", so use `git submodule init' and `git
submodule update' as needed.

By default, the `PKGS' value is remembered by `pkg-links', but a
previously remembered value is not used. Supply `LINK_MODE=--restore'
as an argument to to `make' to use the previously remembered `PKGS'
value, if any. The `again' (or `win32-again') makefile target is the
same as `in-place' (or `win32-in-place'), but with
`LINK_MODE=--restore'. To suppress remembering a `PKGS' value, supply
`LINK_MODE='; the `PKGS' value is saved by default because
`LINK_MODE=--save' is the default.

The `pkg-links' makefile target installs links to a specific
"devel-pkgs" directory and adjusts the build's configuration to
include "devel-pkgs" in the search path for packages. As a result, the
package links are installation specific, but still isolated from any
`installation'-scope packages that you install later. The `pkg-links'
target alsosets the default package scope for the build to
`installation' mode. Both of those configuration adjustments are made
only if no configuration file "racket/etc/config.rktd" exists already.

Trying Packages Locally
-----------------------

Suppose that you've built base "racket" and you want to see what it
looks like to install individual packages as if provided by a server
(i.e., not through development-mode links).

Use `make local-catalog' to create a package catalog that provides
mappings for all of the packages in "pkgs" as well as packages for
pre-built native libraries.

To install a package, try

 racket/bin/raco pkg install -i --catalog build/local/catalog --deps search-auto <pkg-name>

The `-i' flag makes the package install specific to the build in the
"racket" directory. The `--catalog build/local/catalog' causes the
installation to use the catalog created by the `local-catalog'
makefile target. The `--deps search-auto' flag installs dependencies
(from the local catalog) automatically.

To remove the package, try

 racket/bin/raco pkg remove -i --auto <pkg-name>

The `--auto' flag undoes automatic installs from `--deps search-auto'.


Building Installers
===================

To build distribution installers, do not use `make in-place' or just
`make', but instead start from a clean repository.

Use one non-Windows machine as a server[*], where packages will be
pre-built. Then, create platform-specific installers on N client
machines, each of which contacts the server machine to obtain
pre-built packages. The server can act as a client, naturally, to
create an installer for the server's platform.

 [*] GNU `make' is required on the server machine, but any `make'
     should work on client machines.

Running Build Farms
-------------------

The `installers' target of the makefile will do everything to generate
installers: build a server on the current machine, run clients on
hosts specified via CONFIG, and start/stop VirtualBox virtual machines
that act as client machines.

See

 pkgs/distro-build/doc.txt

for a description of the site-configuration module and requirements on
client hosts.

If "my-site-config.rkt" is a configuration module, then

 make installers CONFIG=my-site-config.rkt

drives the build farm, and the resulting installers are in
"build/installers", with a hash table mapping descriptions to
installer filenames in "build/installer/table.rktd". A log file
for each client are written to "build/log".

The default CONFIG path is "build/site.rkt", so you could put your
configuration file there and omit the `CONFIG' argument to
`make'. Supply `CONFIG_MODE=...' to pass a configuration mode on to
your site-configuration module (accessible via the `current-mode'
parameter). Supply `CLEAN_MODE=--clean' to make the default `#:clean?'
configuration for a client #t instead of #f.

A configuration file can specify the packages to include, host address
of the server, distribution name, installer directory, and
documentation search URL, but defaults can be provided as `make'
arguments via `PKGS', `SERVER' plus `SERVER_PORT` plus `SERVER_HOSTS`,
`DIST_NAME', `DIST_BASE', and `DIST_DIR', `DOC_SEARCH',
respectively. The site configuration's top-level options for packages
and documentation search URL are used to configure the set of packages
that are available to client machines to include in installers.

For each installer written to "build/installers", the installer's name
is

 <dist-base>-<version>-<platform>-<dist-suffix>.<ext>

where <dist-base> defaults to "racket" (but can be set via
`DIST_BASE'), <platform> is from `(system-library-subpath #f)' but
normalizing the Windows results to "i386-win32" and "x86_63-win32",
-<dist-suffix> is omitted unless a `#:dist-suffix' string is specified
for the client in the site configuration, and <ext> is
platform-specific: ".sh" for Unix, ".dmg" for Mac Os X, and ".exe" for
Windows.

Generating Installer Web Sites
------------------------------

The `site' target of the makefile uses the `installers' target to
generate a set of installers, and then it combines the installers,
packages, a package catalog, and log files into a directory that is
suitable for access via a web server.

Supply the same `CONFIG=...' and `CONFIG_MODE=...' arguments for
`site' as for `installers'. The configuration file should have a
`#:dist-base-url' entry for the URL where installers and packages will
be made available; the `installers' target uses `#:dist-base-url' to
embed suitable configuration into the installers. Specifically,
installers are configured to access pre-built packages and
documentation from the site indicated by `#:dist-base-url'.

Note that `#:dist-base-url' should almost always end with "/", since
others URLs will be constructed as relative to `#:dist-base-url'.

The site is generated as "build/site" by default. A `#:site-dest'
entry in the configuration file can select an alternate destination.

Use the `site-from-installers' makefile target to perform the part of
`site' that happens after `installers' (i.e., to generate a `site'
from an already-generated set of installers).

Managing Snapshot Web Sites
---------------------------

The `snapshot-site' makefile target uses `site' (so supply the same
`CONFIG=...' and `CONFIG_MODE=...' arguments), and then treats the
resulting site as a snapshot with additional snapshot-management
tasks.

For snapshot management, the destination of the files generated for
`site' (as specified by `#:site-dest') should be within a directory of
snapshots. The configuration file can use `(current-stamp)' to get a
string that represents the current build, and then use the string both
for `#:dist-base-url' and `#:site-dest'. Normally, the stamp string is
a combination of the date and git commit hash.

Snapshot management includes creating an "index.html" file in the
snapshots directory (essentially a copy of the snapshot's own
"index.html") and pruning snapshot subdirectories to keep the number
of snapshots at the amount specified by `#:max-snapshots'
configuration-file entry (with a default value of 5).

Use the `snapshot-at-site' makefile target to perform the part of
`snapshot-site that happens after `site (i.e., to manage snapshots
around an already-generated site).

Separate Server and Clients
---------------------------

Instead of using the `installers' makefile target and a site
configuration file, you can run server and client processes manually.

Roughly, the steps are

 1. On the server machine:
      make server PKGS="..."

    Add `DOC_SEARCH="..."' to the `server' line to build documentation
    so that it redirects to the given URL when a remote search is
    necessary.

    Add `SERVER_PORT=...' to the `server' line to choose the port on
    which the server listens to clients. The default is port 9440.

    Add `SERVER_HOSTS="..."' to determine the interfaces at which the
    server listens. The default is "localhost" which listens only on
    the loopback device (for security). Supply the empty string to
    listen on all interfaces. Supply multiple addresses by separating
    them with a comma.

 2. On each client machine:
      make client SERVER=... PKGS="..."
    or
      nmake win32-client SERVER=... PKGS="..."

    Add `SERVER_PORT=...' to the `client' line if you picked an
    alternate port in the `server` line; it shoul dbe the same as for
    the `server' line.

    Add `DOC_SEARCH="..."' to the `client' line, if needed;
    normally, it should be the same as for the `server' line.

    Add `RELEASE_MODE=--release' to the `client' line to build
    a "release" installer, as opposed to a snapshot installer.

    Add `DIST_NAME="..."' to the `client' line to give an installer a
    different human-readable distribution name, instead of "Racket".

    Add `DIST_BASE="..."' to the `client' line to adjust the installer
    name's prefix, instead of starting with "racket".

    Add `DIST_DIR="..."' to the `client' line to make the installer
    use a different directory name on installation on Unix, instead of
    "racket".

    Add `DIST_SUFFIX="..."' to the `client' line to add a suffix
    string for the installer's name, such as an identifier for a
    particular variant of Linux.

    Add `DIST_DESC="..."' to the `client' line to set the installer's
    description, which is used as a key in the generated table of
    installer files.

    Add `DIST_CATALOGS_q='...'' to the `client' line to declare a
    space-separated sequence of catalog URLs to set an installation's
    initial package catalog URLs. Use the empty string in place of a
    URL to indicate that the default path should be spliced.

    Add `README="..."' to specify a file to download from the server
    to use as the "README" file in the generated installer.

In more detail:

 1a. Build "racket" on a server.
    
     The `base' target of the makefile will do that, if you haven't
     done it already. (The server only works on non-Windows platforms,
     currently.)

 1b. On the server, build packages and start a catalog server.

     The `server-from-base' target of the makefile will do that.

     Alternatively, use the `server' target, which combines `base' and
     `server-from-base' (i.e., steps 1a and 1b).

     The `SERVER_PORT' and `SERVER_HOSTS' variables of the makefile
     configure the prt and interfaces of the server.

     The `PKGS' variable of the makefile determines which packages are
     built for potential inclusion in a distribution.

     The `DOC_SEARCH' variable of the makefile determine a URL that is
     embedded in rendered documentation for cases where a remote
     search is needed (because other documentation is not installed).

     The `SRC_CATALOG' variable determines the catalog that is used to
     get package sources and native-library packages, but a value of
     "local" triggers a bootstrap mode where packages are represented
     by directories in the makefile's directory. For now, "local" is
     the default.

     The server provides README files from the "build/readmes"
     directory. If "README.txt" does not exist when the sever is
     started, when a default file is created (and clients download
     "README.txt" by default).

     If you stop the server and want to restart it, use the
     `built-package-server' makefile target instead of starting over
     with the `server' target.

 2a. On each client (one for each platform to bundle), build "racket".

     This is the same as step 1, but on each client. If the client and
     server are the same, there's nothing more to do for step 3.

 2b. On each client, create an installer.

     The `client' (or `win32-client') target of the makefile will do
     that.

     Provide `SERVER' as the hostname of the server machine, but a
     "locahost"-based tunnel back to the server is more secure and
     avoids the need to specify `SERVER_HOSTS' when starting the
     server in step 1b. Also, provide `SERVER_PORT' if an alternate
     port was specified in step 1b.

     Provide the same `PKGS' (or a subset) as in step 1b if you want a
     different set than the ones listed in the makefile. Similarly,
     `DOC_SEARCH' normally should be the same as in step 1b, but for a
     client, it affects future documentation builds in the
     installation.

     Alternatively, use the `client' target, which combines `base' and
     `client-from-base' (i.e., steps 2a and 2b).

     On Windows, you need NSIS installed, either in the usual location
     or with `makensis' in your command-line path.

     To create a release installer, provide `RELEASE_MODE' as
     "--release" to `make'. A release installer has slightly different
     defaults that are suitable for infrequently updated release
     installations, as opposed to frequently updated snapshot
     installations.

     To change the human-readable name of the distribution as embedded
     in the installer, provide `DIST_NAME' to `make'. The default
     distribution name is "Racket". Whatever name you pick, the Racket
     version number is automatically added for various contexts.

     To change the base name of the installer file, provide `DIST_BASE
     to `make'. The default is "racket".

     To change the directory name for Unix installation, provide
     `DIST_DIR' to `make'. The default is "racket".

     To add an extra piece to the installer's name, such as an
     identifier for a variant of Linux, provide `DIST_SUFFIX' to
     `make'. The default is "", which omits the prefix and its
     preceding hyphen.

     To set the description string for the installer, provide
     `DIST_DESC' to `make'. The description string is recorded
     alongside the installer.

     To set the initial package catalogs URLs for an installation,
     provide `DIST_CATALOGS_q' to `make'. Separate multiple URLs with
     a space, and use an empty string in place of a URL to indicate
     that the default catalogs should be used. The "_q" in the
     variable name indicates that its value can include double quotes
     (but not single quotes) --- which are needed to specify an empty
     string, for example.

     To select a "README" file for the client, provide `README' to
     `make'. The `README' value is used as a file name to download
     from the server.

On each client, step 2b produces a "bundle/installer.txt" file that
contains the path to the generated installer on one line, followed by
the description on a second line. The installer is also uploaded to
the server, which leaves the installer in a "build/installers"
directory and records a mapping from the installer's description to
its filename in "build/installers/table.rktd".

If you provide `JOB_OPTIONS=...' for either a client or server build,
the options are used both for `raco setup' and `raco pkg
install'. Normally, `JOB_OPTIONS' is used to control parallelism.