From 77dc6a2af65979c009e21300e8ac0f5fa43afa8f Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Georges=20Dup=C3=A9ron?= Date: Tue, 5 Mar 2019 09:17:31 +0100 Subject: [PATCH] Preliminary notes on the build system --- build-system/mek.md | 94 +++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 94 insertions(+) create mode 100644 build-system/mek.md diff --git a/build-system/mek.md b/build-system/mek.md new file mode 100644 index 0000000..99053ab --- /dev/null +++ b/build-system/mek.md @@ -0,0 +1,94 @@ +Problems solved: +* Forgotten dependencies: build in isolated environment +* Reactive builds (auto-rebuild): mek watch +* Access to the build environment: eval $(mek) +* What changed since last good build(s) without doing git commit all the time: mek source --good (also: cd toto.err.source/) +* Track downloads of third-party components: build in isolated environment without network access +* Archive of source and all dependencies, including downloaded stuff + +Usage: +``` +mek build toto # builds toto or toto.err in case of an error. + # Cached errors are displayed early, but the build is retried in case it was a cosmic ray. + +mek clean # only necessary if a hardware or system failure (e.g. out of memory) made a build appear as SUCCESSFUL when it was not + # (e.g. a poorly-written test that expected an error, caught an OOM but the real execution would not have produced an error) + +mek watch # rebuilds everything incrementally as soon as sources are modified (threads 1 and 3 or 2 and 3, see below). +mek daemon # same but with '&' after initialization +mek watch toto # rebuilds toto incrementally as soon as one of its transitive dependencies is modified + +ls # show progress next to (future) targets and a symlink to the backup of their source +ls _build.err # stderr of the last build (if non-empty) +ls _build/err # stderr of the last build +ls _build/source/ # source of the last build +ls _build/good.source/ # source of the last successful build +ls _build/err.source/ # source of the last failed build +ls toto.err # stderr when building toto (if non-empty) +ls toto.source # source of the last build of toto +ls toto.good.source # source of the last successful build of toto +ls toto.err.source # source of the last failed build of toto + +mek source # dumps the source of the last build +mek source --good # dumps the source of the last successful build +mek source --err # dumps the source of the last failed build +mek source toto # dumps the source of the last successful build of toto +mek source toto.err # dumps the source of the last failed build of toto + +eval $(mek) # builds and adds the output bin directory to $PATH etc. +eval "$(mek)" # same +$(mek) # same +. mekfile.sh # same +mek; copy-paste output # same +$(mek daemon) # same but detaches right away and builds incrementally in the background + +eval $(mek build toto) # adds an output bin directory containing only toto (can be a collection of outputs) to $PATH etc. +mek shell toto # subshell in the directory and with the env of the recipe that builds toto +mek shell toto.err # same as above +eval $(mek shell toto) # cd to build directory for toto and add to $PATH etc. + +mek archive toto # toto.tar now contains the source of toto and of all its dependencies with a build.sh +``` + +Example session: +``` +$ $(mek daemon) +$ ls +(toto 60%) toto.source toto.c toto.h +$ ls +toto toto.source toto.c toto.h +$ echo "bad stuff" >> toto.h +$ ls +toto (34%) toto.c toto.h +$ ls +toto toto.source toto.err toto.err.source toto.c toto.h +$ meld $(mek source --good) $(mek source) +$ echo "fix bug" >> toto.h +$ ls +toto toto.source toto.c toto.h +``` + +In a `mekfile`: +``` +toto: toto.h +# automatic dependency on the gcc executable and on toto.c +# "," is the unquote from scheme, it escapes from the implicitly-quoted shell command. +# Maybe gcc (a variable pointing to a third-party tool) should be distinguished from toto.c (a local file) +> ,gcc ,toto.c -o ,output +``` + +* Uses hashes, not timestamps +* Builds are done in isolated environments (cd, proot, Nix, chroot, container, VM, whatever is available), which only contain the dependencies. +* Transitions are atomic (mv of a symlink), so that the bin folder in the $PATH contains executables from the same version of the source, not toto from one version and tata from another. +* Two builds can run in parallel without interfering with each other, yet if work can be shared it will be. + +Note about build threads: +* Invariant when threads 2 and 3 are enabled: if the user constantly modifies a file, e.g. `while sleep 1; do date > somesource; done` which produces an infinite stream of changes, and one of the versions causes the compiler to deadlock / go in an infinite loop, `mek` will still eventually produce an infinite stream of output binaries, where the latest produced binary is not based on a "very old" change (i.e. it is not a queue of jobs that grows indefinitely). It tries to build the latest changes, but it is resistant to the compiler hanging forever on some inputs, and it is resistant to a rapid stream of changes that could cause a naive algorithm to always restart without ever finishing any build. +* Thread 1 builds, and then checks for new changes. (If the build gets stuck in an infinite loop or deadlocks, the build never finishes and new changes are never taken into account.) +* Thread 2 builds, but aborts and restarts as soon as there's any change. Upon completion it kills threads 1 and 3 because it got a successful build of a more recent version. (If you're constantly modifying and the build takes a while, it never gets a chance to finish.) +* Thread 3 works like thread 1 but aborts if there are new changes and the build took longer than a timeout, e.g. twice the time of the last successful build by any thread and increasing geometrically + + +Random requirements: +* meta-rules: a rule which returns rules. Can be memoized easily, and be part of the normal reactive flow. +* Easy changes of config: dev / build, -O3, -Odebug etc. \ No newline at end of file