#| resolver.ss -- PLaneT client 1. Introduction The PLaneT system is a method for automatically sharing code packages, both as libraries and as full applications, that gives every user of a PLaneT client the illusion of having a local copy of every code package on the server, but is parsimonious in its transmission. It consists of a centralized server that holds all packages and individual clients that hold some portion of the archive locally. Maintenance of that archive should be transparent, and is the complete responsibility of the PLaneT client. 2. Client behavior The PLaneT client receives user requests (i.e., the "(require (planet ...))" forms) and loads the appropriate modules in response. In the course of handling these requests it may download new code packages from the PLaneT server. 2.1 User interface The structure of user PLaneT invocations is listed below. PLANET-REQUEST ::= (planet FILE-NAME PKG-SPEC [PATH ...]?) FILE-NAME ::= string PKG-SPEC ::= string | (FILE-PATH ... PKG-NAME) | (FILE-PATH ... PKG-NAME VER-SPEC) VER-SPEC ::= Nat | (Nat MINOR) MINOR ::= Nat | (Nat Nat) | (= Nat) | (+ Nat) | (- Nat) FILE-PATH ::= string PKG-NAME ::= string OWNER-NAME ::= string PATH ::= string All strings must be legal filename strings. When encountered, a planet-request is interpreted as requiring the given file name from the given logical package, specified by the package spec and the collection specification, if given. If no VER-SPEC is provided, the most recent version is assumed. If no owner-name/path ... clause is provided, the default package is assumed. 2. PLaneT protocol PLaneT clients communicate request PLaneT servers over a TCP connection using a specialized protocol. The protocol is described below. 2.1 Overview 1. PLaneT client establishes TCP connection to PLaneT server. 2. Client transmits a version specifier. 3. Server either refuses that version and closes connection or accepts. 4. Client transmits a sequence of requests terminated by a special end-of-request marker. Simultaneously, server transmits responses to those requests. 5. Once the server has handled every request, it closes the connection. I am concerned about the overhead of opening and closing TCP connections for a large program with many requires, so I want to allow many requests and responses over the same connection. Unfortunately there's a wrinkle: the standard client, implemented the obvious way, would be unable to send more than one request at a time because it gets invoked purely as a response to a require form and must load an appropriate file before it returns. This means I can't batch up multiple requires, at least not with an obvious implementation. A possible solution would be to implement an install program that walks over the syntax tree of a program and gathers all requires, then communicates with the server and learns what additional packages would be necessary due to those requires, and then downloads all of them at once. We would have to implement both methods simultaneously, though, to allow for REPL-based PLaneT use and dynamic-require (unless we want it to be a runtime exception to use PLaneT from the REPL or via dynamic-require, something I'd rather not do), so I want a protocol that will allow both forms of access easily. This protocol does that, and doesn't require too much additional overhead in the case that the client only takes one package at a time. 2.2 Communication Details After a TCP connection is established, the client transmits a VERSION-SPECIFIER: VERSION-SPECIFIER ::= "PLaneT/1.0\n" The server responds with a VERSION-RESPONSE: VERSION-RESPONSE ::= 'ok "\n" | ('invalid string) "\n" where the string in the invalid case is descriptive text intended for display to the user that may indicate some specific message about the nature of the error. If the server sends 'invalid, the server closes the connection. Otherwise, the client may send any number of requests, followed by an end-of-request marker: REQUESTS ::= { REQUEST "\n"}* 'end "\n" REQUEST ::= (SEQ-NO 'get PKG-LANG PKG-NAME (Nat | #f) (Nat | #f) (Nat | #f) [OWNER-NAME PATH ...]?) PKG-LANG ::= String SEQ-NO ::= Nat The fields in a request are a uniquely identifying sequence number, the literal symbol 'get, the name of the package to receive, the required major version and the lowest and highest acceptable version (with #f meaning that there is no constraint for that field, and a #f in major-version field implying that both other fields must also be #f), and the package path. As the client is transmitting a REQUESTS sequence, the server begins responding to it with RESPONSE structures, each with a sequence number indicating to which request it is responding (except in the case of input too garbled to extract a sequence number): RESPONSE ::= | ('error 'malformed-input string) "\n" | (SEQ-NO 'error 'malformed-request string) "\n" | (SEQ-NO 'bad-language string) "\n" | (SEQ-NO 'get 'ok Nat Nat Nat) "\n" BYTE-DATA | (SEQ-NO 'get 'error ERROR-CODE string) "\n" ERROR-CODE ::= 'not-found If the server receives a malformed request, it may close connection after sending a malformed-request response without processing any other requests. Otherwise it must process all requests even in the event of an error. On a successful get, the three numbers the server returns are the matched package's major version, the matched package's minor version, and the number of bytes in the package. 3 Client Download Policies Mzscheme invokes the PLaneT client once for each instance of a require-planet form in a program being run (i.e., the transitive closure of the "requires" relation starting from some specified root module; this closure is calculable statically). At each of these invocations, the client examines its internal cache to see if an appropriate module exists that matches the specification given by the user (for details see the next section). If one does, the client loads that module and returns. If none does, it initiates a transaction with the server using the PLaneT protocol described in the previous subsection and sends a single request consisting of the user's request. It installs the resulting .plt file and then loads the appropriate file. The client keeps a cache of downloaded packages locally. It does so in the $PLTCOLLECTS/planet/cache/ directory and subdirectories, in an intuitive manner: each item in the package's path in the PLaneT require line correspond to a subdirectory in the cache directory, starting with the owner name. (They should be unpacked relative to some user-specific rather than installation-specific place, possibly, but that's difficult to do so we won't do it yet). To check whether a package is installed when attempting to satisfy a requirement, the client checks its cache to see if an appropriate entry exists in its link-table for that require line. If one exists, it uses the named package directly. If none exists, it checks to see if there is an appropriate subdirectory. ||# (module resolver mzscheme (require (lib "match.ss") (lib "file.ss") (lib "port.ss") (lib "list.ss") (lib "date.ss") (lib "url.ss" "net") (lib "head.ss" "net") (lib "struct.ss") "config.ss" "private/planet-shared.ss" "private/linkage.ss") (provide (rename resolver planet-module-name-resolver) pkg-spec->full-pkg-spec get-package-from-cache get-package-from-server download-package install-pkg get-planet-module-path/pkg) (define install? (make-parameter #t)) ;; if #f, will not install packages and instead give an error (define (resolver spec module-path stx) ;; ensure these directories exist (make-directory* (PLANET-DIR)) (make-directory* (CACHE-DIR)) (establish-diamond-property-monitor) (cond [(or spec stx) (planet-resolve spec module-path stx)] [else module-path])) ; ========================================================================================== ; DIAMOND PROPERTY STUFF ; make sure a module isn't loaded twice with two different versions ; ========================================================================================== (define VER-CACHE-NAME #f) (define (establish-diamond-property-monitor) (unless VER-CACHE-NAME (set! VER-CACHE-NAME (gensym))) (unless (namespace-variable-value VER-CACHE-NAME #t (lambda () #f)) (namespace-set-variable-value! VER-CACHE-NAME (make-hash-table 'equal)))) (define (the-version-cache) (namespace-variable-value VER-CACHE-NAME)) (define (pkg->diamond-key pkg) (cons (pkg-name pkg) (pkg-route pkg))) (define (add-pkg-to-diamond-registry! pkg) (let ((orig (hash-table-get (the-version-cache) (pkg->diamond-key pkg) (lambda () #f)))) (cond [(not orig) (hash-table-put! (the-version-cache) (pkg->diamond-key pkg) pkg)] [(and (eq? (pkg-maj pkg) (pkg-maj orig)) (eq? (pkg-min pkg) (pkg-min orig))) (void)] [else (raise (make-exn:fail (string->immutable-string (format "Package ~a loaded twice with multiple versions: attempted to load version ~a.~a while version ~a.~a was already loaded" (pkg-name pkg) (pkg-maj pkg) (pkg-min pkg) (pkg-maj orig) (pkg-min orig))) (current-continuation-marks)))]))) ; ========================================================================================== ; MAIN LOGIC ; Handles the overall functioning of the resolver ; ========================================================================================== ; planet-resolve : PLANET-REQUEST symbol syntax[PLANET-REQUEST] -> symbol ; resolves the given request. Returns a name corresponding to the module in the correct ; environment (define (planet-resolve spec module-path stx) (let-values ([(path pkg) (get-planet-module-path/pkg spec module-path stx)]) (add-pkg-to-diamond-registry! pkg) (do-require path (pkg-path pkg) module-path stx))) ;; get-planet-module-path/pkg :PLANET-REQUEST symbol syntax[PLANET-REQUEST] -> path PKG ;; returns the matching package and the file path to the specific request (define (get-planet-module-path/pkg spec module-path stx) (match (cdr spec) [(file-name pkg-spec path ...) (match-let* ([pspec (pkg-spec->full-pkg-spec pkg-spec stx)] [pkg (or (get-linkage module-path pspec) (add-linkage! module-path pspec (or (get-package-from-cache pspec) (get-package-from-server pspec) (raise-syntax-error #f (format "Could not find package matching ~s" (list (pkg-spec-name pspec) (pkg-spec-maj pspec) (list (pkg-spec-minor-lo pspec) (pkg-spec-minor-hi pspec)) (pkg-spec-path pspec))) stx))))]) (values (apply build-path (pkg-path pkg) (append path (list file-name))) pkg))] [_ (raise-syntax-error 'require (format "Illegal PLaneT invocation: ~e" (cdr spec)) stx)])) ;; get-path : planet-request -> path ; pkg-spec->full-pkg-spec : PKG-SPEC syntax -> FULL-PKG-SPEC (define (pkg-spec->full-pkg-spec spec stx) (define (pkg name maj lo hi path) (make-pkg-spec name maj lo hi path stx)) (match spec [(? string?) (pkg spec #f #f #f '())] [((? string? path) ...) (pkg (last path) #f 0 #f (drop-last path))] [((? string? path) ... (? number? maj)) (pkg (last path) maj 0 #f (drop-last path))] [((? string? path) ... (? number? maj) min-spec) (let ((pkg (lambda (min max) (pkg (last path) maj min max (drop-last path))))) (match min-spec [(? number? min) (pkg min #f)] [((? number? lo) (? number? hi)) (pkg lo hi)] [('= (? number? min)) (pkg min min)] [('+ (? number? min)) (pkg min #f)] [('- (? number? min)) (pkg 0 min)]))] [_ (raise-syntax-error 'require (format "Invalid PLaneT package specifier: ~e" spec) stx)])) ; ========================================================================================== ; PHASE 2: CACHE SEARCH ; If there's no linkage, there might still be an appropriate cached module. ; ========================================================================================== ; get-package-from-cache : FULL-PKG-SPEC -> PKG | #f (define (get-package-from-cache pkg-spec) (lookup-package pkg-spec (CACHE-DIR))) ; ========================================================================================== ; PHASE 3: SERVER RETRIEVAL ; Ask the PLaneT server for an appropriate package if we don't have one locally. ; ========================================================================================== ; get-package-from-server : FULL-PKG-SPEC -> PKG | #f ; downloads and installs the given package from the PLaneT server and installs it in the cache, ; then returns a path to it (define (get-package-from-server pkg) (with-handlers ([exn:fail? (lambda (e) (raise (make-exn:fail (string->immutable-string (format "Error downloading module from PLaneT server: ~a" (exn-message e))) (exn-continuation-marks e))))]) (match (download-package pkg) [(#t path maj min) (install-pkg pkg path maj min)] [(#f str) #f]))) (define (download-package pkg) ((if (USE-HTTP-DOWNLOADS?) download-package/http download-package/planet) pkg)) (define (current-time) (let ((date (seconds->date (current-seconds)))) (parameterize ((date-display-format 'rfc2822)) (format "~a ~a:~a:~a" (date->string date) (date-hour date) (date-minute date) (date-second date))))) ; install-pkg : FULL-PKG-SPEC path[file] Nat Nat -> PKG ; install the given pkg to the planet cache and return a PKG representing the installed file (define (install-pkg pkg path maj min) (let* ((owner (car (pkg-spec-path pkg))) (extra-path (cdr (pkg-spec-path pkg))) (the-dir (apply build-path (CACHE-DIR) (append (pkg-spec-path pkg) (list (pkg-spec-name pkg) (number->string maj) (number->string min)))))) (if (directory-exists? the-dir) (raise (make-exn:fail "Internal PLaneT error: trying to install already-installed package" (current-continuation-marks))) (begin (with-logging (LOG-FILE) (lambda () (printf "\n============= Installing ~a on ~a =============\n" (pkg-spec-name pkg) (current-time)) ;; oh man is this a bad hack! (parameterize ((current-namespace (make-namespace))) ((dynamic-require '(lib "plt-single-installer.ss" "setup") 'install-planet-package) path the-dir (list owner (pkg-spec-name pkg) extra-path maj min))))) (make-pkg (pkg-spec-name pkg) (pkg-spec-path pkg) maj min the-dir))))) ; download-package : FULL-PKG-SPEC -> RESPONSE ; RESPONSE ::= (list #f string) | (list #t path[file] Nat Nat) ; downloads the given package and returns (list bool string): if bool is #t, ; the path is to a file that contains the package. If bool is #f, the package ; didn't exist and the string is the server's informative message. ; raises an exception if some protocol failure occurs in the download process (define (download-package/planet pkg) (define-values (ip op) (tcp-connect (PLANET-SERVER-NAME) (PLANET-SERVER-PORT))) (define (close-ports) (close-input-port ip) (close-output-port op)) (define (request-pkg-list pkgs) (for-each/n (lambda (pkg seqno) (write-line (list* seqno 'get (DEFAULT-PACKAGE-LANGUAGE) (pkg-spec-name pkg) (pkg-spec-maj pkg) (pkg-spec-minor-lo pkg) (pkg-spec-minor-hi pkg) (pkg-spec-path pkg)) op)) pkgs) (write-line 'end op) (flush-output op)) (define (state:initialize) (fprintf op "PLaneT/1.0\n") (flush-output op) (match (read ip) ['ok (state:send-pkg-request)] [('invalid (? string? msg)) (state:abort (string-append "protocol version error: " msg))] [bad-msg (state:abort (format "server protocol error (received invalid response): ~a" bad-msg))])) (define (state:send-pkg-request) (request-pkg-list (list pkg)) (state:receive-package)) (define (state:receive-package) (match (read ip) [(_ 'get 'ok (? nat? maj) (? nat? min) (? nat? bytes)) (let ((filename (make-temporary-file "planettmp~a.plt"))) (read-char ip) ; throw away newline that must be present (read-n-chars-to-file bytes ip filename) (list #t filename maj min))] [(_ 'error 'malformed-request (? string? msg)) (state:abort (format "Internal error (malformed request): ~a" msg))] [(_ 'get 'error 'not-found (? string? msg)) (state:failure (format "Server had no matching package: ~a" msg))] [(_ 'get 'error (? symbol? code) (? string? msg)) (state:abort (format "Unknown error ~a receiving package: ~a" code msg))] [bad-response (state:abort (format "Server returned malformed message: ~e" bad-response))])) (define (state:abort msg) (raise (make-exn:i/o:protocol msg (current-continuation-marks)))) (define (state:failure msg) (list #f msg)) (with-handlers ([void (lambda (e) (close-ports) (raise e))]) (begin0 (state:initialize) (close-ports)))) ;; ------------------------------------------------------------ ;; HTTP VERSION OF THE PROTOCOL ;; pkg->servlet-args : FULL-PKG-SPEC -> environment[from (lib "url.ss" "net")] ;; gets the appropriate query arguments to request the given package from the ;; PLaneT HTTP download servlet (define (pkg->servlet-args pkg) (let ((get (lambda (access) (format "~s" (access pkg))))) `((lang . ,(format "~s" (DEFAULT-PACKAGE-LANGUAGE))) (name . ,(get pkg-spec-name)) (maj . ,(get pkg-spec-maj)) (min-lo . ,(get pkg-spec-minor-lo)) (min-hi . ,(get pkg-spec-minor-hi)) (path . ,(get pkg-spec-path))))) ;; get-http-response-code : header[from (lib "head.ss" "net")] -> string ;; gets the HTTP response code in the given header (define (get-http-response-code header) (let ((parsed (regexp-match #rx"^HTTP/[^ ]* ([^ ]*)" header))) (and parsed (cadr parsed)))) ;; download-package/http : FULL-PKG-SPEC -> RESPONSE ;; a drop-in replacement for download-package that uses HTTP rather than the planet protocol. ;; The HTTP protocol does not allow any kind of complicated negotiation, but it appears that ;; many more users can make HTTP requests than requests from nonstandard protocols. (define (download-package/http pkg) (let* ((args (pkg->servlet-args pkg)) (target (copy-struct url (string->url (HTTP-DOWNLOAD-SERVLET-URL)) (url-query args))) (ip (get-impure-port target)) (head (purify-port ip)) (response-code/str (get-http-response-code head)) (response-code (string->number response-code/str))) (define (abort msg) (close-input-port ip) (raise (make-exn:i/o:protocol msg (current-continuation-marks)))) (case response-code [(#f) (abort (format "Server returned invalid HTTP response code ~s" response-code/str))] [(200) (let ((maj/str (extract-field "Package-Major-Version" head)) (min/str (extract-field "Package-Minor-Version" head))) (unless (and maj/str min/str (nat? (string->number maj/str)) (nat? (string->number min/str))) (printf "~a" head) (abort "Server did not include valid major and minor version information")) (let* ((filename (make-temporary-file "planettmp~a.plt")) (op (open-output-file filename 'truncate)) (maj (string->number maj/str)) (min (string->number min/str))) (copy-port ip op) (close-input-port ip) (close-output-port op) (list #t filename maj min)))] [(404) (list #f (format "Server had no matching package: ~a" (read-line ip)))] [(400) (abort (format "Internal error (malformed request): ~a" (read-line ip)))] [else (abort (format "Internal error (unknown HTTP response code ~a)" response-code))]))) ; ========================================================================================== ; MODULE MANAGEMENT ; Handles interaction with the module system ; ========================================================================================== ; do-require : path path symbol syntax -> symbol ; requires the given filename, which must be a module, in the given path. (define (do-require file-path package-path module-path stx) (parameterize ((current-load-relative-directory package-path)) ((current-module-name-resolver) file-path module-path stx))) ; ============================================================ ; UTILITY ; A few small utility functions (define (last l) (car (last-pair l))) ;; make-directory*/paths : path -> (listof path) ;; like make-directory*, but returns what directories it actually created (define (make-directory*/paths dir) (let ((dir (if (string? dir) (string->path dir) dir))) (let-values ([(base name dir?) (split-path dir)]) (cond [(directory-exists? dir) '()] [(directory-exists? base) (make-directory dir) (list dir)] [else (let ((dirs (make-directory*/paths base))) (make-directory dir) (cons dir dirs))])))) )