raco pkg update: make --lookup and --clone work right together

The combination of `--lookup`, `--clone`, and `--catalog` can provide
a one-step path from a snapshot's built implementation of a package to
a repository-linked clone, for example.  In that situation, however,
`raco pkg` will have limited ability to detect that packages
originally drawn from the same repository are consistenly installed as
clones (and we can revisit if it turns out to be an issue).

pkgs/racket-pkgs/racket-doc/pkg/scribblings/git-workflow.scrbl

@@ -61,14 +61,29 @@ develops only a few of them. The intended workflow is as follows:
  @item{Install all the relevant packages with @command-ref{install}.}
 
  @item{For each package to be developed out of a particular Git
-  repository named by @nonterm{git-pkg-source}, update the installation with
+  repository named by @nonterm{pkg-name}, update the installation with
 
-  @commandline{@command{update} --clone @nonterm{dir} @nonterm{git-pkg-source}}
+  @commandline{@command{update} --clone @nonterm{dir} @nonterm{pkg-name}}
 
   which discards the original installation of the package and replaces
-  it with a local clone as @nonterm{dir}. (As a convenience, when
+  it with a local clone as @nonterm{dir}.
-  @nonterm{git-pkg-source} and the last element of @nonterm{dir} are
+
-  the same, then @nonterm{git-pkg-source} can be omitted.)}
+  As a convenience, when @nonterm{git-pkg-source} and the last element
+  of @nonterm{dir} are the same, then @nonterm{pkg-name} can be
+  omitted. Put another way, the argument to @DFlag{clone} can be a
+  path to @nonterm{pkg-name}:
+
+  @commandline{@command{update} --clone @nonterm{path-to}/@nonterm{pkg-name}}}
+
+ @item{If a package's current installation is not drawn fro a Git
+  repository (e.g., it's drawn from a catalog of built packages for a
+  distribution or snapshot), but @nonterm{catalog} maps the package
+  name to the right Git repository, then combine @DFlag{clone} with
+  @DFlag{lookup} and @DFlag{catalog}:
+
+  @commandline{@command{update} --lookup --catalog @nonterm{catalog} --clone @nonterm{path-to}/@nonterm{pkg-name}}
+
+  A suitable @nonterm{catalog} might be @url{http://pkgs.racket-lang.org}.}
 
  @item{Manage changes to each of the developed packages in the usual
   way with @exec{git} tools, but @command-ref{update} is also available
@@ -82,7 +97,7 @@ affects the branch used for the initial checkout, while a non-empty
 path causes a subdirectory of the checkout to be linked for the
 package.
 
-The @exec{git} tools and @exec{raco pkg} tools interact in specific
+The @exec{git} and @exec{raco pkg} tools interact in specific
 ways:
 
 @itemlist[
@@ -110,10 +125,9 @@ ways:
        does not inherently require clone sharing among the packages,
        but since non-sharing or inconsistent installation modes could
        be confusing, @command-ref{install} and @command-ref{update}
-       reject non-sharing or inconsistent installations unless
+       report non-sharing or inconsistent installations. In typical cases,
-       overridden with @DFlag{multi-clone}. In typical cases,
+       the default @exec{@DFlag{multi-clone} ask} mode can automatically
-       @exec{@DFlag{multi-clone} ask} or @exec{@DFlag{multi-clone}
+       fix inconsistencies.}
-       convert} can automatically fix inconsistencies.}
 
  @item{When pulling changes to repositories that have local copies,
        @command-ref{update} pulls changes with the equivalent of @exec{git

pkgs/racket-pkgs/racket-doc/pkg/scribblings/pkg.scrbl

@@ -578,7 +578,7 @@ argument.
  @item{@DFlag{lookup} --- Causes a @tech{package name} as a @nonterm{pkg-source} to be used
        as a replacement, instead of the name of a installed package that may have updates.
        (If the named package was installed through a package name, then there's effectively
-        no difference.)}
+        no difference unless a different catalog is used.)}
 
  @item{@DFlag{type} @nonterm{type} or @Flag{t} @nonterm{type} --- Same as for @command-ref{install}.}
  @item{@DFlag{name} @nonterm{pkg} or @Flag{n} @nonterm{pkg} --- Same as for @command-ref{install}.}
@@ -593,7 +593,7 @@ argument.
  @item{@DFlag{clone} @nonterm{dir} --- Same as for
     @command-ref{install}, except that a @nonterm{pkg-source} as a
     @tech{package name} is treated as the name of an installed
-    package. In that case, the package must be currently installed
+    package (unless @DFlag{lookup} is specified). In that case, the package must be currently installed
     from a Git or GitHub source---possibly as directed by a
     catalog---and that source is used for the clone (which replaces
     the existing package installation).}

pkgs/racket-pkgs/racket-test/tests/pkg/tests-clone.rkt

@@ -217,6 +217,48 @@
        (delete-directory/files (build-path clone-dir "a"))
        (delete-directory/files a-dir)))
 
+    ;; ----------------------------------------
+    ;; Combining --clone and --lookup
+
+    (with-fake-root
+      (shelly-begin
+       (make-directory a-dir)
+       $ (~a "cd " a-dir "; git init")
+       (set-file (build-path a-dir "main.rkt") "#lang racket/base 1")
+       (~a "cd " a-dir "; git add .; git commit -m change; git update-server-info")
+       $ (commit-changes-cmd)
+       
+       (define (update-a-in-catalog!)
+         (hash-set! *index-ht-1* "a"
+                    (hasheq 'checksum
+                            (current-commit a-dir)
+                            'source
+                            "http://localhost:9998/a/.git")))
+       (update-a-in-catalog!)
+       $ "raco pkg config --set catalogs http://localhost:9990"      
+       
+       $ (~a "raco pkg install " a-dir)
+       $ "racket -l a" =stdout> "1\n"
+
+       (set-file (build-path a-dir "main.rkt") "#lang racket/base 2")
+       ;; didn't commit, yet
+       $ "racket -l a" =stdout> "2\n"
+
+       (shelly-case
+        "convert directory-linked to clone via --lookup"
+        $ (~a "raco pkg update --clone " (build-path clone-dir "a"))
+        =exit> 1
+        $ (~a "raco pkg update --lookup --clone " (build-path clone-dir "a"))
+        =exit> 0
+        $ "racket -l a" =stdout> "1\n"
+        
+        $ (commit-changes-cmd)
+        $ "raco pkg update a"
+        $ "racket -l a" =stdout> "2\n")
+       
+       (delete-directory/files (build-path clone-dir "a"))
+       (delete-directory/files a-dir)))
+
     ;; ----------------------------------------
     ;; Detecting when packages should share a clone
 

racket/collects/pkg/lib.rkt

@@ -105,6 +105,7 @@
                         #:strip (or/c #f 'source 'binary 'binary-lib)
                         #:force-strip? boolean?
                         #:link-dirs? boolean?
+                        #:lookup-for-clone? boolean?
                         #:multi-clone-behavior (or/c 'fail 'force 'convert 'ask))
         (or/c #f 'skip (listof (or/c path-string? (non-empty-listof path-string?)))))]
   [pkg-remove

racket/collects/pkg/main.rkt

@@ -300,6 +300,8 @@
                 scope scope-dir installation user pkg-source a-type #t name
                 (lambda ()
                   install-copy-checks ...
+                  (define clone-path (and (eq? a-type 'clone)
+                                          (path->complete-path clone)))
                   (define setup-collects
                     (with-pkg-lock
                         (parameterize ([current-pkg-catalogs (and catalog
@@ -307,15 +309,15 @@
                           (pkg-update (for/list ([pkg-source (in-list pkg-source)])
                                         (cond
                                          [lookup
-                                          (pkg-desc pkg-source a-type name checksum #f)]
+                                          (pkg-desc pkg-source a-type name checksum #f
+                                                    #:path clone-path)]
                                          [else
                                           (define-values (pkg-name pkg-type) 
                                             (package-source->name+type pkg-source a-type))
                                           (if (eq? pkg-type 'name)
                                               pkg-name
                                               (pkg-desc pkg-source a-type name checksum #f
-                                                        #:path (and (eq? a-type 'clone)
+                                                        #:path clone-path))]))
-                                                                    (path->complete-path clone))))]))
                                       #:from-command-line? #t
                                       #:all? all
                                       #:dep-behavior (or (and auto 'search-auto)
@@ -334,6 +336,7 @@
                                                   (and binary 'binary)
                                                   (and binary-lib 'binary-lib))
                                       #:force-strip? force
+                                      #:lookup-for-clone? lookup
                                       #:multi-clone-behavior (or multi-clone
                                                                  (if batch
                                                                      'fail

racket/collects/pkg/private/install.rkt

@@ -1080,6 +1080,7 @@
                     #:strip [strip-mode #f]
                     #:force-strip? [force-strip? #f]
                     #:link-dirs? [link-dirs? #f]
+                    #:lookup-for-clone? [lookup-for-clone? #f]
                     #:multi-clone-behavior [clone-behavior 'fail])
   (define download-printf (if quiet? void printf))
   (define metadata-ns (make-metadata-namespace))
@@ -1103,9 +1104,11 @@
                                                     #:use-cache? use-cache?
                                                     #:from-command-line? from-command-line?
                                                     #:link-dirs? link-dirs?)
-                                (map (convert-clone-name-to-clone-repo/update
+                                (map (if lookup-for-clone?
-                                      db
+                                         (convert-clone-name-to-clone-repo/install catalog-lookup-cache
-                                      from-command-line?)
+                                                                                   download-printf)
+                                         (convert-clone-name-to-clone-repo/update db
+                                                                                  from-command-line?))
                                      pkgs)))
   (cond
     [(empty? pkgs)