Use code-inspectors to prevent optimization in unsafe contexts. (#531)

The optimizer should only run when the code being compiled could
directly access racket/unsafe/ops. This prevents unsoundness in Typed
Racket from giving untrusted code access to dangerous operations.

typed-racket-doc/typed-racket/scribblings/guide/optimization.scrbl

@@ -10,9 +10,7 @@ benchmarking, see @secref[#:doc '(lib
 "scribblings/guide/guide.scrbl")]{performance}.}
 
 Typed Racket provides a type-driven optimizer that rewrites well-typed
-programs to potentially make them faster. It should in no way make
-your programs slower or unsafe.
-
+programs to potentially make them faster.
 
 @section{Turning the optimizer off}
 
@@ -23,6 +21,11 @@ program:
 
 @racketmod[typed/racket #:no-optimize]
 
+The optimizer is also disabled when executing a typed racket program in
+a sandbox (see @secref["Sandboxed_Evaluation" #:doc
+'(lib "scribblings/reference/reference.scrbl")]) and when the environment
+variable @envvar{PLT_TR_NO_OPTIMIZE} is set (to any value).
+
 @section{Getting the most out of the optimizer}
 Typed Racket's optimizer can improve the performance of various common
 Racket idioms. However, it does a better job on some idioms than on

typed-racket-doc/typed-racket/scribblings/reference/optimization.scrbl

@@ -12,8 +12,7 @@ in the guide for tips to get the most out of the optimizer.
 }
 
 Typed Racket provides a type-driven optimizer that rewrites well-typed
-programs to potentially make them faster. It should in no way make
-your programs slower or unsafe.
+programs to potentially make them faster.
 
 Typed Racket's optimizer is turned on by default. If you want to
 deactivate it (for debugging, for instance), you must add the
@@ -21,3 +20,10 @@ deactivate it (for debugging, for instance), you must add the
 program:
 
 @racketmod[typed/racket #:no-optimize]
+
+The optimizer is also disabled if the environment variable @envvar{PLT_TR_NO_OPTIMIZE} is set (to any value) or if the current code inspector (see
+@secref["modprotect" #:doc '(lib "scribblings/reference/reference.scrbl")])
+is insufficiently powerful to access @racketmodname[racket/unsafe/ops],
+for example when executing in a sandbox (see @secref["Sandboxed_Evaluation"
+#:doc '(lib "scribblings/reference/reference.scrbl")]). This prevents untrusted
+code from accessing these operations by exploiting errors in the type system.

typed-racket-lib/typed-racket/tc-setup.rkt

@@ -17,9 +17,21 @@
          tc-module/full
          tc-toplevel/full)
 
+(define (authorized-code-inspector?)
+  ;; Check whether the current code inspector has sufficient
+  ;; privileges to access unsafe operations.
+  (with-handlers ([exn:fail? (lambda (e) #f)])
+    (module->namespace 'racket/unsafe/ops)
+    #t))
+
 (define (maybe-optimize body)
   ;; do we optimize?
-  (if (and (optimize?) (not (getenv "PLT_TR_NO_OPTIMIZE")))
+  ;; only if the current module requested optimization,
+  ;; PLT_TR_NO_OPTIMIZE is not set, and the
+  ;; current code inspector has sufficient privileges
+  (if (and (optimize?)
+           (not (getenv "PLT_TR_NO_OPTIMIZE"))
+           (authorized-code-inspector?))
       (begin
         (do-time "Starting optimizer")
         (begin0 (stx-map optimize-top body)

typed-racket-test/fail/sandbox.rkt

@@ -0,0 +1,40 @@
+#;
+(exn-pred #rx"expected: pair\\?\n *given: #<int-wrapper>")
+#lang racket/base
+
+;; This test checks that TR optimizations are disabled within a
+;; sandboxed context
+
+(require racket/sandbox)
+
+(environment-variables-set!	(current-environment-variables)
+                            (string->bytes/locale "PLT_TR_NO_OPTIMIZE")
+                            #f)
+
+(define (make-sandbox-code-inspector)
+  (make-inspector (current-code-inspector)))
+
+(parameterize ([sandbox-memory-limit #f]
+               [sandbox-eval-limits #f]
+               [sandbox-make-code-inspector make-sandbox-code-inspector])
+  (let ([eval (make-evaluator 'racket/base
+                              '(module untyped racket/base
+                                 (provide extract-field
+                                          (struct-out int-wrapper))
+
+                                 (require syntax/location)
+                                 (require syntax/modresolve)
+
+                                 (struct int-wrapper (value))
+
+                                 (module typed typed/racket
+                                   (provide extract-integer)
+                                   (define (extract-integer [p : (Pair Integer Integer)])
+                                     (cdr p)))
+
+                                 (require 'typed)
+
+                                 (define extract-field
+                                   (eval 'extract-integer (module->namespace (quote-module-path typed)))))
+                              '(require (submod "." untyped)))])
+    (eval '(extract-field (int-wrapper 42)))))