racket/collects/teachpack/error-composition.scrbl

#lang scribble/doc
@(require scribble/manual scribble/decode (for-label racket/base))

@title[#:tag "error-guidelines"]{
  Error Message Composition Guidelines for Student Languages}

This section lists some guidelines for writing good error messages for
novices, as informed by @seclink["research"]{our research}.  Please
follow these guidelines when you write code that is intended for
beginners, including libraries and teachpacks.  It ensures that error
messages from your code fits messages from the student languages and
from other teachpacks.

@(define (samp . text) @splice[@list{@emph{``@splice[text]''}}])
@(define (word . text) @splice[@list{‘@splice[text]’}])

@section{General Guidelines}

@itemize[
  @item{Frustrated students will peer at the error message for clues on
    how to proceed.  Avoid offering hints, and avoid proposing any
    specific modification. Students will follow well-meaning-but-wrong
    advice uncritically, if only because they have no reason to doubt
    the authoritative voice of the tool.}

  @item{Be concise and clear.  Students give up reading error messages
    if the text is too long, uses obscure words, or employs difficult
    grammar.}]

@section{Message Structure and Form}

@itemize[
  @item{Start the message with the name of the construct whose
    constraint is being violated, followed by a colon.}

  @item{State the constraint that was violated (@samp{expected a...}),
    then contrast with what was found.  For example, @samp{this function
    expects two arguments, but found only one}.  If needed, explain how
    what was found fails to satisfy the constraint.  Write somewhat
    anthropomorphically with an objective voice that is neither friendly
    nor antagonistic.}

  @item{If an expression contains multiple errors, report the leftmost
    error first.  E.g., the error in @racket{(define 1 2 3)} is
    @samp{expected the variable name, but found a number}, not
    @samp{expected 2 parts after define, but found 3}.  Before raising
    an error about a sub-part of a macro, call
    @racket[syntax-local-expand-expression] on sub-expressions to its
    left, so that such errors are shown first.}

  @item{State the number of parts instead of saying @samp{found too many
    parts}.  Write the code necessary to make plurals agree.}]

@section{Vocabulary}

@subsection{Permitted Words}

Use only the following vocabulary words to describe code:

@nested{@word{function}, @word{variable}, @word{argument},
  @word{function body}, @word{expression}, @word{part}, @word{clause},
  @word{top level}, @word{structure name}, @word{type name}, @word{field
  name}, @word{binding}.}

@itemize[
  @item{Use @word{binding} for the square-braced pair in a @racket{let}
    and similar binding forms.}

  @item{Use @word{argument} for actual arguments and @word{variable} for
    formal arguments in the body of a definition.}

  @item{Use @word{part} when speaking about an s-expression that is not
    an expression, either because it is malformed, because it occurs in
    a non-expression position, or because it is a valid piece of syntax
    for a macro invocation.  A well-formed and well-placed call to a
    function, primitive, or macro is not a @word{part}, it is an
    @word{expression}.}]

@subsection{Prohibited Words}

These guidelines use few terms intentionally, emphasizing commonality
among concepts rather than technical precision (which most students do
not appreciate anyway).

@tabular[
@list[
  @list[@para{@bold{Instead of}}
        @para{@bold{Use}}]
  @list[@para{procedure, primitive name, primitive operator, predicate,
          selector, constructor}
        @para{@samp{function}''}]
  @list[@para{s-expression}
        @para{@samp{expression}}]
  @list[@para{identifier}
        @para{@samp{argument} or @samp{variable}, depending on the use
          in the program}]
  @list[@para{defined name}
        @para{@samp{function} or @samp{variable}}]
  @list[@para{sequence}
        @para{@samp{at least one (in parentheses)}}]
  @list[@para{function header}
        @para{@samp{after define}, @samp{after the name},
          @samp{after the first argument}, ...}]
  @list[@para{keyword}
        @para{mention the construct directly by name, such as
          @samp{expected a variable but found a cond}}]
  @list[@para{built-in}  @para{Nothing --- avoid this term}]
  @list[@para{macro} @para{Nothing --- avoid this term}]]]

@subsection{General Vocabulary Guidelines}

@itemize[
  @item{Avoid modifiers that are not necessary to disambiguate. Write
    @word{variable} instead of @word{local variable}, @word{defined
    variable}, or @word{input variable}.  Write @word{clause} instead of
    @word{question-answer clause}.  If they appear necessary for
    disambiguation, try to find some other way to achieve this (and drop
    the modifier).}

  @item{When introducing macros with sub-parts, reuse existing
    vocabulary words, such as @word{clause} or @word{binding} (if
    appropriate), or just @word{part}, instead of defining new terms.}

  @item{Use @word{name} only when describing the syntax of a definition
    form.  For example, the define form in BSL should say @samp{expected
    at least one variable after the function name}.  Outside of the
    definition form, simply use the word @word{function} rather than
    distinguish between (1) a function, (2) the variable that binds the
    function, and (3) the name of that variable.

    [Rationale: Students learn this distinction when they learn about
    lambda.  The first is the lambda implicit in the definition, the
    second is the variable introduced by the definition that can appear
    as the first argument to @racket{set!}, the third is the particular
    sequence of letters.  But BSL should avoid this complexity, and
    ASL’s error messages should maintain consistency with BSL.]}

  @item{Avoid introducing technical vocabulary, even if well-known to a
    mathematician.}]

@section{Punctuation}

@itemize[
  @item{Do not use any punctuation beyond those of the normal English
    language.  Do not write @litchar{<>} around type names, and do not
    write quotes around keywords.}]

@section{Runtime Behavior}

@itemize[
  @item{When specifying a function's behavior, say @samp{the function
    takes ... and returns ...}}

  @item{When describing a contract violation, say @samp{the function
    expects ... but received ...}}

  @item{As much as possible, identify expressions by the value they
    evaluate to, e.g. @samp{the value of @racket{(f x)} is 5}.  If it is
    necessary to explicate evaluation times, the context discusses
    mutable state or order of evaluation, then say that the expressions
    @samp{evaluate to} a value.  Function calls are a special case of
    expression.  Prefer @samp{the function call returns}, instead of
    @samp{it evaluates to}, except when trying to draw attention to the
    evaluation of the arguments.}]

@section[#:tag "research"]{Supporting Research}

These guidelines arose from a collections of research studies held at
the Worcester Polytechnic Institute, Brown University, and Northeastern
University.  Further experiment details and results are described in:
@itemize[
  @item{@hyperlink["http://www.cs.brown.edu/~sk/Publications/Papers/Published/mfk-mind-lang-novice-inter-error-msg/"]{
          Mind Your Language: On Novices' Interactions with Error
          Messages}

    This paper reports on a series of studies that explore beginning
    students' interactions with the vocabulary and source-expression
    highlighting in DrRacket.  Our findings demonstrate that the error
    message DrRacket's old error messages significantly failed to convey
    information accurately to students.}

  @item{@hyperlink["http://www.cs.brown.edu/~sk/Publications/Papers/Published/mfk-measur-effect-error-msg-novice-sigcse/"]{
          Measuring the Effectiveness of Error Messages Designed for
          Novice Programmers}

    This paper presents a fine-grained grading rubric for evaluating the
    performance of individual error messages.  We applied the rubric to
    a course worth of student work, which allowed us to characterize
    some ways error messages fail.}]