Commit Graph

4 Commits

Author SHA1 Message Date
Robby Findler
11994bd4f8 fix a bug in the module lexer; it was returning the wrong length for the tokens
it creates when the #lang line isn't well-formed (eg "#lang racke").

closes PR 12399
2011-11-28 21:16:31 -06:00
Robby Findler
65fd0234ad add a unicode test to scheme-lexer.rkt
and change the tests so they all run with port line
counting enabled (or else the unicode test fails)

adjust module-lexer.rkt tests so they can run in either
port-counting mode or not (but currently run them all in
port-counting mode because scheme-lexer doesn't work without it)

also make a first stab at what needs to change in the module
lexer to make it work in non port line-counting mode
2011-11-03 22:41:27 -05:00
Robby Findler
d00aed6f1b fix the module reader for the case when there are non-ASCII unicode characters
in comments before the #lang line

also add an #:init-position argument to peeking-input-port
2011-11-03 17:13:17 -05:00
Robby Findler
1eaf53d4cb adjust the module lexer so that it treats the entire range that
'read-language' uses as a single token in the case that read-language
fails. This helps it to deal with things like s-exp and at-exp
properly

closes PR 12260
2011-10-08 08:08:36 -05:00