5 Commits

Author	SHA1	Message	Date
Adam Sampson	1f490e9f7f	Lex multiline strings as multiple tokens. ... This is because the structure pass needs to be able to tell where new lines start, and if the line number changes in the middle of a token then it'll get confused.	2007-08-21 20:01:46 +00:00
Adam Sampson	3fb0cf088f	Implement continuation line rules.	2007-08-21 17:46:30 +00:00
Adam Sampson	1fc6dcd0e9	Add Outdents when reaching the end of a file. ... This doesn't make a difference for regular programs and headers -- but it does for .inc files containing processes.	2007-08-21 17:25:10 +00:00
Adam Sampson	223a14d767	Add a preprocessor that works with the new lexer. ... The parser now gets a stream of tokens, rather than needing to worry about loading files itself. This also reworks the lexer's idea of what constitutes a Token -- it's now a pair (Meta, TokenType), so it's always easy to pull out/rewrite the metadata -- and adds proper support for lexing preprocessor directives, rather than just treating them as reserved words.	2007-08-20 23:46:57 +00:00
Adam Sampson	a9f1e52103	Structure analyser for the new lexer.	2007-08-20 00:48:55 +00:00