ANTLR How to use lexer rules having same starting?

antlr, grammar, lexer

Solution

As soon `DIGIT+ ':'` is matched, the lexer expects this to be followed by another `DIGIT` to match a `TIMECONSTANT`. If this does not happen, it cannot fall back on another lexer rule that matches `DIGIT+ ':'` and the lexer will not give up on the already matched `':'` to match an `INTEGER`.

A possible solution would be to optionally match `':' DIGIT+` at the end of the `INTEGER` rule and change the type of the token if this gets matched:

grammar T;  

parse
 : (t=. {System.out.printf("\%-15s '\%s'\n", tokenNames[$t.type], $t.text);})* EOF
 ;

INTEGER      : DIGIT+ ((':' DIGIT)=> ':' DIGIT+ {$type=TIMECONSTANT;})?;
COLON        : ':';
SPACE        : ' ' {skip();};

fragment DIGIT : '0'..'9';
fragment TIMECONSTANT : ;

When parsing the input:

11: 12:13 : 14

the following will be printed:

INTEGER         '11'
COLON           ':'
TIMECONSTANT    '12:13'
COLON           ':'
INTEGER         '14'

EDIT

Not too nice, but works...

True. However, this is not an ANTLR short coming: most lexer generators I know will have a problem properly tokenizing such a `TIMECONSTANT` (when `INTEGER` and `COLON` are also present). ANTLR at least facilitates a way to handle it in the lexer :)

You could also let this be handled by the parser instead of the lexer:

time_const : INTEGER COLON INTEGER;
INTEGER    : '0'..'9'+;
COLON      : ':';
SPACE      : ' ' {skip();};

However, if your language's lexer ignores white spaces, then input like:

12 :    34

would also be match by the `time_const` rule, of course.

Problem

How to use lexer rules having same starting? I am trying to use two similar lexer rules (having the same starting): ``` TIMECONSTANT: ('0'..'9')+ ':' ('0'..'9')+; INTEGER : ('0'..'9')+; COLON : ':'; ``` Here is my sample grammar: ``` grammar TestTime; text : (timeexpr | caseblock)*; timeexpr : TIME; caseblock : INT COLON ID; TIME : ('0'..'9')+ ':' ('0'..'9')+; INT : ('0'..'9')+; COLON : ':'; ID : ('a'..'z')+; WS : (' ' | '\t' | '\r' | '\n') {$channel=HIDDEN;}; ``` When i try to parse text: ``` 12:44 123 : abc 123: abc ``` First two lines are parsed correctly, 3rd - generates error. For some reason, '123:' ANTLR parses as TIME (while it is not)... So, is it possible to make grammar with such lexems? Having such rules is necessary in my language for using both case-blocks and datetime constants. For example in my language it is possible to write: ``` case MyInt of 1: a := 01.01.2012; 2: b := 12:44; 3: .... end; ```

Original source