I've coded a simple lexer and parser using ANTLR4 grammars to make a language plugin for NetBeans 7.3 to help team write more quickly our layout files (a mix of XHTML and widgets definitions also in form of XHTML tags but with custom properties, characteristics, and with some differencies against XHTML syntax).
Template file example:
<div style="dyn_layout_panel">
#symbol#
<w_label=label, text="Try to close this window" />
<w_buttonclose=button, text = "CLOSE", on_press=press_close />
<w_buttonterminate=button, text="TERMINATE", on_press=press_terminate />
<w_mydatepicker=datepicker, parent=tab0, ary=[10, "str", /regex/i], start_date=2013-10-05, on_selected=datepicker_selected />
<w_myeditbox=editbox, parent=tab0, validation=USER_REGEX, validation_regex=/^[0-9]+[a-z]*$/i,
validation_msg="User regex don't match editbox contents.", on_keyreturn=tab0_editbox_keyreturn />
<div style="dyn_layout_panel">
$SYMBOL_2$
Some text that make a text node.
</div>
</div>
I use AnltrWorks 2 to write and debug lexer and parser and all seem to be fine, in NetBeans also I don't get any exception and the parser work properly but during editing/typing I lose token colors near the cursor.
Screenshot of problem:
Adding a debug console output for each keystroke I see that the lexer enter in IN_TAG or IN_WIDGET mode correctly, but after a WHITESPACE it returns to the default mode and match te rest of text inside a tag as a TEXT_NODE token.
I know that a lexer can have only one active mode at a time, so because it matches the TEXT_NODE rule when in IN_TAG or IN_WIDGET modes?
Lexer grammar file:
lexer grammar LayoutLexer;
COMMENT
: '/*' .*? '*/' -> channel(HIDDEN)
;
WS : ( ' '
| '\t'
| EOL
)+? -> channel(HIDDEN)
;
WDG_START_OPEN : '<w_' PROPERTY -> pushMode(IN_WIDGET) ;
WDG_END_OPEN : '</w_' PROPERTY -> pushMode(IN_WIDGET) ;
TAG_START_OPEN : '<' ATTRIBUTE -> pushMode(IN_TAG) ;
TAG_END_OPEN : '</' ATTRIBUTE -> pushMode(IN_TAG) ;
EXT_REF
: ( ('#' REF_NAME '#') | ('$' SYMBOL '$') | ('§' REF_NAME '§') )
;
fragment
REF_NAME
: ( [a-z]+ [0-9a-z_]*? )
;
fragment
EOL : ( '\r\n' | '\n\r' | '\n' )
;
EQUAL
: '='
;
TEXT_NODE
: ( (~('\r'|'\n'|'<'|'#'|'$'|'§'))+ )
;
ERROR
: ( .+? )
;
mode IN_TAG;
TAG_CLOSE : '>' -> popMode ;
TAG_EMPTY_CLOSE : '/>' -> popMode ;
TAG_WS : WS -> type(WS), channel(HIDDEN) ;
TAG_COMMENT : COMMENT -> type(COMMENT), channel(HIDDEN) ;
TAG_EQ : EQUAL -> type(EQUAL) ;
ATTRIBUTE
: ( LITERAL [0-9a-zA-Z_]* )
;
VAL
: ( '"' ( ESC_SEQ | ~('\\'|'"') )*? '"'
| '\'' ( ESC_SEQ | ~('\\'|'\'') )*? '\'' )
;
TAG_ERR : ERROR -> type(ERROR) ;
mode IN_WIDGET;
WDG_CLOSE : '>' -> popMode ;
WDG_EMPTY_CLOSE : '/>' -> popMode ;
WDG_WS : WS -> type(WS), mode(IN_WIDGET), channel(HIDDEN) ;
WDG_COMMENT : COMMENT -> type(COMMENT), channel(HIDDEN) ;
WDG_EQ : EQUAL -> type(EQUAL), pushMode(WDG_ASSIGN) ;
COMMA
: ','
;
fragment
MINUS
: '-'
;
STRING
: ( '"' ( ESC_SEQ | ~('\\'|'"') )*? '"'
| '\'' ( ESC_SEQ | ~('\\'|'\'') )*? '\'' )
;
fragment
ESC_SEQ
: '\\' ('b'|'t'|'n'|'f'|'r'|'\"'|'\''|'\\')
| UNICODE_ESC
| OCTAL_ESC
;
fragment
OCTAL_ESC
: '\\' ('0'..'3') ('0'..'7') ('0'..'7')
| '\\' ('0'..'7') ('0'..'7')
| '\\' ('0'..'7')
;
fragment
UNICODE_ESC
: '\\' 'u' HEX_DIGIT HEX_DIGIT HEX_DIGIT HEX_DIGIT
;
fragment
HEX_DIGIT
: [0-9a-fA-F]
;
fragment
DIGIT
: [0-9]
;
fragment
HEX_NUMBER
: '0x' HEX_DIGIT+
;
fragment
HTML_NUMBER
: (INT_NUMBER | FLOAT_NUMBER) HTML_UNITS
;
fragment
FLOAT_NUMBER
: MINUS? INT_NUMBER '.' DIGIT+
;
fragment
INT_NUMBER
: MINUS? DIGIT+
;
EVENT_HANDLER
: 'on_' PROPERTY
;
PROPERTY
: ( LITERAL [0-9a-zA-Z_]* )
;
fragment
LITERAL
: ( LITERAL_U | LITERAL_L )
;
fragment
LITERAL_U
: [A-Z]+
;
fragment
LITERAL_L
: [a-z]+
;
WDG_ERR : ERROR -> type(ERROR) ;
mode WDG_ASSIGN;
PHP_REF
: ( LITERAL_L ('_' | LITERAL_L | [0-9])* ) -> popMode
;
VALUE : (WDG_VAL | ARRAY) -> popMode;
ASGN_WS : WS -> type(WS), channel(HIDDEN);
ASGN_COMMA : COMMA -> type(COMMA);
ARY_START
: '['
;
ARY_END
: ']'
;
BIT_OR
: '|'
;
ARRAY
: ARY_START ARY_VALUE (ASGN_COMMA ARY_VALUE)* ARY_END
;
fragment
ARY_VALUE : ASGN_WS? WDG_VAL ASGN_WS? -> type(VALUE);
fragment
WDG_VAL
: (STRING
| UTC_DATE
| HEX_NUMBER
| HTML_NUMBER
| FLOAT_NUMBER
| INT_NUMBER
| BOOLEAN
| BITFIELD
| REGEX
| CSS_CLASS)
;
fragment
HTML_UNITS
: ('%'|'in'|'cm'|'mm'|'em'|'ex'|'pt'|'pc'|'px')
;
fragment
BOOLEAN
: ('true'|'false')
;
fragment
BITFIELD
: SYMBOL (WS? BIT_OR WS? SYMBOL)*
;
SYMBOL
: LITERAL_U [0-9A-Z_]*
;
UTC_DATE
: (DIGIT DIGIT DIGIT DIGIT '-' DIGIT DIGIT '-' DIGIT DIGIT)
;
REGEX
: ('/' ('\\'.|.)*? '/' ('g'|'m'|'i')* )
;
CSS_CLASS
: ( LITERAL_L ('-' | '_' | LITERAL_L | [0-9])* )
;
WDG_ASSIGN_ERR : ERROR -> type(ERROR), popMode;
Parser grammar file:
parser grammar LayoutParser;
options
{
tokenVocab=LayoutLexer;
language=Java;
}
document : (element | TEXT_NODE | EXT_REF)* EOF;
element
locals
[
String currentTag
]
: ( ( html_open_tag (element | TEXT_NODE | EXT_REF)* html_close_tag )
| ( wdg_open_tag (element | TEXT_NODE | EXT_REF)* wdg_close_tag )
| ( html_empty_tag | wdg_empty_tag ) )
;
html_empty_tag
: TAG_START_OPEN (ATTRIBUTE EQUAL VAL)* TAG_EMPTY_CLOSE
;
html_open_tag
: ( tag=TAG_START_OPEN (ATTRIBUTE EQUAL VAL)* TAG_CLOSE )
{$element::currentTag = $tag.text.substring(1);}
;
html_close_tag
: tag=TAG_END_OPEN TAG_CLOSE
{
if (!$element::currentTag.equals($tag.text.substring(2)))
notifyErrorListeners("HTML tag mismatch '" + $element::currentTag + "' - '" + $tag.text.substring(2) + "'");
}
;
wdg_empty_tag
: WDG_START_OPEN EQUAL PHP_REF ( COMMA (wdg_prop | wdg_event) )* WDG_EMPTY_CLOSE
;
wdg_open_tag
: tag=WDG_START_OPEN EQUAL PHP_REF ( COMMA (wdg_prop | wdg_event) )* WDG_CLOSE
{$element::currentTag = $tag.text.substring(1);}
;
wdg_close_tag
: tag=WDG_END_OPEN WDG_CLOSE
{
if (!$element::currentTag.equals($tag.text.substring(2)))
notifyErrorListeners("Widget alias mismatch '" + $element::currentTag + "' - '" + $tag.text + "'");
}
;
wdg_prop
: PROPERTY (EQUAL (ARRAY | VALUE | PHP_REF | UTC_DATE | REGEX | CSS_CLASS))?
;
wdg_event
: EVENT_HANDLER EQUAL PHP_REF
;
Depending on the implementation of syntax highlighting, the IDE may or may not start at the beginning of the document when lexing the input for syntax highlighting. If it does not start at the beginning of the document, then before returning any tokens, you need to ensure that the lexer instance is initialized in the correct mode (both the _mode and _modeStack fields need to be initialized to their correct state at the point where lexing starts).
If your lexer reads or writes any custom fields during lexing, you may need to restore those fields as well.
Examples
GoWorks (NetBeans based, LGPL License). This implementation does not use the lexer facilities in the NetBeans API, but instead implements the functionality at a lower level. For now you can ignore the MarkOccurrences* and SemanticHighlighter classes.
package org.tvl.goworks.editor.go.highlighter
package org.antlr.works.editor.antlr4.highlighting
ANTLR 4 IntelliJ Plugin (IntelliJ IDEA, BSD license).
package org.antlr.intellij.adaptor.lexer
package org.antlr.intellij.plugin (in particular, the SyntaxHighlighter classes)
Additional efficiency notes
Your REF_NAME, VAL, and STRING rules use non-greedy loops that do not need to be non-greedy. In each of these rules, change +? to + and change *? to *.
Your WS and ERROR rules use a non-greedy operator +? which is equivalent to not having a closure at all. The unnecessary use of a non-greedy operator in these cases only serves to slow down your lexer. To preserve the existing behavior, you can remove +? from these rules (replacing with + would change behavior).
Additional functionality notes
ANTLR 4 does not perform any error correction during lexing. If the input does not match a token, then the input simply does not match a token. This issue affects your VAL and STRING tokens in particular, which will not get syntax highlighting prior to adding the closing " or ' character. For syntax highlighting these types of tokens, I prefer to use an additional mode in the lexer, allowing me to produce separate tokens for the escape sequences embedded in the string, as well as syntax highlighting an unterminated string at the end of the line (unless your language allows strings to span multiple lines, in which case you'd stop at the end of the input).
For future references
All problems are related to the wrong implementation I done of NetBeans Lexer<T> class; many tutorials on the web do not take into account that a lexer may have more than one mode and that the lexer state must be backuped and restored between Lexer allocation/releases as mentioned by 280Z28.
This is the code I use to make syntax highlighting consistent:
public class LayoutEditorLexer implements Lexer<LayoutTokenId> {
private LexerRestartInfo<LayoutTokenId> info;
private LayoutLexer lexer;
private class LexerState {
public int Mode = -1;
public IntegerStack Stack = null;
public LexerState(int mode, IntegerStack stack)
{
Mode = mode;
Stack = new IntegerStack(stack);
}
}
public LayoutEditorLexer(LexerRestartInfo<LayoutTokenId> info) {
this.info = info;
AntlrCharStream charStream = new AntlrCharStream(info.input(), "LayoutEditor", false);
lexer = new LayoutLexer(charStream);
lexer.removeErrorListeners();
lexer.addErrorListener(ErrorListener.INSTANCE);
LexerState lexerMode = (LexerState)info.state();
if (lexerMode != null)
{
lexer._mode = lexerMode.Mode;
lexer._modeStack.addAll(lexerMode.Stack);
}
}
#Override
public org.netbeans.api.lexer.Token<LayoutTokenId> nextToken() {
Token token = lexer.nextToken();
int ttype = token.getType();
if (ttype != LayoutLexer.EOF)
{
LayoutTokenId tokenId = LayoutLanguageHierarchy.getToken(ttype);
return info.tokenFactory().createToken(tokenId);
}
return null;
}
#Override
public Object state()
{
// Here many tutorials simply returns null.
return new LexerState(lexer._mode, lexer._modeStack);
}
#Override
public void release()
{
}
}
Related
Having the subsequent simple grammar, I'd like to simultaneously parse strings and numbers:
grammar Simple;
aRule : 'fs' '(' value["textual"] ')' ;
bRule : 'fi' '(' value["numeral"] ')' ;
cRule : 'f' '(' (value["textual"] | value["numeral"]) ')' ;
value[String k]
: {$k.equals("any") || $k.equals("textual")}? string
| {$k.equals("any") || $k.equals("numeral")}? numeric
;
string
: STRING_LITERAL
;
numeric
: ('+' | '-')? INTEGER_LITERAL
;
STRING_LITERAL
: '\'' (~('\'' | '\r' | '\n') | '\'' '\'' | NEWLINE)* '\''
;
INTEGER_LITERAL
: '0' | [1-9] [0-9]*
;
SPACES
: [ \t\r\n]+ -> skip
;
fragment NEWLINE : '\r'? '\n';
Now, I'd like to parse the following expressions:
fs('asdf') // works
fi(512) // works
f('asdf') // works
f(512) // fails
If I switch textual and numeral in cRule, then f('asdf') fails and f(512) works.
Any ideas?
UPDATE1
grammar Simple;
rules : aRule | bRule | cRule ;
aRule : 'fs' '(' value["textual"] ')' ;
bRule : 'fi' '(' value["numeral"] ')' ;
cRule : 'f' '(' (tRule | nRule) ')' ;
tRule : value["textual"] ;
nRule : value["numeral"] ;
value[String k]
: {$k.equals("any") || $k.equals("textual")}? string
| {$k.equals("any") || $k.equals("numeral")}? numeric
;
string : STRING_LITERAL ;
numeric : ('+' | '-')? INTEGER_LITERAL ;
STRING_LITERAL : '\'' (~('\'' | '\r' | '\n') | '\'' '\'' | NEWLINE)* '\'' ;
INTEGER_LITERAL : '0' | [1-9] [0-9]* ;
SPACES : [ \t\r\n]+ -> skip ;
fragment NEWLINE : '\r'? '\n';
Even w/ this updated grammar --- as suggest by #GRosenberg --- for f(512) I still get no viable alternative at input '512'. Again fs('asdf'), fi(512) and f('asdf') work.
I agree with Lucas, this is a totally overcomplicated grammar. If you want to accept only a string then, by all means, specify only string in your grammar. Why using a value rule with all different options and limiting it then to a single one? That's a typical shot in the foot. Instead do like this:
rules : aRule | bRule | cRule ;
aRule : 'fs' '(' string ')' ;
bRule : 'fi' '(' numberic ')' ;
cRule : 'f' '(' (tRule | nRule) ')' ;
tRule : string ;
nRule : numeric ;
It's also much easier to read if you spell out what you want your language to look out, instead of trying to parameterize some generic rule.
The relevant generated code is listed below. In it, notice that a predicate constraint failure results in an exception. This explains why the second alt is always skipped -- either the first alt is correct or the whole rule fails.
Whether the generated code is somehow 'improper' is a question probably best directed to the ANTLR4 Google group or Github issues page.
In any case, the solution is to separate using subrules:
cRule : F LPAREN ( dRule | eRule ) RPAREN EOF ;
dRule : value["textual"] ;
eRule : value["numeral"] ;
UPDATE
The value rule is the one that needs to be split into subrules:
cRule : F LPAREN ( valuet["textual"] | valuen["numeral"] ) RPAREN EOF ;
valuet[String k]
: {$k.equals("any") || $k.equals("textual")}? string
;
valuen[String k]
: {$k.equals("any") || $k.equals("numeral")}? numeric
;
Tested/works.
public final ValueContext value(String k) throws RecognitionException {
ValueContext _localctx = new ValueContext(_ctx, getState(), k);
enterRule(_localctx, 2, RULE_value);
try {
setState(21);
_errHandler.sync(this);
switch (getInterpreter().adaptivePredict(_input, 1, _ctx)) {
case 1:
enterOuterAlt(_localctx, 1); {
setState(17);
if (!(_localctx.k.equals("any") || _localctx.k.equals("textual")))
throw new FailedPredicateException(this, "$k.equals(\"any\") || $k.equals(\"textual\")");
setState(18);
string();
}
break;
case 2:
enterOuterAlt(_localctx, 2); {
setState(19);
if (!(_localctx.k.equals("any") || _localctx.k.equals("numeral")))
throw new FailedPredicateException(this, "$k.equals(\"any\") || $k.equals(\"numeral\")");
setState(20);
numeric();
}
break;
}
} catch (RecognitionException re) {
_localctx.exception = re;
_errHandler.reportError(this, re);
_errHandler.recover(this, re);
} finally {
exitRule();
}
return _localctx;
}
I'm trying to use antlr4 to parse a ssh command result, but I can not figure out why this code doesn't work, I keep getting an "extraneous input" error.
Here is a sample of the file I'm trying to parse :
system
home[1] HOME-NEW
sp
cpu[1]
cpu[2]
home[2] SECOND-HOME
sp
cpu[1]
cpu[2]
Here is my grammar file :
listAll
: ( system | home | NL)*
;
elements
: (sp | cpu )*
;
home
: 'home[' number ']' value NL elements
;
system
: 'system' NL
;
sp
: 'sp' NL
;
cpu
: 'cpu[' number ']' NL
;
value
: VALUE
;
number
: INT
;
VALUE : STRING+;
STRING: ('a'..'z'|'A'..'Z'| '-' | ' ' | '(' | ')' | '/' | '.' | '[' | ']');
INT : ('0'..'9')+ ;
NL : '\r'? '\n';
WS : (' '|'\t')* {skip();} ;
The entry point is 'listAll'.
Here is the result I get :
(listAll \r\n (system system \r\n) home[1] HOME-NEW \r\n sp \r\n cpu[1] \r\n cpu[2] \r\n[...])
The parsing failed after 'system'. And I get this error :
line 2:1 extraneous input 'home[1] HOME-NEW' expecting {, system', NL, WS}
Does anybody know why this is not working ?
I am a beginner with Antlr, and I'm not sure I really understand how it works !
Thank you all !
You need to combine NL and WS as one WS element and skip it using -> skip (not {skip()})
And since the WS will be skipped automatically, no need to specify it in all the rules.
Also, your STRING had a space (' ') which was causing the error and taking up the next input.
Here is your complete grammar :
listAll : ( system | home )* ;
elements : ( sp | cpu )* ;
home : 'home[' number ']' value elements;
system : 'system' ;
sp : 'sp' ;
cpu : 'cpu[' number ']' ;
value : VALUE ;
number : INT ;
VALUE : STRING+;
STRING : ('a'..'z'|'A'..'Z'| '-' | '(' | ')' | '/' | '.' | '[' | ']') ;
INT : [0-9]+ ;
WS : [ \t\r\n]+ -> skip ;
Also, I'll suggest you to go through the ANTLR4 Documentation
I want to write a lexer that has multiple modes. But the modes are mostly similar. The only difference is that I reference the same character by a different name. I have it working, the problem is I have to copy the entire lexer change all the names, add types, and add one line for each mode.
Here is the general problem I want to solve. I want a comma to have high precedence outside a '[' ']'. I want it to have a low precedence inside the '[', ']'. To do that I push and pop modes with the '[' and ']'. But since the only thing I am changing is the precedence I have to copy all the rules into each mode and given them different names.
One additional thing, once inside a '[' there would be no way to get back to the high precedence comma. So when the grammar moves into a '{', '}' section the comma reverts back to high precedence.
In order to accomplish this I have the initial default mode plus CONJUNCTION (high precedence) and NON_CONJUNCTION (low precedence). I copy all the rules from the default mode and rename them to C_ or NC_. Then I assign their type back to the type of the default mode.
I would rather accomplish this without coping, renaming, and assigning types to all the rules from the default mode.
Here is my lexer:
lexer grammar DabarLexer;
OPEN_PAREN : '(' -> pushMode(NON_CONJUNCTION) ;
CLOSE_PAREN : ')' -> popMode;
OPEN_BRACKET : '[' -> pushMode(NON_CONJUNCTION) ;
CLOSE_BRACKET : ']' -> popMode ;
OPEN_CURLY : '{' -> pushMode(CONJUNCTION) ;
CLOSE_CURLY : '}' -> popMode ;
SPACE : ' ' ;
HEAVY_COMMA : ',';
ENDLINE : '\n' ;
PERIOD : '.' ;
SINGLE_QUOTE : '\'' ;
DOUBLE_QUOTE : '"' ;
INDENTION : '\t' -> skip;
fragment SYMBOL : HEAVY_COMMA | OPEN_BRACKET | CLOSE_BRACKET | OPEN_PAREN | CLOSE_PAREN | OPEN_CURLY | CLOSE_CURLY | SPACE | ENDLINE | PERIOD | SINGLE_QUOTE | DOUBLE_QUOTE | INDENTION ;
ESCAPE : '\\' SYMBOL ;
fragment NON_SYMBOL : ~[(){}',; \n.\t"\[\]] ;
IDENTIFIER : (NON_SYMBOL | ESCAPE)+ ;
LITERAL : (SINGLE_QUOTE (NON_SYMBOL | ESCAPE)+ SINGLE_QUOTE) | DOUBLE_QUOTE (NON_SYMBOL | ESCAPE)+ DOUBLE_QUOTE ;
mode CONJUNCTION ;
C_HEAVY_COMMA : ',' -> type(HEAVY_COMMA);
C_OPEN_PAREN : '(' -> type(OPEN_PAREN), pushMode(NON_CONJUNCTION) ;
C_CLOSE_PAREN : ')' -> type(CLOSE_PAREN), popMode;
C_OPEN_BRACKET : '[' -> type(OPEN_BRACKET), pushMode(NON_CONJUNCTION) ;
C_CLOSE_BRACKET : ']' -> type(CLOSE_BRACKET), popMode ;
C_OPEN_CURLY : '{' -> type(OPEN_CURLY), pushMode(CONJUNCTION) ;
C_CLOSE_CURLY : '}' -> type(CLOSE_CURLY), popMode ;
C_SPACE : ' ' -> type(SPACE);
C_ENDLINE : '\n' -> type(ENDLINE);
C_PERIOD : '.' -> type(PERIOD);
C_SINGLE_QUOTE : '\'' -> type(SINGLE_QUOTE);
C_DOUBLE_QUOTE : '"' -> type(DOUBLE_QUOTE);
C_INDENTION : '\t' -> type(INDENTION),skip;
fragment C_SYMBOL : ( HEAVY_COMMA | C_OPEN_BRACKET | C_CLOSE_BRACKET | C_OPEN_PAREN | C_CLOSE_PAREN | C_OPEN_CURLY | C_CLOSE_CURLY | C_SPACE | C_ENDLINE | C_PERIOD | C_SINGLE_QUOTE | C_DOUBLE_QUOTE | C_INDENTION ) ;
C_ESCAPE : '\\' C_SYMBOL -> type(ESCAPE);
fragment C_NON_SYMBOL : ~[(){}',; \n.\t"\[\]] ;
C_IDENTIFIER : (C_NON_SYMBOL | C_ESCAPE)+ -> type(IDENTIFIER);
C_LITERAL : ((C_SINGLE_QUOTE (C_NON_SYMBOL | C_ESCAPE)+ C_SINGLE_QUOTE) | C_DOUBLE_QUOTE (C_NON_SYMBOL | C_ESCAPE)+ C_DOUBLE_QUOTE) -> type(LITERAL);
mode NON_CONJUNCTION ;
LIGHT_COMMA : ',' ;
NC_OPEN_PAREN : '(' -> type(OPEN_PAREN), pushMode(NON_CONJUNCTION) ;
NC_CLOSE_PAREN : ')' -> type(CLOSE_PAREN), popMode;
NC_OPEN_BRACKET : '[' -> type(OPEN_BRACKET), pushMode(NON_CONJUNCTION) ;
NC_CLOSE_BRACKET : ']' -> type(CLOSE_BRACKET), popMode ;
NC_OPEN_CURLY : '{' -> type(OPEN_CURLY), pushMode(CONJUNCTION) ;
NC_CLOSE_CURLY : '}' -> type(CLOSE_CURLY), popMode ;
NC_SPACE : ' ' -> type(SPACE);
NC_ENDLINE : '\n' -> type(ENDLINE);
NC_PERIOD : '.' -> type(PERIOD);
NC_SINGLE_QUOTE : '\'' -> type(SINGLE_QUOTE);
NC_DOUBLE_QUOTE : '"' -> type(DOUBLE_QUOTE);
NC_INDENTION : '\t' -> type(INDENTION),skip;
fragment NC_SYMBOL : ( LIGHT_COMMA | NC_OPEN_BRACKET | NC_CLOSE_BRACKET | NC_OPEN_PAREN | NC_CLOSE_PAREN | NC_OPEN_CURLY | NC_CLOSE_CURLY | NC_SPACE | NC_ENDLINE | NC_PERIOD | NC_SINGLE_QUOTE | NC_DOUBLE_QUOTE | NC_INDENTION ) ;
NC_ESCAPE : '\\' NC_SYMBOL -> type(ESCAPE);
fragment NC_NON_SYMBOL : ~[(){}',; \n.\t"\[\]] ;
NC_IDENTIFIER : (NC_NON_SYMBOL | NC_ESCAPE)+ -> type(IDENTIFIER);
NC_LITERAL : ((NC_SINGLE_QUOTE (NC_NON_SYMBOL | NC_ESCAPE)+ NC_SINGLE_QUOTE) | NC_DOUBLE_QUOTE (NC_NON_SYMBOL | NC_ESCAPE)+ NC_DOUBLE_QUOTE) -> type(LITERAL);
Your current solution is very similar to the solution I use. For example, take a look at the TemplateComment mode of the grammar I use for for StringTemplate 4 support in ANTLRWorks 2. One helpful thing I implemented in ANTLR 4 a while back is it won't create duplicate token types for a rule in this form.
// No TemplateComment_NEWLINE token type is created here, because the
// type(NEWLINE) action means this rule produces tokens of a specific type.
TemplateComment_NEWLINE : NEWLINE -> type(NEWLINE), channel(HIDDEN);
I know that this has been discussed thousand times but I still cannot figure out why is following grammar failing. In interpreter everything works fine, without any errors or warnings. However when running the generated code, I'm getting mismatched input as shown below.
For this grammar:
grammar xxx;
options {
language = Java;
output = AST;
}
#members {
#Override
public String getErrorMessage(RecognitionException e,
String[] tokenNames)
{
List stack = getRuleInvocationStack(e, this.getClass().getName());
String msg = null;
if ( e instanceof NoViableAltException ) {
NoViableAltException nvae = (NoViableAltException)e;
msg = " no viable alt; token="+e.token+
" (decision="+nvae.decisionNumber+
" state "+nvae.stateNumber+")"+
" decision=<<"+nvae.grammarDecisionDescription+">>";
}
else {
msg = super.getErrorMessage(e, tokenNames);
}
return stack+" "+msg;
}
#Override
public String getTokenErrorDisplay(Token t) {
return t.toString();
}
}
obj
: first=subscription
(COMMA other=subscription)*
;
subscription
: ID
(EQUALS arguments_in_brackets)?
filters
;
arguments_in_brackets
: LOPAREN arguments ROPAREN
;
arguments
: (arguments_body)
;
arguments_body
: argument (arguments_more)?
;
arguments_more
: SEMICOLON arguments_body
;
argument
: id_equals argument_body
;
argument_body
: STRING
| INT
| FLOAT
;
filters
: LSPAREN expression RSPAREN
;
expression
: or
;
or
: first=and
(OR^ second=and)*
;
and : first=atom
(AND^ second=atom)*
;
atom
: filter
| atom_expression
;
atom_expression
: LCPAREN
expression
RCPAREN
;
filter
: id_equals arguments_in_brackets
;
id_equals
: WS* ID WS* EQUALS WS*
;
COMMA: WS* ',' WS*;
LCPAREN : WS* '(' WS*;
RCPAREN : WS* ')' WS*;
LSPAREN : WS* '[' WS*;
RSPAREN : WS* ']' WS*;
LOPAREN : WS* '{' WS*;
ROPAREN : WS* '}' WS*;
AND: WS* 'AND' WS*;
OR: WS* 'OR' WS*;
NOT: WS* 'NOT' WS*;
EQUALS: WS* '=' WS*;
SEMICOLON: WS* ';' WS*;
ID : ('a'..'z'|'A'..'Z'|'_') ('a'..'z'|'A'..'Z'|'0'..'9'|'_')*
;
INT : '0'..'9'+
;
FLOAT
: ('0'..'9')+ '.' ('0'..'9')* EXPONENT?
| '.' ('0'..'9')+ EXPONENT?
| ('0'..'9')+ EXPONENT
;
// : '"' ( ESC_SEQ | ~('\\'|'"') )* '"'
// : '"' (~'"')* '"'
STRING
: '"' (~'"')* '"'
;
fragment
EXPONENT : ('e'|'E') ('+'|'-')? ('0'..'9')+ ;
fragment
HEX_DIGIT : ('0'..'9'|'a'..'f'|'A'..'F') ;
fragment
ESC_SEQ
: '\\' ('b'|'t'|'n'|'f'|'r'|'\"'|'\''|'\\')
| UNICODE_ESC
| OCTAL_ESC
;
fragment
OCTAL_ESC
: '\\' ('0'..'3') ('0'..'7') ('0'..'7')
| '\\' ('0'..'7') ('0'..'7')
| '\\' ('0'..'7')
;
fragment
UNICODE_ESC
: '\\' 'u' HEX_DIGIT HEX_DIGIT HEX_DIGIT HEX_DIGIT
;
NEWLINE: '\r'? '\n' {skip();} ;
WS: (' '|'\t')+ {skip();} ;
And for this input:
status={name="Waiting";val=5}[ownerEmail1={email="dsa#fdsf.ds"} OR internalStatus={status="New"}],comments={type="fds"}[(internalStatus={status="Owned"} AND ownerEmail2={email="dsa#fds.ds"}) OR (role={type="Contributor"} AND status={status="Closed"})]
I'm getting:
line 1:67 [obj, subscription, filters, expression, or, and, atom, filter, arguments_in_brackets] mismatched input [#18,67:80='internalStatus',<11>,1:67] expecting ROPAREN
line 1:157 [obj, subscription, filters, expression, or, and, atom, atom_expression, expression, or, and, atom, filter, arguments_in_brackets] mismatched input [#42,157:167='ownerEmail2',<11>,1:157] expecting ROPAREN
Can someone give me any clues why is this failing please? I've tried to rewrite it in many ways but the error is still the same.
The problem is that you're using WS tokens in other lexer rules and are therefor skipping these tokens. This causes the lexer to discard these tokens entirely, and can then not be used in parser rules.
So, if you have a rule like:
WS : ' ' {skip();};
and then use this rule in NOT:
NOT : WS* 'NOT' WS*;
it causes the NOT token to be skipped as well.
If you're already skipping these WS chars, you don't need to include them in your other lexer rules: simply remove all WS* in other rules:
...
NOT : 'NOT';
...
(also remove them from parser rules: all skipped tokens from the lexer are never available in parser rules anyway!)
I am currently creating a more or less simple expression evaluator using ANTLR.
My grammar is straightforward (at least i hope so) and looks like this:
grammar SXLGrammar;
options {
language = Java;
output = AST;
}
tokens {
OR = 'OR';
AND = 'AND';
NOT = 'NOT';
GT = '>'; //greater then
GE = '>='; //greater then or equal
LT = '<'; //lower then
LE = '<='; //lower then or equal
EQ = '=';
NEQ = '!='; //Not equal
PLUS = '+';
MINUS = '-';
MULTIPLY = '*';
DIVISION = '/';
CALL;
}
#header {
package somepackage;
}
#members {
}
#lexer::header {
package rise.spics.sxl;
}
rule
: ('='|':')! expression
;
expression
: booleanOrExpression
;
booleanOrExpression
:
booleanAndExpression ('OR'^ booleanAndExpression)*
;
booleanAndExpression
:
booleanNotExpression ('AND'^ booleanNotExpression)*
;
booleanNotExpression
:
('NOT'^)? booleanAtom
;
booleanAtom
:
| compareExpression
;
compareExpression
:
commonExpression (('<' | '>' | '=' | '<=' | '>=' | '!=' )^ commonExpression)?
;
commonExpression
:
multExpr
(
(
'+'^
| '-'^
)
multExpr
)*
| DATE
;
multExpr
:
atom (('*'|'/')^ atom)*
| '-'^ atom
;
atom
:
INTEGER
| DECIMAL
| BOOLEAN
| ID
| '(' expression ')' -> expression
| functionCall
;
functionCall
:
ID '(' arguments ')' -> ^(CALL ID arguments?)
;
arguments
:
(expression) (','! expression)*
| WS
;
BOOLEAN
:
'true'
| 'false'
;
ID
:
(
'a'..'z'
| 'A'..'Z'
)+
;
INTEGER
:
('0'..'9')+
;
DECIMAL
:
('0'..'9')+ ('.' ('0'..'9')*)?
;
DATE
:
'!' '0'..'9' '0'..'9' '0'..'9' '0'..'9' '-' '0'..'9' '0'..'9' '-' '0'..'9' '0'..'9' (' ' '0'..'9' '0'..'9' ':''0'..'9' '0'..'9' (':''0'..'9' '0'..'9')?)?
;
WS
: (' '|'\t' | '\n' | '\r' | '\f')+ { $channel = HIDDEN; };
Now if i try to parse an invalid Expression like "= true NOT true", the graphical test-tool of the eclipse plugin throws an NoViableAltException: line 1:6 no viable alternative at input 'NOT', which is correct and supposed.
Now if i try to parse the expression in a Java Program, nothing happens. The Program
String expression = "=true NOT false";
CharStream input = new ANTLRStringStream(expression);
SXLGrammarLexer lexer = new SXLGrammarLexer(input);
TokenStream tokenStream = new CommonTokenStream(lexer);
SXLGrammarParser parser = new SXLGrammarParser(tokenStream);
CommonTree tree = (CommonTree) parser.rule().getTree();
System.out.println(tree.toStringTree());
System.out.println(parser.getNumberOfSyntaxErrors());
would output:
true
0
that means, the AST created by the parser exists of one node and ignores the rest. I'd like to handle syntax errors in my application, but its not possible if the generated parser doesn't find any error.
I also tried to alter the parser by overwriting the displayRecognitionError() method with something like this:
public void displayRecognitionError(String[] tokenNames,
RecognitionException e) {
String msg = getErrorMessage(e, tokenNames);
throw new RuntimeException("Error at position "+e.index+" " + msg);
}
but displayRecognitionError gets never called.
If i try something like "=1+", a error gets displayed. I guess theres something wrong with my grammar, but why does the eclipse plugin throw that error while the generated parser does not?
If you want rule to consume the entire token-stream, you have to specify where you expect the end of your input. Like this:
rule
: ('='|':')! expression EOF
;
Without the EOF your parser reads the true as boolean an ignores the rest.