We Keep Coding

ANTLR mismatched input 'foo(some_foo)' expecting {'foo'}

I'm writing a parser using ANTLR and am now at the stage of testing my parser/lexer.
I stumbled over a strange bug while trying to parse basically a variable assignment. (Like this)
Foo = mpsga(LT);
I get the error : mismatched input 'line 1:6 mismatched input 'mpsga(LT)' expecting 'mpsga'
This is especially strange for when I remove the brackets (or the argument LT),
the parser recognizes mpsga and it only misses the brackets (or the argument).
My Grammar looks something like this:
Lexer
lexer grammar FooLexer;
COMMENT
:
'#' ~[\r\n]* -> channel ( HIDDEN )
;
NEWLINE
:
(
'\r'? '\n'
| '\r'
)+ -> channel ( HIDDEN )
;
EQUALSSIGN
:
'='
;
SEMICOLON
:
';'
;
MPSGA_255_1
:
'LT'
;
MPSGA
:
'mpsga'
;
WHITESPACE
:
(
' '
| '\t'
)+ -> channel ( HIDDEN )
;
BRACKET_OPEN
:
'('
;
BRACKET_CLOSED
:
')'
;
VAR
:
[a-zA-Z][0-9a-zA-Z_]*
;
Parser
parser grammar FooParser;
options {
tokenVocab = FooLexer;
}
stmt_block
:
stmt_list EOF
;
stmt
:
VAR EQUALSSIGN expr SEMICOLON NEWLINE?
;
stmt_list
:
stmt
| stmt_list stmt
;
expr
:
extvar
;
extvar
:
MPSGA BRACKET_OPEN mpsga_field BRACKET_CLOSED
;
mpsga_field
:
MPSGA_255_1
;
When I try to parse this Foo = mpsga(LT); in Java i get the error.
Any help is appreciated!
Edit:
My Parse hierachy looks like the following:
Foo = mpsga(LT);
stmt_block
->stmt_list:1
-->stmt
--->"Foo"
--->"="
--->expr
---->extvar
----->"mpsga(LT)"
---->";"
-><EOF>
Foo = mpsga(LT;
stmt_block
->stmt_list:1
-->stmt
--->"Foo"
--->"="
--->expr
---->extvar
----->"mpsga"
----->"("
----->mpsga_field
------>"LT"
----->"<missing ')'>"
---->";"
-><EOF>
DISCLAIMER: I solved the problem. For anyone experiencing the same issue: I had some Lexer rules that were ambiguous for the mpsga part.

It's the argument: your grammar accepts 'foo' or 'foo2' as constants, not some_foo.

How to create dynamic ANTLR4 lexer rule

I have next grammar
grammar SearchEngine;
#lexer::members {
private java.util.Set<String> extraCriteria;
public SearchEngineLexer(CharStream input, java.util.Set<String> extraCriteria) {
this(input);
this.extraCriteria = extraCriteria;
}
}
query: expression EOF;
expression: criteria operator literal_value | expression 'AND' expression | expression 'OR' expression;
criteria : 'SERVICE_NAME' | ..a lot of hardcoded criterias here | EXTRA_CRITERIA;
EXTRA_CRITERIA: {extraCriteria.stream().filter(c -> c.equals(getText())).findFirst().isPresent()}? . ;
It accepts queries like SERVICE_NAME = 'something' OR EXCEPTION IS NULL and so on. The rest part of my grammar is not important, because it works without EXTRA_CRITERIA definition.
So, I created new lexer with "TestCriteria" as extra criteria and trying to parse my query:
Lexer lexer = new SearchEngineLexer(CharStreams.fromString("TestCriteria != 'test' OR SERVICE_NAME = 'EchoService'"), Collections.singleton("TestCriteria"));
TokenStream tokenStream = new CommonTokenStream(lexer);
SearchEngineParser parser = new SearchEngineParser(tokenStream);
parser.setErrorHandler(new BailErrorStrategy());
SearchEngineParser.QueryContext context = parser.query();
But when I execute this code I retrieve
line 1:0 token recognition error at: 'Te'
line 1:2 token recognition error at: 'st'
line 1:4 token recognition error at: 'C'
line 1:5 token recognition error at: 'ri'
line 1:7 token recognition error at: 'te'
line 1:9 token recognition error at: 'ri'
line 1:11 token recognition error at: 'a'
org.antlr.v4.runtime.misc.ParseCancellationException
at org.antlr.v4.runtime.BailErrorStrategy.recoverInline(BailErrorStrategy.java:66)
at de.telekom.tvpp.mtool.language.SearchEngineParser.criteria(SearchEngineParser.java:277)
at de.telekom.tvpp.mtool.language.SearchEngineParser.expression(SearchEngineParser.java:180)
at de.telekom.tvpp.mtool.language.SearchEngineParser.query(SearchEngineParser.java:117)
at de.telekom.tvpp.mtool.language.App.main(App.java:22)
Caused by: org.antlr.v4.runtime.InputMismatchException
at org.antlr.v4.runtime.BailErrorStrategy.recoverInline(BailErrorStrategy.java:61)
... 4 more
Where am I wrong? How to setup ANTL4 to use dynamic rule?

I had a mistake in the expression. This is my solution:
grammar SearchEngine;
#lexer::members {
private java.util.Set<String> extraCriteria;
public SearchEngineLexer(CharStream input, java.util.Set<String> extraCriteria) {
this(input);
this.extraCriteria = extraCriteria;
}
private boolean isExtraCriteria() {
return extraCriteria.stream().anyMatch(term -> ahead(term, _input));
}
private boolean ahead(final String word, final CharStream input) {
for (int i = 0; i < word.length(); i++) {
char wordChar = word.charAt(i);
int inputChar = input.LA(i + 1);
if (inputChar != wordChar) {
return false;
}
}
input.seek(input.index() + word.length() - 1);
return true;
}
}
query: expression EOF;
expression: criteria operator literal_value | expression 'AND' expression | expression 'OR' expression;
criteria : 'SERVICE_NAME' | ..a lot of hardcoded criterias here | EXTRA_CRITERIA;
EXTRA_CRITERIA: {isExtraCriteria()}? . ;

Error Separating Multiple Rules in ANTLR

I've almost finished my first adventure with ANTLR, and it's been quite a trip. Unfortunately, always only counts in horseshoes, hand grenades, and nuclear weapons, right?
Anyways, I'm trying to parse an input that looks like this:
; IF AGE IS LESS THAN 21, STILL RETURN TRUE FOR OVERSEAS LOCATION \r\n
SHOW "AGE REQUIREMENTS FAILED" FOR \r\n
IF AGE < 21 THEN \r\n
LOCATION = "OVERSEAS" \r\n
ENDIF \r\n
\r\n
; NEED SOMEONE WHO HAS WORKED FOR US FOR > 1 YEAR EXCEPT FOR CEO \r\n
SHOW "MINIMUM TIME REQUIREMENT NOT MET" FOR \r\n
IF STARTDATE > TODAY - 1 YEAR THEN \r\n
EMPLID=001 \r\n
ENDIF \r\n
Generally, if the test fails, the message is shown.
Anyways, a set can contain 1 or more SHOW rules. Processing of a single SHOW rule works, but it won't "split" when an inputstream contains > 1 SHOW rules.
Here are the relevant rules from the grammar:
showGroup returns [List<PolicyEvaluation> value]
#init {List<PolicyEvaluation> peList = new ArrayList<PolicyEvaluation>();}
: (expr1=show)* {peList.add($expr1.value);}
{
System.out.println("Entered policyGroup rule");
$value = peList;
}
;
// evaluate a single SHOW statement
show returns [PolicyEvaluation value]
: ('SHOW' expr1=STRING 'FOR')? expr2=ifStatement EOL*
{
System.out.println("Entered show rule");
Boolean expr2Value = (Boolean) $expr2.value;
PolicyEvaluation pe = new PolicyEvaluation();
if (expr1 == null) {
pe.setValue(expr2Value);
pe.setMessage(null);
} else {
if (expr2Value == false) {
pe.setValue(false);
pe.setMessage(expr1.getText());
} else {
pe.setValue(true);
pe.setMessage(null);
}
}
$value = pe;
}
;
// rules leading up to the show rule
// domain-specific grammar rules
STRING: '"' ID (' ' ID)* '"'
{
System.out.println("Entered STRING lexer rule");
// strip the quotes once we match this token
setText(getText().substring(1, getText().length()-1));
}
;
COMMENT: ';' (ID|' ')* EOL {$channel = HIDDEN;};
EOL: ('\r'|'\n'|'\r\n') {$channel = HIDDEN;};
SPACE: ' ' {$channel = HIDDEN;};
Maybe this is something simple. Any help is appreciated.
Jason

Try changing this: (expr1=show)* {peList.add($expr1.value);}
to this: (expr1=show {peList.add($expr1.value);})*
The action as it is will only fire after all show matches have completed, leaving you to operate on the last expr1.

How can I parse a special character differently in two terminal rules using antlr?

I have a grammar that uses the $ character at the start of many terminal rules, such as $video{, $audio{, $image{, $link{ and others that are like this.
However, I'd also like to match all the $ and { and } characters that don't match these rules too. For example, my grammar does not properly match $100 in the CHUNK rule, but adding the $ to the long list of acceptable characters in CHUNK causes the other production rules to break.
How can I change my grammar so that it's smart enough to distinguish normal $, { and } characters from my special production rules?
Basically what I'd to be able to do is say, "if the $ character doesn't have {, video, image, audio, link, etc. after it, then it should go to CHUNK".
grammar Text;
#header {
}
#lexer::members {
private boolean readLabel = false;
private boolean readUrl = false;
}
#members {
private int numberOfVideos = 0;
private int numberOfAudios = 0;
private StringBuilder builder = new StringBuilder();
public String getResult() {
return builder.toString();
}
}
text
: expression*
;
expression
: fillInTheBlank
{
builder.append($fillInTheBlank.value);
}
| image
{
builder.append($image.value);
}
| video
{
builder.append($video.value);
}
| audio
{
builder.append($audio.value);
}
| link
{
builder.append($link.value);
}
| everythingElse
{
builder.append($everythingElse.value);
}
;
fillInTheBlank returns [String value]
: BEGIN_INPUT LABEL END_COMMAND
{
$value = "<input type=\"text\" id=\"" +
$LABEL.text +
"\" name=\"" +
$LABEL.text +
"\" class=\"FillInTheBlankAnswer\" />";
}
;
image returns [String value]
: BEGIN_IMAGE URL END_COMMAND
{
$value = "<img src=\"" + $URL.text + "\" />";
}
;
video returns [String value]
: BEGIN_VIDEO URL END_COMMAND
{
numberOfVideos++;
StringBuilder b = new StringBuilder();
b.append("<div id=\"video1\">Loading the player ...</div>\r\n");
b.append("<script type=\"text/javascript\">\r\n");
b.append("\tjwplayer(\"video" + numberOfVideos + "\").setup({\r\n");
b.append("\t\tflashplayer: \"/trainingdividend/js/jwplayer/player.swf\", file: \"");
b.append($URL.text);
b.append("\"\r\n\t});\r\n");
b.append("</script>\r\n");
$value = b.toString();
}
;
audio returns [String value]
: BEGIN_AUDIO URL END_COMMAND
{
numberOfAudios++;
StringBuilder b = new StringBuilder();
b.append("<p id=\"audioplayer_");
b.append(numberOfAudios);
b.append("\">Alternative content</p>\r\n");
b.append("<script type=\"text/javascript\">\r\n");
b.append("\tAudioPlayer.embed(\"audioplayer_");
b.append(numberOfAudios);
b.append("\", {soundFile: \"");
b.append($URL.text);
b.append("\"});\r\n");
b.append("</script>\r\n");
$value = b.toString();
}
;
link returns [String value]
: BEGIN_LINK URL END_COMMAND
{
$value = "" + $URL.text + "";
}
;
everythingElse returns [String value]
: CHUNK
{
$value = $CHUNK.text;
}
;
BEGIN_INPUT
: '${'
{
readLabel = true;
}
;
BEGIN_IMAGE
: '$image{'
{
readUrl = true;
}
;
BEGIN_VIDEO
: '$video{'
{
readUrl = true;
}
;
BEGIN_AUDIO
: '$audio{'
{
readUrl = true;
}
;
BEGIN_LINK
: '$link{'
{
readUrl = true;
}
;
END_COMMAND
: { readLabel || readUrl }?=> '}'
{
readLabel = false;
readUrl = false;
}
;
URL
: { readUrl }?=> 'http://' ('a'..'z'|'A'..'Z'|'0'..'9'|'.'|'/'|'-'|'_'|'%'|'&'|'?'|':')+
;
LABEL
: { readLabel }?=> ('a'..'z'|'A'..'Z') ('a'..'z'|'A'..'Z'|'0'..'9')*
;
CHUNK
//: (~('${'|'$video{'|'$image{'|'$audio{'))+
: ('a'..'z'|'A'..'Z'|'0'..'9'|' '|'\t'|'\n'|'\r'|'-'|','|'.'|'?'|'\''|':'|'\"'|'>'|'<'|'/'|'_'|'='|';'|'('|')'|'&'|'!'|'#'|'%'|'*')+
;

You can't negate more than a single character. So, the following is invalid:
~('${')
But why not simply add '$', '{' and '}' to your CHUNK rule and remove the + at the end of the CHUNK rule (otherwise it would gobble up to much, possibly '$video{' further in the source, as you have noticed yourself already)?.
Now a CHUNK token will always consist of a single character, but you could create a production rule to fix this:
chunk
: CHUNK+
;
and use chunk in your production rules instead of CHUNK (or use CHUNK+, of course).
Input like "{ } $foo $video{" would be tokenized as follows:
CHUNK {
CHUNK
CHUNK }
CHUNK
CHUNK $
CHUNK f
CHUNK o
CHUNK o
CHUNK
BEGIN_VIDEO $video{
EDIT
And if you let your parser output an AST, you can easily merge all the text that one or more CHUNK's match into a single AST, whose inner token is of type CHUNK, like this:
grammar Text;
options {
output=AST;
}
...
chunk
: CHUNK+ -> {new CommonTree(new CommonToken(CHUNK, $text))}
;
...

An alternative solution which doesn't generate that many single-character tokens would be to allow chunks to contain a $ sign only as the first character. That way your input data will get split up at the dollar signs only.
You can achieve this by introducing a fragment lexer rule (i.e., a rule that does not define a token itself but can be used in other token regular expressions):
fragment CHUNKBODY
: 'a'..'z'|'A'..'Z'|'0'..'9'|' '|'\t'|'\n'|'\r'|'-'|','|'.'|'?'|'\''|':'|'\"'|'>'|'<'|'/'|'_'|'='|';'|'('|')'|'&'|'!'|'#'|'%'|'*';
The CHUNK rule then looks like:
CHUNK
: { !readLabel && !readUrl }?=> (CHUNKBODY|'$')CHUNKBODY*
;
This seems to work for me.

ANTLR : AST eval problems

Allo,
I would like to eval an AST that i generated.
I wrote a grammar generating an AST, and now I'm triying to write the grammar to evaluate this tree.
Here's my grammar :
tree grammar XHTML2CSVTree;
options {
tokenVocab=XHTML2CSV;
ASTLabelType=CommonTree;
}
#members {
// variables and methods to be included in the java file generated
}
/*------------------------------------------------------------------
* TREE RULES
*------------------------------------------------------------------*/
// example
tableau returns [String csv]
: ^(TABLEAU {String retour="";}(l=ligne{retour += $l.csv;})* {System.out.println(retour);})
;
ligne returns [String csv]
: ^(LIGNE {String ret="";}(c=cellule{ret += $c.csv;})+)
;
cellule returns [String csv]
: ^(CELLULE s=CHAINE){ $csv = $s.text;}
;
And here's the grammar building the AST :
grammar XHTML2CSV;
options {
output=AST;
ASTLabelType=CommonTree;
}
tokens {
CELLULE;
LIGNE;
TABLEAU;
CELLULEG = '<td>'; // simple lexemes
CELLULED = '</td>';
DEBUTCOL = '<tr>';
FINCOL = '</tr>';
DTAB = '<table';
FTAB = '>';
FINTAB = '</table>';
// anonymous tokens (usefull to give meaningfull name to AST labels)
// simple lexemes
}
#members {
// variables and methods to be included in the java file generated
}
/*------------------------------------------------------------------
* PARSER RULES
*------------------------------------------------------------------*/
tableau
: DTAB STAB* FTAB ligne* FINTAB -> ^(TABLEAU ligne*)
;
ligne
: DEBUTCOL cellule+ FINCOL -> ^(LIGNE cellule+)
;
cellule
: CELLULEG CHAINE CELLULED -> ^(CELLULE CHAINE)
;
/*------------------------------------------------------------------
* LEXER RULES
*------------------------------------------------------------------*/
STAB
: ' '.*'=\"'.*'\"'
;
WS
: ( '\t' | ' ' | '\r' | '\n'| '\u000C' )+ {$channel = HIDDEN;}
; // skip white spaces
CHAINE : (~('\"' | ',' | '\n' | '<' | '>'))+
;
// complex lexemes
XHTML2CSV.g works, i can see the AST generated in ANTLRworks,
but i cannot parse this AST to generated CSV code.
I get errors :
XHTML2CSVTree.java:144: ';' expected
match(input, Token.DOWN, null);
^
XHTML2CSVTree.java:144: not a statement
match(input, Token.DOWN, null);
^
XHTML2CSVTree.java:144: ';' expected
match(input, Token.DOWN, null);
^
XHTML2CSVTree.java:144: not a statement
match(input, Token.DOWN, null);
^
XHTML2CSVTree.java:144: ';' expected
match(input, Token.DOWN, null);
^
5 errors
If someone could help me,
Thanks.
eo
Edit :
My main class looks like :
import org.antlr.runtime.ANTLRFileStream;
import org.antlr.runtime.CommonTokenStream;
import org.antlr.runtime.tree.CommonTreeNodeStream;
import org.antlr.runtime.tree.Tree;
public class Main {
public static void main(String args[]) throws Exception {
try {
XHTML2CSVLexer lex = new XHTML2CSVLexer(new ANTLRFileStream(args[0])); // create lexer to read the file specified from command line (i.e., first argument, e.g., java Main test1.xhtml)
CommonTokenStream tokens = new CommonTokenStream(lex); // transform it into a token stream
XHTML2CSVParser parser = new XHTML2CSVParser(tokens); // create the parser that reads from the token stream
Tree t = (Tree) parser.cellule().tree; // (try to) parse a given rule specified in the parser file, e.g., my_main_rule
CommonTreeNodeStream nodes = new CommonTreeNodeStream(t); // transform it into a common data structure readable by the tree pattern
nodes.setTokenStream(tokens); // declare which token to use (i.e., labels of the nodes defined in the parser, mainly anonymous tokens)
XHTML2CSVTree tparser = new XHTML2CSVTree(nodes); // instantiate the tree pattern
System.out.println(tparser.cellule()); // apply patterns
} catch (Exception e) {
e.printStackTrace();
}
}
}

The ligne rule in you tree grammar:
ligne returns [String csv]
: ^(LIGNE {Sting ret="";r}(c=cellule{ret += $c.csv;})+)
; // ^ ^
// | |
// problem 1, problem 2
has 2 problems:
it contains Sting where it should be String;
there's a trailing r that is messing up your custom Java code.
It should be:
ligne returns [String csv]
: ^(LIGNE {String ret="";}(c=cellule{ret += $c.csv;})+)
;
EDIT
If I generate a lexer and parser (1), generate a tree walker (2), compile all .java source files (3) and run the Main class (4):
java -cp antlr-3.3.jar org.antlr.Tool XHTML2CSV.g
java -cp antlr-3.3.jar org.antlr.Tool XHTML2CSVTree.g
javac -cp antlr-3.3.jar *.java
java -cp .:antlr-3.3.jar Main test.txt
the following gets printed to the console:
table data
where the file test.txt contains:
<td>table data</td>
So I don't see any problem. Perhaps you're trying to parse a <table>? This would go wrong since both your parser and tree-walker are invoking the cellule rule, not the tableau rule.