I'm trying to figure out just how exactly to use ANTLR, but I'm having a really difficult time digesting the things I've found. So far, here are my resources:
How to create AST with ANTLR4?
How can I import an ANTLR lexer grammar into another grammar using Gradle 2.10?
https://github.com/antlr/grammars-v4/tree/master/java8
https://dzone.com/articles/parsing-any-language-in-java-in-5-minutes-using-an
https://raw.githubusercontent.com/antlr/antlr4/master/doc/getting-started.md
A little bit of background
I'm experimenting with moving from JavaParse to ANTLR because I want to handle ASTs of languages besides Java. My understanding of ANTLR and the predefined grammars (linked above) is that this is feasible.
Setup
IntelliJ 15 CE
Gradle
Java 1.8
This ANTLR resource
I created a very simple and standard gradle project in IntelliJ and I'm still encountering issues:
The problem
I'm missing the Java8Lexer and Java8Parser classes. I have no idea where to find these.
build.gradle
group 'com.antlr-demo'
version '1.0-SNAPSHOT'
apply plugin: 'java'
sourceCompatibility = 1.8
repositories {
mavenCentral()
}
dependencies {
compile 'org.antlr:antlr4-master:4.5'
testCompile group: 'junit', name: 'junit', version: '4.11'
}
Test.java
Even in this very trivialized example, none of the two classes I need are not being imported.
public static void parseFile(String f) { // found in Test.java:257
try {
if ( !quiet ) System.err.println(f);
// Create a scanner that reads from the input stream passed to us
Lexer lexer = new Java8Lexer(new ANTLRFileStream(f)); // missing
CommonTokenStream tokens = new CommonTokenStream(lexer);
// Create a parser that reads from the scanner
Java8Parser parser = new Java8Parser(tokens); // missing
if ( diag ) parser.addErrorListener(new DiagnosticErrorListener());
if ( bail ) parser.setErrorHandler(new BailErrorStrategy());
if ( SLL ) parser.getInterpreter().setPredictionMode(PredictionMode.SLL);
// start parsing at the compilationUnit rule
ParserRuleContext t = parser.compilationUnit();
if ( notree ) parser.setBuildParseTree(false);
if ( gui ) t.inspect(parser);
if ( printTree ) System.out.println(t.toStringTree(parser));
}
catch (Exception e) {
System.err.println("parser exception: "+e);
e.printStackTrace(); // so we can get stack trace
}
}
It's not described in the pom file...
I have created an open source project on github that does the parser/lexer generation and the AST creation automatically and in-memory. You can find it on github https://github.com/julianthome/inmemantlr.
The code for getting the AST from a JAVA program is pretty simple:
// the ANTLR grammar
File f = new File("src/test/ressources/Java.g4");
// plug the ANTLR grammar in
GenericParser gp = new GenericParser(f, "Java");
// load the file that we'd like to parse into a String variable
String s = FileUtils.loadFileContent("src/test/ressources/HelloWorld.java");
// this listener will create an AST from the java file
gp.setListener(new DefaultTreeListener());
// compile Parser/Lexer
gp.compile();
ParserRuleContext ctx = ctx = gp.parse(s);
// get access to AST
Ast ast = dlist.getAst();
// print AST in dot format
System.out.println(ast.toDot());
If you are interested, you could have a closer look at the test cases in the repository.
You are missing one step. You got the java8 grammar but you haven't yet created a a parser from it. This usually involves running the antlr4 jar on the grammar file (FAQs and more), which is very simple (example taken from the Getting Started page):
$ antlr4 Hello.g4
$ javac Hello*.java
There is a gradle plugin whose name is antlr to translate g4 files into java files. You can get details of antlr plugin at https://docs.gradle.org/current/userguide/antlr_plugin.html
In addition, you can see the real project using gradle + antlr: https://github.com/todylu/xcodeprojectParser
the dependency in antlr grammar project : antlr "org.antlr:antlr4:4.5.3"
the dependency in demo project : compile "org.antlr:antlr4-runtime:4.5.3"
By the way, It's better to install ANTLR v4 grammar plugin into Intellij Idea to assist you edit g4 file.
Related
I am new to Gradle and trying to migrate an existing system build from ant to Gradle.
As part of this I need to run a java program on every file in a directory. Directory contains xml files and the java code will parse and convert .xml to .java files (and these Java files would be build to generate class and package in final jar) after performing some business specific transformation.
below is a function I wrote in Gradle
private runJavaFile(String dirPath) {
FileTree tree = fileTree(dir: dirPath, include: '**/*.xml')
tree.each {
def xmlfile = it.path
def javaFile = it.path.replaceFirst(".xml", ".java")
javaexec { //// getting error on this line
classpath configurations.all
main = 'XmlToJavaParser'
args = ["$xmlfile", "$javaFile", 'Java']
}
}
}
I am calling this function from a Gradle task by passing the dir path which contains the xml files to be parsed.
While running the task, I am getting below error:
> Resolving configuration 'apiElements' directly is not allowed
Any help would be appreciated.
Let me know if any more information is needed.
In Gradle, a configuration represents a group of artifacts and their dependencies. You typically have several configurations depending on what you want to do. For instance, you could have one where you declare which dependencies are needed for compilation, which are only needed at runtime, or which are needed for running a particular Java application.
In your case, you are saying that the classpath to the XmlToJavaParser class is "all configurations combined" and that doesn't really make sense. You are also not allowed to do that as some configurations from the Java plugin are not resolvable like this, which is why you get an error.
So to fix it, you should declare your own configuration for XmlToJavaParser. You can then declare dependencies for it like you normally do. Example (using the Groovy DSL):
configurations {
xmlJavaParser {
canBeResolved = true
canBeConsumed = false
}
}
dependencies {
xmlJavaParser "org.example:xml-java-parser:1.0" // or whatever you need
}
private runJavaFile(String dirPath) {
// ...
javaexec {
classpath = configurations.xmlJavaParser // The configuration is referenced here
main = 'XmlToJavaParser'
args = ["$xmlfile", "$javaFile", 'Java']
}
}
There are also other ways to go about it. But the main point is to not use configurations.all as a classpath.
I'm trying ANTLR 4.8. I'm having problems coding a correct main file that calls to the lexer and parser classes.
After correctly parsing my ANTLR g4 file (getting all files and classes provided by antlr) I've coded the following main java file:
import java.io.*;
import org.antlr.v4.runtime.TokenStream;
import org.antlr.runtime.*;
import org.antlr.runtime.TokenSource;
import org.antlr.v4.runtime.CharStream;
import org.antlr.v4.runtime.CharStreams;
public class example3Ppal {
public static void main(String[] args){
try{
CharStream input = CharStreams.fromFileName(args[0]);
//Create a Lexer with the previously created CharStream
example3Lexer mylexer = new example3Lexer(input);
//Conecting lexer and parser
CommonTokenStream tokens = new CommonTokenStream((TokenSource) mylexer);
example3Parser myparser = new example3Parser((TokenStream) tokens);
myparser.operation();
} catch (java.lang.RuntimeException re) {
System.out.println(re.getMessage());
} catch (IOException e) {
e.printStackTrace();
}
}
}
I started from a former main file, so I had to change from ANTLRFileStream and such to CharStreams. Everything seems to work until I try to connect lexer and parser.
Following examples provided in ANTLR website a lexer object is enough to create a "CommonTokenStream" object that, in addition, should be enough to create a parser object.
Well I firstly tried without any cast but both, eclipse and NetBeans, ask me to cast "mylexer" and "tokens" objects. I don't undestand why, because lexer superclass implements "TokenSource" interface, as well as "CommonTokenStream" does with "TokenStream" interface. In addition both environments allow me to use a "CommonTokenStream" constructor without any attributes while this constructor doesn't exist in the ANTLR documentation
I've read many comments here about similar questions, but I haven't found any that could be applied to my situation.
The result is that it compiles but when I run the program I receive the following error message:
"mylexer cannot be cast to org.antlr.runtime.TokenSource"
There are no prior installations of ANTLR in my computer, the "antlr-4.8-complete" jar file is correctly included as an external jar lib in the projects and it is also included in the CLASSPATH environment variable. I don't know why what should work isn't working for me, could somebody help me? I'm begining to think about reinstalling java, eclipse, netBeans and ANTLR.
Thanks in advance.
You're mixing imports from ANTLR 4 and ANTLR 3. Any import that doesn't have v4 in it, is importing classes from ANTLR 3 (which is possible because the ANTLR 4 jar included ANTLR3 since ANTLR 4 uses ANTLR 4).
If you switch all your imports to org.antlr.v4 and remove the casts, the code should work.
I'm generating javadoc for my Android project with this gradle task:
android.applicationVariants.all { variant ->
task("generate${variant.name.capitalize()}Javadoc", type: Javadoc) {
description "Generates Javadoc for $variant.name."
source = variant.javaCompile.source
classpath = files(variant.javaCompile.classpath.files, project.android.getBootClasspath())
exclude '**/BuildConfig.java'
exclude '**/R.java'
options.links("http://docs.oracle.com/javase/7/docs/api/");
options.linksOffline("http://d.android.com/reference","${android.sdkDirectory}/docs/reference");
options {
failOnError false
}
destinationDir = file("${project.projectDir}/javadoc")
}
}
It excludes R.java, so i don't get R.html in output dir.
However, i'm getting very annoying errors cannot find symbol class R in the process of generating doc for my usual java classes, in the line import com.mypackagename.R. I use common android things like R.string.string_res, so i can't remove this import.
Is there a proper way to include symbol R to index, but not include it to a javadoc, or, at least, simply to supress this error?
You can try to add next two lines to your code:
classpath += files("build/generated/source/r/${variant.flavorName}/release")
classpath += files("build/generated/source/buildConfig/${variant.flavorName}/release")
But in this case your task should depend on one of the tasks which generates R classes.
We are developing a Java project that is able to instrument (change) class files at build time. We defined a Gradle task that invokes a java based Ant task which takes an inputDir (e.g. build/classes) and an outputDir (e.g. build/classes-instrumented) and possible other parameters. The task gets invoked separately for main and test class files after compilation. Since the "normal" java sourceSet is not a good fit, our first thought was to implement our own sourceSet but couldn't find an easy way. A reasonable alternative, similar to ANTLR etc, seemed to be extra variables. Since I needed several, I went for a Map.
sourceSets.all { ext.instrumentation = [:] }
sourceSets.all {
instrumentation.inputDir = null
instrumentation.outputDir = null
instrumentation.classPath = null
}
def postfix = '-instrumented'
Below you see how we initialize the variables.
sourceSets {
main {
instrumentation.inputDir = sourceSets.main.output.classesDir
instrumentation.outputDir = instrumentation.inputDir + postfix
instrumentation.classPath = sourceSets.main.output + configurations.compile
}
test {
instrumentation.inputDir = sourceSets.test.output.classesDir
instrumentation.outputDir = instrumentation.inputDir + postfix
}
}
However it fails with "Could not find method main() for arguments [build_f2cvmoa3v4hnjefifhpuk6ira$_run_closure5_closure23#12a14b74] on root
project 'Continuations'."
We are using Gradle 2.1
I have the following questions:
any idea why the first one fails?
Is the extra variable a reasonable solution to approach the problem?
Thanks a lot for your help
solution: install last version.
I had the same problem, I read gradle documentation of gradle 3, but gradle 2.7 was installed.
checked gradle version 2.7
then read gradle 2.7 doc https://docs.gradle.org/2.7/userguide/tutorial_java_projects.html#N103CD , but found no info about sourceSet in java plugin for that version
installed gradle 3 --> problem solved
I'm trying to execute a simple HelloWorld rule within an OSGi application. During parsing and compiling however, the following exception occurs:
java.lang.NullPointerException
at org.drools.rule.builder.RuleBuilder.build(RuleBuilder.java:47)
at org.drools.compiler.PackageBuilder.addRule(PackageBuilder.java:446)
at org.drools.compiler.PackageBuilder.addPackage(PackageBuilder.java:304)
at org.drools.compiler.PackageBuilder.addPackageFromDrl(PackageBuilder.java:167)
The DRL file is found by the application, since introducing syntax errors results in a failed compilation warning. I guess I'm overlooking something trivial, but haven't found it yet...
I'm using Drools 4.0.7, since this one was available on the Springsource Enterprise Bundle Repository. Here are my application code and drl:
//read in the source
Reader source = new InputStreamReader( getClass().getResourceAsStream( "hello.drl" ) );
PackageBuilder builder = new PackageBuilder();
//this wil parse and compile in one step
builder.addPackageFromDrl( source );
// Check the builder for errors
if ( builder.hasErrors() ) {
System.out.println( builder.getErrors().toString() );
throw new RuntimeException( "Unable to compile \"hello.drl\".");
}
//get the compiled package (which is serializable)
org.drools.rule.Package pkg = builder.getPackage();
//add the package to a rulebase (deploy the rule package).
RuleBase ruleBase = RuleBaseFactory.newRuleBase();
ruleBase.addPackage( pkg );
StatefulSession session = ruleBase.newStatefulSession();
session.fireAllRules();
#created on: May 1, 2011
package test
rule "A stand alone rule"
when
eval(true)
then
System.out.println("hello world");
end
As always, help is highly appreciated.
KR,
Niels
EDIT: During debugging I noticed that the internal builder object within the PackageBuilder was null, as were the package and packagedescription. I got around the original problem by adding this description manually:
PackageBuilder builder = new PackageBuilder();
PackageDescr packageDescr = new PackageDescr("be.ugent.intec.doctr.processor.job.fever");
builder.addPackage(packageDescr);
//this will parse and compile in one step
builder.addPackageFromDrl( source );
My rule was edited to the following form:
package be.ugent.intec.doctr.processor.job.fever
rule "hello"
when
eval( true )
then
System.out.println("hello there");
end
This however results in a compile failure:
BR.recoverFromMismatchedToken
[1,0]: unknown:1:0 mismatched token: [#0,0:6='println',<7>,1:0];
java.lang.RuntimeException: Unable to compile "hello.drl".
at be.ugent.intec.doctr.processor.job.fever.FeverJob.execute(FeverJob.java:45)
at org.quartz.core.JobRunShell.run(JobRunShell.java:202)
at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:525)
When removing the package line from the rule, then my example goes all the way through, without printing anything however. Am I again overlooking anything? I guess this is related to an issue within the drl itself, considering everything stands or falls with the declaration of the package. Just to be clear, the drl is loaded in a class contained in the package be.ugent.intec.doctr.processor.job.fever.
Thx!
A NullPointerException shouldn't happen during compilation: either you get a clear compilation error during parsing (which includes line number) or it works. Drools 4.0.7 is old. This is probably already fixed in a newer version of drools. If it isn't, raise a JIRA issue.
Try a more up-to-date version of drools, preferably even a 5.2 version (5.2.0.CR1 will be out later today or tomorrow), which uses the new, better parser.