I want to be a able to execute a groovy script from my Java application.
I want to reload the groovy script on runtime if needed. According to their tutorials, I can do something like that :
long now = System.currentTimeMillis();
for(int i = 0; i < 100000; i++) {
try {
GroovyScriptEngine groovyScriptEngine = new GroovyScriptEngine("");
System.out.println(groovyScriptEngine.run("myScript.groovy", new Binding()););
} catch (Exception e) {
e.printStackTrace();
}
}
long end = System.currentTimeMillis();
System.out.println("time " + (end - now));//24 secs
myScript.groovy
"Hello-World"
This works fine and the script is reloaded everytime i change a line in myScript.groovy.
The problem is that this is not time efficient, what it does is parsing the script from the file every time.
Is there any other alternative ? E.g something smarter that checks if the script is already parsed and if it did not change since the last parse do not parse it again.
<< edited due to comments >>
Like mentioned in one of the comments, separating parsing (which is slow) from execution (which is fast) is mandatory if you need performance.
For reactive reloading of the script source we can for example use the java nio watch service:
import groovy.lang.*
import java.nio.file.*
def source = new File('script.groovy')
def shell = new GroovyShell()
def script = shell.parse(source.text)
def watchService = FileSystems.default.newWatchService()
source.canonicalFile.parentFile.toPath().register(watchService, StandardWatchEventKinds.ENTRY_MODIFY)
boolean keepWatching = true
Thread.start {
while (keepWatching) {
def key = watchService.take()
if (key.pollEvents()?.any { it.context() == source.toPath() }) {
script = shell.parse(source.text)
println "source reloaded..."
}
key.reset()
}
}
def binding = new Binding()
def start = System.currentTimeMillis()
for (i=0; i<100; i++) {
script.setBinding(binding)
def result = script.run()
println "script ran: $result"
Thread.sleep(500)
}
def delta = System.currentTimeMillis() - start
println "took ${delta}ms"
keepWatching = false
The above starts a separate watcher thread which uses the java watch service to monitor the parent directory for file modifications and reloads the script source when a modification is detected. This assumes java version 7 or later. The sleep is just there to make it easier to play around with the code and should naturally be removed if measuring performance.
Storing the string System.currentTimeMillis() in script.groovy and running the above code will leave it looping twice a second. Making modifications to script.groovy during the loop results in:
~> groovy solution.groovy
script ran: 1557302307784
script ran: 1557302308316
script ran: 1557302308816
script ran: 1557302309317
script ran: 1557302309817
source reloaded...
script ran: 1557302310318
script ran: 1557302310819
script ran: 1557302310819
source reloaded...
where the source reloaded... lines are printed whenever a change was made to the source file.
I'm not sure about windows but I believe at least on linux that java uses the fsnotify system under the covers which should make the file monitoring part performant.
Should be noted that if we are really unlucky, the script variable will be reset by the watcher thread between the two lines:
script.setBinding(binding)
def result = script.run()
which would break the code as the reloaded script instance would not have the binding set. To fix this, we can for example use a lock:
import groovy.lang.*
import java.nio.file.*
import java.util.concurrent.locks.ReentrantLock
def source = new File('script.groovy')
def shell = new GroovyShell()
def script = shell.parse(source.text)
def watchService = FileSystems.default.newWatchService()
source.canonicalFile.parentFile.toPath().register(watchService, StandardWatchEventKinds.ENTRY_MODIFY)
lock = new ReentrantLock()
boolean keepWatching = true
Thread.start {
while (keepWatching) {
def key = watchService.take()
if (key.pollEvents()?.any { it.context() == source.toPath() }) {
withLock {
script = shell.parse(source.text)
}
println "source reloaded..."
}
key.reset()
}
}
def binding = new Binding()
def start = System.currentTimeMillis()
for (i=0; i<100; i++) {
withLock {
script.setBinding(binding)
def result = script.run()
println "script ran: $result"
}
Thread.sleep(500)
}
def delta = System.currentTimeMillis() - start
println "took ${delta}ms"
keepWatching = false
def withLock(Closure c) {
def result
lock.lock()
try {
result = c()
} finally {
lock.unlock()
}
result
}
which convolutes the code somewhat but keeps us safe against concurrency issues which tend to be hard to track down.
Related
We are evaluating GroovyShell interpreter (v2.4) in our application for dynamically executing standard Java syntax.
Earlier, we were using Java BeanShell Interpreter for it, but it has an issue under high load which prompted us to look for an alternative such as Groovy.
Sample Java Code
static String script = "int y = x * x; System.out.println(\"** value of y ** :: \" + y ); ";
GroovyShell gs = new GroovyShell();
Script evalScript = gs.parse("void evalMethod() {" + script + "}");
// bind variables
Binding binding = new Binding();
binding.setVariable("x", 5);
evalScript.setBinding(binding);
// invoke eval method
evalScript.invokeMethod("evalMethod", null);
We are seeing thread lock contention when multiple threads execute above code simultaneously. We have multiple threads in the blocked state which is degrading application performance.
Blocked Thread Call Stack
java.lang.Thread.State: BLOCKED (on object monitor)
at java.lang.ClassLoader.loadClass(ClassLoader.java:404)
- waiting to lock <0x00000007bc425e40> (a java.lang.Object)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
at java.lang.ClassLoader.loadClass(ClassLoader.java:411)
- locked <0x00000007bd369ba8> (a groovy.lang.GroovyClassLoader)
at groovy.lang.GroovyClassLoader.loadClass(GroovyClassLoader.java:677)
at groovy.lang.GroovyClassLoader.loadClass(GroovyClassLoader.java:545)
at org.codehaus.groovy.control.ClassNodeResolver.tryAsLoaderClassOrScript(ClassNodeResolver.java:185)
at org.codehaus.groovy.control.ClassNodeResolver.findClassNode(ClassNodeResolver.java:170)
at org.codehaus.groovy.control.ClassNodeResolver.resolveName(ClassNodeResolver.java:126)
at org.codehaus.groovy.control.ResolveVisitor.resolveToOuter(ResolveVisitor.java:676)
at org.codehaus.groovy.control.ResolveVisitor.resolve(ResolveVisitor.java:313)
at org.codehaus.groovy.control.ResolveVisitor.transformPropertyExpression(ResolveVisitor.java:845)
at org.codehaus.groovy.control.ResolveVisitor.transform(ResolveVisitor.java:696)
at org.codehaus.groovy.control.ResolveVisitor.transformMethodCallExpression(ResolveVisitor.java:1081)
at org.codehaus.groovy.control.ResolveVisitor.transform(ResolveVisitor.java:702)
at org.codehaus.groovy.ast.ClassCodeExpressionTransformer.visitExpressionStatement(ClassCodeExpressionTransformer.java:142)
at org.codehaus.groovy.ast.stmt.ExpressionStatement.visit(ExpressionStatement.java:42)
at org.codehaus.groovy.ast.CodeVisitorSupport.visitBlockStatement(CodeVisitorSupport.java:37)
at org.codehaus.groovy.ast.ClassCodeVisitorSupport.visitBlockStatement(ClassCodeVisitorSupport.java:166)
at org.codehaus.groovy.control.ResolveVisitor.visitBlockStatement(ResolveVisitor.java:1336)
at org.codehaus.groovy.ast.stmt.BlockStatement.visit(BlockStatement.java:71)
at org.codehaus.groovy.ast.ClassCodeVisitorSupport.visitClassCodeContainer(ClassCodeVisitorSupport.java:104)
at org.codehaus.groovy.ast.ClassCodeVisitorSupport.visitConstructorOrMethod(ClassCodeVisitorSupport.java:115)
at org.codehaus.groovy.ast.ClassCodeExpressionTransformer.visitConstructorOrMethod(ClassCodeExpressionTransformer.java:53)
at org.codehaus.groovy.control.ResolveVisitor.visitConstructorOrMethod(ResolveVisitor.java:201)
at org.codehaus.groovy.ast.ClassCodeVisitorSupport.visitMethod(ClassCodeVisitorSupport.java:126)
at org.codehaus.groovy.ast.ClassNode.visitContents(ClassNode.java:1081)
at org.codehaus.groovy.ast.ClassCodeVisitorSupport.visitClass(ClassCodeVisitorSupport.java:53)
at org.codehaus.groovy.control.ResolveVisitor.visitClass(ResolveVisitor.java:1279)
at org.codehaus.groovy.control.ResolveVisitor.startResolving(ResolveVisitor.java:176)
at org.codehaus.groovy.control.CompilationUnit$12.call(CompilationUnit.java:663)
at org.codehaus.groovy.control.CompilationUnit.applyToSourceUnits(CompilationUnit.java:943)
at org.codehaus.groovy.control.CompilationUnit.doPhaseOperation(CompilationUnit.java:605)
at org.codehaus.groovy.control.CompilationUnit.compile(CompilationUnit.java:554)
at groovy.lang.GroovyClassLoader.doParseClass(GroovyClassLoader.java:298)
at groovy.lang.GroovyClassLoader.parseClass(GroovyClassLoader.java:268)
- locked <0x00000007bd372240> (a java.util.HashMap)
at groovy.lang.GroovyShell.parseClass(GroovyShell.java:688)
at groovy.lang.GroovyShell.parse(GroovyShell.java:700)
at groovy.lang.GroovyShell.parse(GroovyShell.java:736)
at groovy.lang.GroovyShell.parse(GroovyShell.java:727)
My questions are:
Has anyone experienced such problem? If yes, can you please suggest any possible resolution for it? I googled around for this problem and few blogs suggested to cache groovy Script object. Is Script object thread-safe? Moreover, I need to bind variables to Script object (via groovy Binding object, which is different for each execution in different threads), so I don't think caching groovy Script object is a viable option.
Are there any best practices that I need to follow while using Groovy Shell interpreter with Java?
Any other library that we can consider? My requirement is to dynamically execute standard Java syntax.
groovy parse&compile is a quite heavy part
if your script is static - then you have to parse it just once
if the scripts always different but repeating - you can cache compiled groovy scripts into
ConcurrentHashMap<String, Class<groovy.lang.Script>>
after
Script evalScript = gs.parse(...);
just put evalScript.getClass() into cache of compiled scripts.
and before gs.parse(...) check if you have compiled script and just take a new instance of the cached class if it exists.
check the following examples and execution speed
String script = "int y = x * x; System.out.println(\"** value of y ** :: \" + y ); ";
def ts = System.currentTimeMillis()
for(int i=0;i<100;i++){
GroovyShell gs = new GroovyShell();
Script evalScript = gs.parse("void evalMethod() {" + script + "}");
// bind variables
Binding binding = new Binding();
binding.setVariable("x", i);
evalScript.setBinding(binding);
// invoke eval method
evalScript.invokeMethod("evalMethod", null);
}
println ">> ${System.currentTimeMillis() - ts} millis"
1165 millis
String script = "int y = x * x; System.out.println(\"** value of y ** :: \" + y ); ";
GroovyShell gs = new GroovyShell();
Class<Script> scriptClass = gs.parse("void evalMethod() {" + script + "}").getClass();
def ts = System.currentTimeMillis()
for(int i=0;i<100;i++){
Script evalScript = scriptClass.newInstance();
// bind variables
Binding binding = new Binding();
binding.setVariable("x", i);
evalScript.setBinding(binding);
// invoke eval method
evalScript.invokeMethod("evalMethod", null);
}
println ">> ${System.currentTimeMillis() - ts} millis"
6 millis
I have an sbt sub-project which compiles messages.json files into new java sources. I've set the task up to run before running tests and before compiling the primary project, or run manually via a new command "gen-messages".
The problem is the message generation takes some time, it always generates all sources, and it is running too often. Some tasks like running tests with coverage end up generating and compiling the messages twice!
How can I monitor the sources to the generator, and only run the source generation if something has changed/or the expected output java files are missing?
Secondly how would I go about running the generator only on changed messages.json files?
Currently the sbt commands I'm using are:
lazy val settingsForMessageGeneration =
((test in Test) <<= (test in Test) dependsOn(messageGenerationCommand)) ++
((compile in Compile) <<= (compile in Compile) dependsOn(messageGenerationCommand)) ++
(messageGenerationCommand <<= messageGenerationTask) ++
(sourceGenerators in Compile += messageGenerationTask.taskValue)
lazy val messageGenerationCommand = TaskKey[scala.collection.Seq[File]]("gen-messages")
lazy val messageGenerationTask = (
sourceManaged,
fullClasspath in Compile in messageGenerator,
runner in Compile in messageGenerator,
streams
) map { (dir, cp, r, s) =>
lazy val f = getFileTree(new File("./subProjectWithMsgSources/src/")).filter(_.getName.endsWith("messages.json"))
f.foreach({ te =>
val messagePackagePath = te.getAbsolutePath().replace("messages.json", "msg").replace("./", "")
val messagePath = te.getAbsolutePath().replace("./", "")
val fi = new File(messagePackagePath)
if (!fi.exists()) {
fi.mkdirs()
}
val ar = List("-d", messagePackagePath, messagePath)
toError(r.run("com.my.MessageClassGenerator", cp.files, ar, s.log))
})
getFileTree(new File("./subProjectWithMsgSources/src/"))
.filter(_.getName.endsWith("/msg/*.java"))
.to[scala.collection.Seq]
}
The message generator creates a directory with the newly created java files - no other content will be in that directory.
Related Questions
sbt generate using project generator
You can use sbt.FileFunction.cached to run your source generator only when your input files or output files have been changed.
The idea is to factor your actual source generation to a function Set[File] => Set[File], and call it via FileFunction.cached.
lazy val settingsForMessageGeneration =
((test in Test) <<= (test in Test) dependsOn(messageGenerationCommand)) ++
((compile in Compile) <<= (compile in Compile) dependsOn(messageGenerationCommand)) ++
(messageGenerationCommand <<= messageGenerationTask) ++
(sourceGenerators in Compile += messageGenerationTask.taskValue)
lazy val messageGenerationCommand = TaskKey[scala.collection.Seq[File]]("gen-messages")
lazy val messageGenerationTask = (
sourceManaged,
fullClasspath in Compile in messageGenerator,
runner in Compile in messageGenerator,
streams
) map { (dir, cp, r, s) =>
lazy val f = getFileTree(new File("./subProjectWithMsgSources/src/")).filter(_.getName.endsWith("messages.json"))
def gen(sources: Set[File]): Set[File] = {
sources.foreach({ te =>
val messagePackagePath = te.getAbsolutePath().replace("messages.json", "msg").replace("./", "")
val messagePath = te.getAbsolutePath().replace("./", "")
val fi = new File(messagePackagePath)
if (!fi.exists()) {
fi.mkdirs()
}
val ar = List("-d", messagePackagePath, messagePath)
toError(r.run("com.my.MessageClassGenerator", cp.files, ar, s.log))
})
getFileTree(new File("./subProjectWithMsgSources/src/"))
.filter(_.getName.endsWith("/msg/*.java"))
.to[scala.collection.immutable.Set]
}
val func = FileFunction.cached(s.cacheDirectory / "gen-messages", FilesInfo.hash) { gen }
func(f.toSet).toSeq
}
I've run into an issue with attempting to parse json in my spark job. I'm using spark 1.1.0, json4s, and the Cassandra Spark Connector, with DSE 4.6. The exception thrown is:
org.json4s.package$MappingException: Can't find constructor for BrowserData org.json4s.reflect.ScalaSigReader$.readConstructor(ScalaSigReader.scala:27)
org.json4s.reflect.Reflector$ClassDescriptorBuilder.ctorParamType(Reflector.scala:108)
org.json4s.reflect.Reflector$ClassDescriptorBuilder$$anonfun$6.apply(Reflector.scala:98)
org.json4s.reflect.Reflector$ClassDescriptorBuilder$$anonfun$6.apply(Reflector.scala:95)
scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
My code looks like this:
case class BrowserData(navigatorObjectData: Option[NavigatorObjectData],
flash_version: Option[FlashVersion],
viewport: Option[Viewport],
performanceData: Option[PerformanceData])
.... other case classes
def parseJson(b: Option[String]): Option[String] = {
implicit val formats = DefaultFormats
for {
browserDataStr <- b
browserData = parse(browserDataStr).extract[BrowserData]
navObject <- browserData.navigatorObjectData
userAgent <- navObject.userAgent
} yield (userAgent)
}
def getJavascriptUa(rows: Iterable[com.datastax.spark.connector.CassandraRow]): Option[String] = {
implicit val formats = DefaultFormats
rows.collectFirst { case r if r.getStringOption("browser_data").isDefined =>
parseJson(r.getStringOption("browser_data"))
}.flatten
}
def getRequestUa(rows: Iterable[com.datastax.spark.connector.CassandraRow]): Option[String] = {
rows.collectFirst { case r if r.getStringOption("ua").isDefined =>
r.getStringOption("ua")
}.flatten
}
def checkUa(rows: Iterable[com.datastax.spark.connector.CassandraRow], sessionId: String): Option[Boolean] = {
for {
jsUa <- getJavascriptUa(rows)
reqUa <- getRequestUa(rows)
} yield (jsUa == reqUa)
}
def run(name: String) = {
val rdd = sc.cassandraTable("beehive", name).groupBy(r => r.getString("session_id"))
val counts = rdd.map(r => (checkUa(r._2, r._1)))
counts
}
I use :load to load the file into the REPL, and then call the run function. The failure is happening in the parseJson function, as far as I can tell. I've tried a variety of things to try to get this to work. From similar posts, I've made sure my case classes are in the top level in the file. I've tried compiling just the case class definitions into a jar, and including the jar in like this: /usr/bin/dse spark --jars case_classes.jar
I've tried adding them to the conf like this: sc.getConf.setJars(Seq("/home/ubuntu/case_classes.jar"))
And still the same error. Should I compile all of my code into a jar? Is this a spark issue or a JSON4s issue? Any help at all appreciated.
I have an application that I'd like to have a prompt in. If it helps, this is a graph database implementation and I need a prompt just like any other database client (MySQL, Postgresql, etc.).
So far I have my own REPL like so:
object App extends Application {
REPL ! Read
}
object REPL extends Actor {
def act() {
loop {
react {
case Read => {
print("prompt> ")
var message = Console.readLine
this ! Eval(message)
}
case More(sofar) => {
//Eval didn't see a semicolon
print(" --> ")
var message = Console.readLine
this ! Eval(sofar + " " + message)
}
case Eval(message) => {
Evaluator ! Eval(message)
}
case Print(message) => {
println(message)
//And here's the loop
this ! Read
}
case Exit => {
exit()
}
case _ => {
println("App: How did we get here")
}
}
}
}
this.start
}
It works, but I would really like to have something with history. Tab completion is not necessary.
Any suggestions on a good library? Scala or Java works.
Just to be clear I don't need an REPL to evaluate my code (I get that with scala!), nor am I looking to call or use something from the command line. I'm looking for a prompt that is my user experience when my client app starts up.
Scala itself, and lots of programs out there, uses a readline-like library for its REPL. Specifically, JLine.
I found another question about this, for which the answers don't seem promising.
BeanShell does some of what you want: http://www.beanshell.org/
I got it. these two blogs really helped.
http://danielwestheide.com/blog/2013/01/09/the-neophytes-guide-to-scala-part-8-welcome-to-the-future.html
http://danielwestheide.com/blog/2013/01/16/the-neophytes-guide-to-scala-part-9-promises-and-futures-in-practice.html
def interprete(code: String) : Future[String] = {
val p = Promise[String]()
Future {
var result = reader.readLine()
p.success(result)
}
writer.write(code + "\n")
writer.flush()
p.future
}
for (ln <- io.Source.stdin.getLines){
val f = interprete(ln)
f.onComplete {
case Success(s) =>
println("future returned: " + s)
case Failure(ex) =>
println(s"interpreter failed due to ${ex.getMessage}")
}
}
I'd like to test which of two implementation in java of a problem is the fastest.
I've 2 jar files with the two implementation I can execute from the terminal. I want to execute both about 100 times and analyse which one is the fastest to do that or that task.
In the output one of the line is "executing time : xx", I need to catch this xx to put in an array or something like that to analyse it later
While I'm executing the jar, I've also to give some input commands (like a name to search or a number).
I don't with which language is it the easiest to do it.
I know the basis in Bash and Python
thank you
Excuse me, but Why you dont make a jar that call n-times any jar?
For Example:
public static void main(String[] args) {
for(int i=0;i<1000;i++) {
params1 = randomNumber();
params2 = randomName();
...
paramsN = randomTime();
MYJAR1 test1 = new MYJAR1(params1,params2,...,paramsN);
timerStart();
test1.start();
timerEnd();
printTimer();
}
}
and make the same for the second jar.
I hope that my idea can help you.
Bye
If you use (say) Perl, you can spawn the process off, capture the output via a redirection and filter the times. e.g.
if (open(PROCESS, "myproc |")) {
while(<PROCESS>) {
if (/executing time : (\d+)/) {
# $1 has the time now
}
}
}
(not compiled or tested!)
Perhaps the simplest way of analysing the data is to redirect the above output to a file, and then load it into Excel. You can then use Excel to calculate averages/max/mins and std. devs (if you wish) trivially.
Ok I've found with 3 different scripts ^^
in the java code, for each function :
long time1 = System.currentTimeMillis();
// some code for function 1
long time2 = System.currentTimeMillis();
try {
OutputStreamWriter osw = new OutputStreamWriter(new FileOutputStream(new File("log.txt"),true));
osw.write("function1,"+(time2 - time1)+"\n");
osw.close();
} catch (FileNotFoundException ex) {
ex.printStackTrace();
} catch (IOException ex) {
ex.printStackTrace();
}
a bash code to run the 500 times the two algorithms
#!/bin/bash
i=0
while [ $i -lt 500 ]
do
./algo1.ex
./algo2.ex
i=$(( $i+1 ))
done
an (or two actually, one for each algorithm) expect code to do the command during the execution
#!/usr/bin/expect -f
spawn java -jar algo1.jar
expect "enter your choice" {send "1\n"}
expect "enter a name :" {send "Peju, M.\n"}
expect "enter your choice" {send "2\n"}
expect "enter a name :" {send "Creasy, R. J.\n"}
expect "enter your choice" {send "0\n"}
exit
As I didn't know how to do it in bash, to count I used a python code
#!/usr/bin/python
# -*- coding: utf8 -*-
import sys
if __name__ == "__main__":
if sys.argv[1:]:
arg = sys.argv[1]
filin = open(arg, 'r')
line = filin.readline()
res= 0, 0, 0
int n = 1
while line !='':
t = line.partition(',')
if t[0] == "function1":
res = (res[0] + int(t[2]), res[1], res[2])
if t[0] == "function2":
res = (res[0], res[1] + int(t[2]), res[2])
if t[0] == "function3":
res = (res[0], res[1], res[2] + int(t[2]))
ligne = filin.readline()
n = n+1
print res
print (res[0]/(n/3.0), res[1]/(n/3.0), res[2]/(n/3.0))
filin.close()
and it works
but thanks for your propositions