We Keep Coding

Esper: Creating new Esper statements, and skipping those that already exist

The project I'm working on uses Esper to create monitoring rules. These rules are either active or inactive, based on a boolean in the SQL row. I would like to set up a check to see if there are any new active rules, create a statement from the, and add them to a hashmap. This will periodically run using Spring scheduler. The code so far looks like this:
private void refreshStatement(Rule rule) throws Expression {
List<String> allRules = dao.getAllRules();
for (String rule : allRules) {
EPStatement statement = epService.getEPAdministrator().createEPL(rule);
statement.addListener(new RuleListener(rule));
ruleMap.put(rule.getId(), statement);
}
}
On the initial run, this works fine. The statements are generated and added to the hashmap ruleMap. Upon the method running a second time due to the scheduler, though, it fails due to the first rule it sees already existing. For example:
ERROR [2018-08-14 12:00:00,000] org.springframework.scheduling.support.TaskUtils$LoggingErrorHandler: Unexpected error occurred in scheduled task.
! com.espertech.esper.epl.expression.core.ExprValidationException: Context by name 'Test_Case' already exists
Is there any good way to check to see if an Esper statement already exists, and to skip the rule if it does? So far, I've tried catching the exception and simply returning a log stating that the EPL statement already exists, that way only new statements would be created:
private Exception e;
private void refreshStatement(Rule rule) throws Exception {
List<String> allRules = dao.getAllRules;
for (String rule : allRules) {
if (e instanceof ExprValidationException) {
log.info("The EPL statement already exists")
}
else {
EPStatement statement = epService.getEPAdministrator().createEPL(rule);
statement.addListener(new RuleListener(rule));
ruleMap.put(rule.getId(), statement);
}
}
}
However, I still got the same exception.
Edit: I just realized that I wrote the for loop wrong. The program will fail on when creating statement, and since that is in the else portion of the loop, it thus never checks for the exception.

You can get the currently-existing statements from EPAdministrator.getStatementNames() and EPAdministrator.getStatement(String name).
Comparing whether a statement already exists is up to your application but EPStatement.getText() returns you the EPL.

Why does SparkLauncher return immediately and spawn no job?

I am using SparkLauncher in Spark v1.6.0. My problem is that when I use this class to launch my Spark jobs, it returns immediately and no job is submitted. My code is as follows.
new SparkLauncher()
.setAppName("test word count")
.setAppResource("file://c:/temp/my.jar")
.setMainClass("my.spark.app.Main")
.setMaster("spark://master:7077")
.startApplication(new SparkAppHandler.Listener() {
#Override public void stateChanged(SparkAppHandle h) { }
#Override public void infoChanged(SparkAppHandle h) { }
});
When I debug into the code, I notice, to my surprise, that all this clazz really does is calls a script spark-submit.cmd using ProcessBuilder.
[C:/tmp/spark-1.6.0-bin-hadoop2.6/bin/spark-submit.cmd, --master, spark://master:7077, --name, "test word count", --class, my.spark.appMain, C:/temp/my.jar]
However, if I run this command (the one that is run by ProcessBuilder) directly on the console, a Spark job is submitted. Any ideas on what's going on?
There's another method SparkLauncher.launch() that is available, but the javadocs say to avoid this method.
Any idea what's going on?

If it works in the console but not from your program, you may need to tell the SparkLauncher where your Spark home is by:
.setSparkHome("C:/tmp/spark-1.6.0-bin-hadoop2.6")
But there could be other things going wrong. You may want to capture additional debugging information by using:
.addSparkArg("--verbose")
and
Map<String, String> env = Maps.newHashMap();
env.put("SPARK_PRINT_LAUNCH_COMMAND", "1");
Pass the env object to the SparkLauncher constructor:
new SparkLauncher(env)

How do you place the new SparkLauncher() statement in the program?
If the main program/unit test immediately finishes after invoking .startApplication(), then the child-process created by it is terminated as well.
You can check the state of the job with the handle created
SparkAppHandle handle = new SparkLauncher()
.setAppName("test word count")
.setAppResource("file://c:/temp/my.jar")
.setMainClass("my.spark.app.Main")
.setMaster("spark://master:7077")
.startApplication();
handle.getState(); // immediately returns UNKNOWN
Thread.sleep(1000); // wait a little bit...
handle.getState(); // the state may have changed to CONNECTED or others
I think that it is because the application takes a certain time to connect to the master, if the program ends before the connection is established, then no job is submitted.

You need to wait for the launcher to get connected to driver nd get you app id and status. For that you can do while loop or something similar.
eg.
while(!handle.getState().isFinal()) {
logger.info("Current state: "+ handle.getState())
logger.info("App Id "+ handle.getAppId());
Thread.sleep(1000L);
// other stuffs you want to do
//
}

IO Exception, my program with input data of x,y sometimes works without any error and randomly sometimes doesnt work with the same x,y input data

i am looping a method with a Thread which reads from a website(dynamically)
all the methods work perfectly, but my problem is that sometimes (3 out of 10 times) that i start the program it throws IO exception at me although i haven't changed my input data from the last known good execution , the exception is coming from the method below:
public String readThisUrlContent() throws ExceptionHandler
{
try {
#SuppressWarnings("static-access")
Document doc = Jsoup.connect(url).timeout(1000).get();
return doc.body().text();
} catch (IOException e) {
throw new ExceptionHandler("IO Exception for reading the site in method setUrlContent in Url class");
}
}
my best guess is that since i'm reading more than one Url with looping this method but the timeout is not sometimes at the best range (considering the internet speed etc. it sometimes doesn't work) but its just my theory and it can be dead wrong but even if its correct i have no idea how to handle it

The problem exactly was the time to live of the opened port. since i had other functions working at the same time program simply needed more connected time so i expanded timeout to (5000) and also reduced the timer of another Time.Schedule method in another method, and so it worked

Lua / Java / LuaJ - Handling or Interrupting Infinite Loops and Threads

I'm using LuaJ to run user-created Lua scripts in Java. However, running a Lua script that never returns causes the Java thread to freeze. This also renders the thread uninterruptible. I run the Lua script with:
JsePlatform.standardGlobals().loadFile("badscript.lua").call();
badscript.lua contains while true do end.
I'd like to be able to automatically terminate scripts which are stuck in unyielding loops and also allow users to manually terminate their Lua scripts while they are running. I've read about debug.sethook and pcall, though I'm not sure how I'd properly use them for my purposes. I've also heard that sandboxing is a better alternative, though that's a bit out of my reach.
This question might also be extended to Java threads alone. I've not found any definitive information on interrupting Java threads stuck in a while (true);.
The online Lua demo was very promising, but it seems the detection and termination of "bad" scripts is done in the CGI script and not Lua. Would I be able to use Java to call a CGI script which in turn calls the Lua script? I'm not sure that would allow users to manually terminate their scripts, though. I lost the link for the Lua demo source code but I have it on hand. This is the magic line:
tee -a $LOG | (ulimit -t 1 ; $LUA demo.lua 2>&1 | head -c 8k)
Can someone point me in the right direction?
Some sources:
Embedded Lua - timing out rogue scripts (e.g. infinite loop) - an example anyone?
Prevent Lua infinite loop
Embedded Lua - timing out rogue scripts (e.g. infinite loop) - an example anyone?
How to interrupt the Thread when it is inside some loop doing long task?
Killing thread after some specified time limit in Java

I struggled with the same issue and after some digging through the debug library's implementation, I created a solution similar to the one proposed by David Lewis, but did so by providing my own DebugLibrary:
package org.luaj.vm2.lib;
import org.luaj.vm2.LuaValue;
import org.luaj.vm2.Varargs;
public class CustomDebugLib extends DebugLib {
public boolean interrupted = false;
#Override
public void onInstruction(int pc, Varargs v, int top) {
if (interrupted) {
throw new ScriptInterruptException();
}
super.onInstruction(pc, v, top);
}
public static class ScriptInterruptException extends RuntimeException {}
}
Just execute your script from inside a new thread and set interrupted to true to stop the execution. The exception will be encapsulated as the cause of a LuaError when thrown.

There are problems, but this goes a long way towards answering your question.
The following proof-of-concept demonstrates a basic level of sandboxing and throttling of arbitrary user code. It runs ~250 instructions of poorly crafted 'user input' and then discards the coroutine. You could use a mechanism like the one in this answer to query Java and conditionally yield inside a hook function, instead of yielding every time.
SandboxTest.java:
public static void main(String[] args) {
Globals globals = JsePlatform.debugGlobals();
LuaValue chunk = globals.loadfile("res/test.lua");
chunk.call();
}
res/test.lua:
function sandbox(fn)
-- read script and set the environment
f = loadfile(fn, "t")
debug.setupvalue(f, 1, {print = print})
-- create a coroutine and have it yield every 50 instructions
local co = coroutine.create(f)
debug.sethook(co, coroutine.yield, "", 50)
-- demonstrate stepped execution, 5 'ticks'
for i = 1, 5 do
print("tick")
coroutine.resume(co)
end
end
sandbox("res/badfile.lua")
res/badfile.lua:
while 1 do
print("", "badfile")
end
Unfortunately, while the control flow works as intended, something in the way the 'abandoned' coroutine should get garbage collected is not working correctly. The corresponding LuaThread in Java hangs around forever in a wait loop, keeping the process alive. Details here:
How can I abandon a LuaJ coroutine LuaThread?

I've never used Luaj before, but could you not put your one line
JsePlatform.standardGlobals().loadFile("badscript.lua").call();
Into a new thread of its own, which you can then terminate from the main thread?
This would require you to make some sort of a supervisor thread (class) and pass any started scripts to it to supervise and eventually terminate if they don't terminate on their own.

EDIT: I've not found any way to safely terminate LuaJ's threads without modifying LuaJ itself. The following was what I came up with, though it doesn't work with LuaJ. However, it can be easily modified to do its job in pure Lua. I may be switching to a Python binding for Java since LuaJ threading is so problematic.
--- I came up with the following, but it doesn't work with LuaJ ---
Here is a possible solution. I register a hook with debug.sethook that gets triggered on "count" events (these events occur even in a while true do end). I also pass a custom "ScriptState" Java object I created which contains a boolean flag indicating whether the script should terminate or not. The Java object is queried in the Lua hook which will throw an error to close the script if the flag is set (edit: throwing an error doesn't actually terminate the script). The terminate flag may also be set from inside the Lua script.
If you wish to automatically terminate unyielding infinite loops, it's straightforward enough to implement a timer system which records the last time a call was made to the ScriptState, then automatically terminate the script if sufficient time passes without an API call (edit: this only works if the thread can be interrupted). If you want to kill infinite loops but not interrupt certain blocking operations, you can adjust the ScriptState object to include other state information that allows you to temporarily pause auto-termination, etc.
Here is my interpreter.lua which can be used to call another script and interrupt it if/when necessary. It makes calls to Java methods so it will not run without LuaJ (or some other Lua-Java library) unless it's modified (edit: again, it can be easily modified to work in pure Lua).
function hook_line(e)
if jthread:getDone() then
-- I saw someone else use error(), but an infinite loop still seems to evade it.
-- os.exit() seems to take care of it well.
os.exit()
end
end
function inithook()
-- the hook will run every 100 million instructions.
-- the time it takes for 100 million instructions to occur
-- is based on computer speed and the calling environment
debug.sethook(hook_line, "", 1e8)
local ret = dofile(jLuaScript)
debug.sethook()
return ret
end
args = { ... }
if jthread == nil then
error("jthread object is nil. Please set it in the Java environment.",2)
elseif jLuaScript == nil then
error("jLuaScript not set. Please set it in the Java environment.",2)
else
local x,y = xpcall(inithook, debug.traceback)
end
Here's the ScriptState class that stores the flag and a main() to demonstrate:
public class ScriptState {
private AtomicBoolean isDone = new AtomicBoolean(true);
public boolean getDone() { return isDone.get(); }
public void setDone(boolean v) { isDone.set(v); }
public static void main(String[] args) {
Thread t = new Thread() {
public void run() {
System.out.println("J: Lua script started.");
ScriptState s = new ScriptState();
Globals g = JsePlatform.debugGlobals();
g.set("jLuaScript", "res/main.lua");
g.set("jthread", CoerceJavaToLua.coerce(s));
try {
g.loadFile("res/_interpreter.lua").call();
} catch (Exception e) {
System.err.println("There was a Lua error!");
e.printStackTrace();
}
}
};
t.start();
try { t.join(); } catch (Exception e) { System.err.println("Error waiting for thread"); }
System.out.println("J: End main");
}
}
res/main.lua contains the target Lua code to be run. Use environment variables or parameters to pass additional information to the script as usual. Remember to use JsePlatform.debugGlobals() instead of JsePlatform.standardGlobals() if you want to use the debug library in Lua.
EDIT: I just noticed that os.exit() not only terminates the Lua script but also the calling process. It seems to be the equivalent of System.exit(). error() will throw an error but will not cause the Lua script to terminate. I'm trying to find a solution for this now.

Thanks to #Seldon for suggesting the use of custom DebugLib. I implemented a simplified version of that by just checking before every instruction if a predefined amount of time is elapsed. This is of course not super accurate because there is some time between class creation and script execution. Requires no separate threads.
class DebugLibWithTimeout(
timeout: Duration,
) : DebugLib() {
private val timeoutOn = Instant.now() + timeout
override fun onInstruction(pc: Int, v: Varargs, top: Int) {
val timeoutElapsed = Instant.now() > timeoutOn
if (timeoutElapsed)
throw Exception("Timeout")
super.onInstruction(pc, v, top)
}
}
Important note: if you sandbox an untrusted script calling load function on Lua-code and passing a separate environment to it, this will not work. onInstruction() seems to be called only if the function environment is a reference to _G. I dealt with that by stripping everything from _G and then adding whitelisted items back.
-- whitelisted items
local sandbox_globals = {
print = print
}
local original_globals = {}
for key, value in pairs(_G) do
original_globals[key] = value
end
local sandbox_env = _G
-- Remove everything from _G
for key, _ in pairs(sandbox_env) do
sandbox_env[key] = nil
end
-- Add whitelisted items back.
-- Global pairs-function cannot be used now.
for key, value in original_globals.pairs(sandbox_globals) do
sandbox_env[key] = value
end
local function run_user_script(script)
local script_function, message = original_globals.load(script, nil, 't', sandbox_env)
if not script_function then
return false, message
end
return pcall(script_function)
end

Discovering the reason for File.mkdirs() failure

If I call one of the methods File.mkdir() or File.mkdirs() in Java, and it returns false, is there a way to know why was the directory not created?

Not really, no. If a SecurityException is NOT thrown, then the most likely cause is a typo in the path, meaning you've accidentally specified a parent path to the new directories that is somehow invalid.
I don't suppose you have it wrapped in a try { ... } catch (Exception e) block, where you don't realize a SecurityException is being thrown, because you're catching an ancestor of SecurityException, do you?
If you have a high belief that everything looks right, and it still fails, I suppose you could simply put it in a loop to retry, say, three times. If it still fails, and depending on your application, you might raise some kind of alert at the UI level, or log the error in a log file (assuming you can write to it).
I suppose it's possible that some deeper I/O issue is preventing it from working, but beyond simply notifying the user of a failure there isn't much you can (or really should) do at an application level. If there's something deeper in the I/O wrong, that's more likely a problem with the system/hardware/OS, or something completely wonky that you have no control over like a subsystem/service crash.
...and if that's happening, that's the responsibility of the IT guy to fix, not your application. Unless of course your app is somehow causing the crash.

I had a mkdirs() failure on windows on a UNC path.
The code looks like this:
public File getOldDirectoryPath(String root, String name)
{
File fulldir = new File(root, name)
boolean created = false
int retry = 0
while (!created) {
retry++
if (!(created = fulldir.exists())) {
if (20 == retry) break
if (!fulldir.mkdirs()) {
sleep(100)
fulldir = new File(root, name)
}
}
}
return fulldir.exists() ? fulldir : null
}
There appears to be some sort of caching involved where exists() returns false (does not exists) but the mkdir on the file system fails because it does exist. Recreating the File() entry or lengthing the timeout did not make a difference.
I discovered a plugin on elasticsearch to fix a SMB problem on Windows. Researching the solution, it uses nio.file instead of io.File. Rewriting the function fixed the issue:
public File getDirectoryPath(String root, String name)
{
Path fulldir = Paths.get(root, name)
boolean created = false
int retry = 0
while (!created) {
retry++
if (!(created = Files.isDirectory(fulldir))) {
if (20 == retry) break
try {
Files.createDirectories(fulldir)
} catch (Throwable thx) {
// error handling
}
}
}
return fulldir.toFile()
}
createDirectories() sometimes fail, but recovers where mkdirs() does not.