We Keep Coding

Caching data in JVM from external database - Initial load and keep syncing

I am using In memory cache(Spring cache abstraction + Caffeine) and I like to load all the records from our oracle database to our application's memory for caching purpose. So for that, I am planning to perform loading data after application start up and also in regular intervals(#Scheduled with CRON) with the intention of updating all cache entries.
I understood the ApplicationReadyEvent will be triggered after the start up and the method hydrate() will be executing. I am trying to figure out how I can avoid the #Scheduled to be triggered during that time. Let's say If application started at 1:59pm and CRON is scheduled to run at 2pm(To update cache) and the initial loading(started at 1:59 pm) takes about 3-4 mins, I don't want #Scheduled to be run at 2pm. can you suggest some ways I can do that please.Is there a way I can devlop this logic just with #Scheduled? Please also suggest any improvements I can do for this technique.
#Scheduled("assume every 5 mins")
#EventListner(ApplicationReadyEvent.class)
public void hydrate() {
//load from oracle table
//Save to cache - some implementation.
}
Might be silly but one solution I think of is
public class Class {
private boolean isRunning = false;
#Scheduled("assume every 5 mins")
#EventListner(ApplicationReadyEvent.class)
public void hydrate() {
if(!isRunning) {
try {
isRunning = true;
//load from oracle table
//Save to cache - some implementation.
} finally {
isRunning = false;
}
}
}
}
Is this any good?
Thanks

Stopping Quartz job created using MethodInvokingJobDetailFactoryBean

i create a job running a Spring bean class with this code
MethodInvokingJobDetailFactoryBeanjobDetail = new MethodInvokingJobDetailFactoryBean();
Class<?> businessClass = Class.forName(task.getBusinessClassType());
jobDetail.setTargetObject(applicationContext.getBean(businessClass));
jobDetail.setTargetMethod(task.getBusinessMethod());
jobDetail.setName(task.getCode());
jobDetail.setGroup(task.getGroup().getCode());
jobDetail.setConcurrent(false);
Object[] argumentArray = builArgumentArray(task.getBusinessMethodParams());
jobDetail.setArguments(argumentArray);
jobDetail.afterPropertiesSet();
CronTrigger trigger = TriggerBuilder.newTrigger().withIdentity(task.getCode() + "_TRIGGER", task.getGroup().getCode() + "_TRIGGER_GROUP")
.withSchedule(CronScheduleBuilder.cronSchedule(task.getCronExpression())).build();
dataSchedulazione = scheduler.scheduleJob((JobDetail) jobDetail.getObject(), trigger);
scheduler.start();
sometimes the task stop to respond if i remove the trigger and the task from scheduler
remain in
List ob = scheduler.getCurrentlyExecutingJobs();
The state of the trigger is NONE but is still in scheduler.getCurrentlyExecutingJobs();
I have tried to implent InterruptableJob in a class that extend MethodInvokingJobDetailFactoryBeanjobDetail
But when i use
scheduler.interrupt(jobKey);
It say that the InterruptableJob is not implemented.
I think is because the instance of the class is MethodInvokingJobDetailFactoryBeanjobDetail
`scheduler.scheduleJob((JobDetail) jobDetail.getObject(), trigger);`
this is the code inside the quartz scheduler
`job = jec.getJobInstance();
if (job instanceof InterruptableJob) {
((InterruptableJob)job).interrupt();
interrupted = true;
} else {
throw new UnableToInterruptJobException(
"Job " + jobDetail.getKey() +
" can not be interrupted, since it does not implement " +
InterruptableJob.class.getName());
}
`
Is there another way to kill a single task?
I use Quartz 2.1.7 and java 1.6 and java 1.8
TIA
Andrea

There is no magic way to force JVM to stop execution of some piece of code.
You can implement different ways to interrupt the job. But the most appropriate way is to implement InterruptableJob.
Implementing this interface is not sufficient. You should implement a job in such way that it really reacts on such requests.
Example
Suppose, your job is processing 1 000 000 records in the database or in a file and it take relatively long time, let say 1 hour. Then one possible implementation can be following. In the method "interrupt()" you set some flag (member variable) to "true", let name it isInterruptionRequested. In the main logic part that is processing 1 000 000 records you can regularly, e.g. each 5 seconds or after each let say 100 records check if this flag isInterruptionRequested is set to "true". If set, you exit from the method where you implemented the main logic.
It is important that you don't check the condition too often. Otherwise, depending on the logic, it may happen that checking if the job interruption was requested may take 80-90% of CPU, much more than the actual logic :)
Thus, even when you implement the InterruptableJob interface properly, it doesn't mean that the job will be stopped immediately. It will be just a hint like "I would like to stop this job when it is possible". When it will be stopped (if at all) depends on how you implement it.

Monitor progress and intermediate results in Spark

I have a simple Spark task, something like this:
JavaRDD<Solution> solutions = rdd.map(new Solve());
// Select best solution by some criteria
The solve routine takes some time. For a demo application, I need to get some property of each solution as soon as it is calculated, before the call to rdd.map terminates.
I've tried using accumulators and SparkListener, overriding the onTaskEnd method, but it seems to be called only at the end of the mapping, not per thread, E.g.
sparkContext.sc().addSparkListener(new SparkListener() {
public void onTaskEnd(SparkListenerTaskEnd taskEnd) {
// do something with taskEnd.taskInfo().accumulables()
}
});
How can I get an asynchronous message for each map function end?
Spark runs locally or in a standalone cluster mode.
Answers can be in Java or Scala, both are OK.

JRuby: Calling Java Code From A Rack App And Keeping It In Memory

I currently know Java and Ruby, but have never used JRuby. I want to use some RAM- and computation-intensive Java code inside a Rack (sinatra) web application. In particular, this Java code loads about 200MB of data into RAM, and provides methods for doing various calculations that use this in-memory data.
I know it is possible to call Java code from Ruby in JRuby, but in my case there is an additional requirement: This Java code would need to be loaded once, kept in memory, and kept available as a shared resource for the sinatra code (which is being triggered by multiple web requests) to call out to.
Questions
Is a setup like this even possible?
What would I need to do to accomplish it? I am not even sure if this is a JRuby question per se, or something that would need to be configured in the web server. I have experience with Passenger and Unicorn/nginx, but not with Java servers, so if this does involve configuration of a Java server such as Tomcat, any info about that would help.
I am really not sure where to even start looking, or if there is a better way to be approaching this problem, so any and all recommendations or relevant links are appreciated.

Yes, a setup it's possibile ( see below about Deployment ) and to accomplish it I would suggest to use a Singleton
Singletons in Jruby
with reference to question: best/most elegant way to share objects between a stack of rack mounted apps/middlewares? I agree with Colin Surprenant's answer, namely singleton-as-module pattern which I prefer over using the singleton mixin
Example
I post here some test code you can use as a proof of concept:
JRuby sinatra side:
#file: sample_app.rb
require 'sinatra/base'
require 'java' #https://github.com/jruby/jruby/wiki/CallingJavaFromJRuby
java_import org.rondadev.samples.StatefulCalculator #import you java class here
# singleton-as-module loaded once, kept in memory
module App
module Global extend self
def calc
#calc ||= StatefulCalculator.new
end
end
end
# you could call a method to load data in the statefull java object
App::Global.calc.turn_on
class Sample < Sinatra::Base
get '/' do
"Welcome, calculator register:#{App::Global.calc.display}"
end
get '/add_one' do
"added one to calculator register, new value:#{App::Global.calc.add(1)}"
end
end
You can start it in tomcat with trinidad or simply with rackup config.ru but you need:
#file: config.ru
root = File.dirname(__FILE__) # => "."
require File.join( root, 'sample_app' ) # => true
run Sample # ..in sample_app.rb ..class Sample < Sinatra::Base
something about the Java Side:
package org.rondadev.samples;
public class StatefulCalculator {
private StatelessCalculator calculator;
double register = 0;
public double add(double a) {
register = calculator.add(register, a);
return register;
}
public double display() {
return register;
}
public void clean() {
register = 0;
}
public void turnOff() {
calculator = null;
System.out.println("[StatefulCalculator] Good bye ! ");
}
public void turnOn() {
calculator = new StatelessCalculator();
System.out.println("[StatefulCalculator] Welcome !");
}
}
Please note that the register in here is only a double but in your real code you can have a big data structure in your real scenario
Deployment
You can deploy using Mongrel, Thin (experimental), Webrick (but who would do that?), and even Java-centric application containers like Glassfish, Tomcat, or JBoss. source: jruby deployments
with TorqueBox that is built on the JBoss Application Server.
JBoss AS includes high-performance clustering, caching and messaging functionality.
trinidad is a RubyGem that allows you to run any Rack based applet wrap within an embedded Apache Tomcat container
Thread synchronization
Sinatra will use Mutex#synchronize method to place a lock on every request to avoid race conditions among threads. If your sinatra app is multithreaded and not thread safe, or any gems you use is not thread safe, you would want to do set :lock, true so that only one request is processed at a given time. .. Otherwise by default lock is false, which means the synchronize would yield to the block directly.
source: https://github.com/zhengjia/sinatra-explained/blob/master/app/tutorial_2/tutorial_2.md

Here are some instructions for how to deploy a sinatra app to Tomcat.
The java code can be loaded once and reused if you keep a reference to the java instances you have loaded. You can keep a reference from a global variable in ruby.
One thing to be aware of is that the java library you are using may not be thread safe. If you are running your ruby code in tomact, multiple requests can execute concurrently, and those requests may all access your shared java library. If your library is not thread safe, you will have to use some sort of synchronization to prevent multiple threads accessing it.

Prevent jar from running if it's already running? [duplicate]

This question already has answers here:
Closed 10 years ago.
Possible Duplicate:
java/shellscript code to find out if a jar file is already running on current machine
I would love to get a cross-platform solution for this, but even if it's unix only- that would be fine.
The simple solution would be to do this from the shell (Pseudocode)(if ps -aux | grep myJar | print {awk 2}.contains myPID, don't run myProgram.
Now unfortunately our linux team doesn't want a script like that running in production since it can (admittedly) have undesired behaviors.
So what I need is to be able to have a file run, and when it runs see if another program is running. If the program is already running and it's below that time limit, it should prevent the program from running.
A bit of an example:
Myprog.jar -- timeout 5 min
Myprog.jar is in a cron that gets called every 4 minutes,
the first time it's called it launches, the second time it's called it's still running, but since it's not over the timeout, it's fine.
If it's still running when the third check comes through (at 8 minutes into execution) it's killed, and its process is replaced by itself afterwards.
If someone can help me understand how to do this (We've been trying to set up a lock file with limited success)
Thanks!

You could make your program open a dummy file for writing with a FileWriter when your program starts, and keep the file open until the program is finished.
When you now start a second instance of your program, it will also try to open this file for writing, which will throw an IOException, because only one process can have a write handle to a file at the same time.

You could use a port as a semaphore. See this question for more info on that. I think a port would be a good cross-platform solution

You can create a temporary file on a fixed location.
private static final File LOCK_FILE = new File("app.lock");
public static boolean checkIfAlreadyRunning()
{
return LOCK_FILE.exists();
}
public static void createLockFile()
{
LOCK_FILE.createNewFile();
Runnable shutDown = new Runnable()
{
public void run()
{
try
{
LOCK_FILE.delete();
} catch (Exception e) { /* Sad but true */ }
}
};
Runtime.getRuntime().addShutdownHook(new Thread(shutDown));
Thread.setUncaughtExceptionHandler(new Thread.UncaughtExceptionHandler()
{
public void uncaughtException(Thread t, Exception e)
{
shutDown.run();
System.exit(-1);
}
});
}

I had exactly the same problem, and it can be pretty tricky to solve. Both File and Socket based approaches can be made to work, but it gets really tricky on some OS's (think of Windows with multiple users in multiple terminal server sessions etc.).
First, determine the scope where you want only one instance. Then decide on a solution.
The ServerSocket method with a fixed port number will allow you one instance per machine (maybe not exactly what you want).
The locking file approach can be tailored to create the locking file in the users temp directoy, so it gives one instance per session/user.
I personally use a combined approach where the locking file specifies a random port and a second instance connects to that port to pass command line parameter to the running instance.