Is it possible to get information about already executed (finished) jobs? I browsed the javadocs, learned how to fetch JobDetails etc. but can't find way to learn about the jobs that has already been executed (and finished).
any hints?
You can get next trigger time using below code and compare it with cuurent time, if execution time is in past then job has already executed:
Scheduler scheduler = new StdSchedulerFactory().getScheduler();
for (String groupName : scheduler.getJobGroupNames()) {
for (JobKey jobKey : scheduler.getJobKeys(GroupMatcher.jobGroupEquals(groupName))) {
String jobName = jobKey.getName();
String jobGroup = jobKey.getGroup();
//get job's trigger
List<Trigger> triggers = (List<Trigger>) scheduler.getTriggersOfJob(jobKey);
Date nextFireTime = triggers.get(0).getNextFireTime();
Date currTime = new Date();
if(currTime>nextFireTime )
System.out.println("[jobName] : " + jobName + " [groupName] : "
+ jobGroup + " - " + has already executed);
}
}
If you want to keep track of detailed history of all executions for jobs, then you simply have to make an implementation to keep track of all this information. You can use listeners to for this purpose.
Depending on what exactly you're trying to accomplish, you may either use JobListeners ,TriggerListeners or SchedulerListeners.
For 'global' JobListeners:
<initialize JobListeners>
public void jobWasExecuted(JobExecutionContext context, JobExecutionException jobException) {
try
{
jobKey = context.getJobDetail().getKey();
schedulerName = context.getScheduler().getSchedulerName();
jobName = jobKey.getName();
groupName = jobKey.getGroup();
//execution
Date startDate = context.getFireTime();
//execution time
long runTime=context.getJobRunTime();
//execution end
long endDateM = startDate.getTime() + runTime;
Date endDate = new Date(endDateM);
//get more information here
}
catch (Exception e)
{
e.printStackTrace();
}
Note: Please be vary of the performance impact listeners can cause. As mentioned in the Quartz Docs:
One thing that CAN slow down quartz itself is using a lot of listeners
(TriggerListeners, JobListeners, and SchedulerListeners). The time
spent in each listener obviously adds into the time spent "processing"
a job's execution, outside of actual execution of the job.
This
doesn't mean that you should be terrified of using listeners, it just
means that you should use them judiciously - don't create a bunch of
"global" listeners if you can really make more specialized ones. Also
don't do "expensive" things in the listeners, unless you really need
to. Also be mindful that many plug-ins (such as the "history" plugin)
are actually listeners.
Related
I changed my config-test.json,but application did not print "new config:...",the before scanhander has print .
JsonObject jsonConfig = new JsonObject();
jsonConfig.put("path", "test.json");
ConfigStoreOptions config = new ConfigStoreOptions();
config.setType("file").setOptional(true).setConfig(jsonConfig);
ConfigRetrieverOptions options =
new ConfigRetrieverOptions().addStore(config).setScanPeriod(5000);
ConfigRetriever configRetriever = ConfigRetriever.create(vertx, options);
configRetriever.setBeforeScanHandler(h -> {
System.out.println("config:" + configRetriever.getCachedConfig());
});
configRetriever.listen(change -> {
JsonObject newConfiguration = change.getNewConfiguration();
System.out.println("new config:" + newConfiguration);
JsonObject old = change.getPreviousConfiguration();
System.out.println("old config:" + old);
});
The Javadoc of setBeforeScanHandler says:
Registers a handler called before every scan. This method is mostly used for logging purpose.
Which means that in your case the program will check for changes in the JSON file every five seconds as specified in .setScanPeriod(5000). If the program finds the change during one of those checks, first the configRetriever.setBeforeScanHandler method will trigger, followed by configRetriever.listen.
The order of messages when there is a change will be:
config : {...}
new config:{...}
old config:{...}
The program does not register the config change immediately, but only on the regularly scheduled interval checks, as is stated in the setScanPeriod JavaDoc:
Configures the scan period, in ms. This is the time amount between two
checks of the configuration updates.
DropwizardMetricServices#submit() I'm using doesn't submit the gauge metric for second time.
i.e. My use-case is to remove the gauge metric from JMX after reading it. And my application can send the same metric (with different value).
For the first time the gauge metric is submitted successfully (then my application removes it once it reads the metric). But, the same metric is not submitted the second time.
So, I'm a bit confused what would be the reason for DropwizardMetricServices#submit() not to work for the second time?
Below is the code:
Submit metric:
private void submitNonSparseMetric(final String metricName, final long value) {
validateMetricName(metricName);
metricService.submit(metricName, value); // metricService is the DropwizardMetricServices
log(metricName, value);
LOGGER.debug("Submitted the metric {} to JMX", metricName);
}
Code that reads and removes the metric:
protected void collectMetrics() {
// Create the connection
Long currTime = System.currentTimeMillis()/1000; // Graphite needs
Socket connection = createConnection();
if (connection == null){
return;
}
// Get the output stream
DataOutputStream outputStream = getDataOutputStream(connection);
if (outputStream == null){
closeConnection();
return;
}
// Get metrics from JMX
Map<String, Gauge> g = metricRegistry.getGauges(); // metricRegistry is com.codahale.metrics.MetricRegistry
for(Entry<String, Gauge> e : g.entrySet()){
String key = e.getKey();
if(p2cMetric(key)){
String metricName = convertToMetricStandard(key);
String metricValue = String.valueOf(e.getValue().getValue());
String metricToSend = String.format("%s %s %s\n", metricName, metricValue, currTime);
try {
writeToStream(outputStream, metricToSend);
// Remove the metric from JMX after successfully sending metric to graphite
removeMetricFromJMX(key);
} catch (IOException e1) {
LOGGER.error("Unable to send metric to Graphite - {}", e1.getMessage());
}
}
}
closeOutputStream();
closeConnection();
}
I think I found the issue.
As per the DropwizardMetricServices doc - https://docs.spring.io/spring-boot/docs/current/api/org/springframework/boot/actuate/metrics/dropwizard/DropwizardMetricServices.html#submit-java.lang.String-double- ,
submit() method Set the specified gauge value.
So, I think it's recommended to use DropwizardMetricServices#submit() method to only set the values of any existing gauge metric in JMX and not for adding any new metric to JMX.
So, once I replaced DropwizardMetricServices#submit() with MetricRegistry#register() (com.codahale.metrics.MetricRegistry) method to submit all my metrics it worked as expected and my metrics are readded to JMX (once they were removed by my application).
But, I'm just wondering what makes DropwizardMetricServices#submit() to only add new metrics to JMX and not any metric that's already been removed (from JMX). Does DropwizardMetricServices cache (in memory) all the metrics submitted to JMX? that makes DropwizardMetricServices#submit() method not to resubmit the metric?
I'm currently trying to check the time taken by a mule flow, my Mule ESB version is 3.8.3 if that helps.
I am using <custom-inteceptor> for this task and thus created a Java class which extends AbstractEnvelopeInterceptor class, the code is as follows,
#Override
public MuleEvent last(MuleEvent event, ProcessingTime time, long startTime, boolean exceptionWasThrown)
throws MuleException
{
long endTime = System.currentTimeMillis();
if(logger.isInfoEnabled())
{
logger.info("+++++++++++++++++++++++++++++++++++++++++++++"
+" Flow : "+event.getFlowConstruct().getName()
+ " Started # " +startTime+" Ended # "+endTime
+" Processing Time : "+(endTime - startTime)+" ms "
+ "+++++++++++++++++++++++++++++++++++++++++++++");
}
return event;
}
I am getting a java.lang.NullPointerException when i try to fetch the flow name i.e. at event.getFlowConstruct().getName() which is show as above
Also even if i remove it to just check the time taken by the flow to process , it gives 0 ms as the output , also my batch process doesn't execute after it is printed in the consol
I have tried to place the <custom-inteceptor> at start as well as the end of the flow but still gives me 0 ms for some reason
The flow is as follows,
<flow name="batch-sample-1Flow">
<http:listener config-ref="HTTP_Listener_Configuration" path="/bs" doc:name="HTTP"/>
<custom-interceptor class="com.utility.CustomTimeInteceptor"/>
<batch:execute name="batch-sample-1Batch" doc:name="batch-sample-1Batch"/>
</flow>
Yes I am calling a batch process which dosen't execute when <custom-inteceptor> is placed before the batch process is called.
Would appreciate if there are other ways to monitor batch processes.
EDIT:
I am extending the AbstractEnvelopeInterceptor class is it something related to using it ?
Just use class AbstractInterceptingMessageProcessor and Interceptor interface to create custom interceptor... and do coding like this to create Custom Interceptor
private static Log logger = LogFactory.getLog(TimerInterceptor.class);
public MuleEvent process(MuleEvent event) throws MuleException {
long startTime = System.currentTimeMillis();
DateFormat dateFormat = new SimpleDateFormat("yyyy/MM/dd HH:mm:ss");
Date stdate = new Date();
String start = dateFormat.format(stdate);
System.out.println(start);
MuleEvent resultEvent = processNext(event);
Date enddate = new Date();
String end = dateFormat.format(enddate);
if (logger.isInfoEnabled()) {
long executionTime = System.currentTimeMillis() - startTime;
logger.info("Custom Timer : "+resultEvent.getFlowConstruct().getName() + " Start at "+start+" and end at "+end +" it took " + executionTime + "ms to process event [" + resultEvent.getId() + "]");
}
return resultEvent;
}
Mule already has a build in component <timer-interceptor> for this. I can be placed within the flow.
<flow name="dzone-mule-appsFlow">
<!-- inbound and your components -->
<timer-interceptor doc:name="Timer Interceptor"/>
<!-- your components -->
</flow>
The processing time will be part of the application log. Find out more at https://docs.mulesoft.com/mule-runtime/3.9/using-interceptors#timer-interceptor
I'm new to Hadoop, so I have some doubts what to do in the next case.
I have an algorithm that includes multiple runs of different jobs and sometimes multiple runs of a single job (in a loop).
How should I achieve this, using Oozie, or using Java code? I was looking through Mahout code and in ClusterIterator function function found this:
public static void iterateMR(Configuration conf, Path inPath, Path priorPath, Path outPath, int numIterations)
throws IOException, InterruptedException, ClassNotFoundException {
ClusteringPolicy policy = ClusterClassifier.readPolicy(priorPath);
Path clustersOut = null;
int iteration = 1;
while (iteration <= numIterations) {
conf.set(PRIOR_PATH_KEY, priorPath.toString());
String jobName = "Cluster Iterator running iteration " + iteration + " over priorPath: " + priorPath;
Job job = new Job(conf, jobName);
job.setMapOutputKeyClass(IntWritable.class);
job.setMapOutputValueClass(ClusterWritable.class);
job.setOutputKeyClass(IntWritable.class);
job.setOutputValueClass(ClusterWritable.class);
job.setInputFormatClass(SequenceFileInputFormat.class);
job.setOutputFormatClass(SequenceFileOutputFormat.class);
job.setMapperClass(CIMapper.class);
job.setReducerClass(CIReducer.class);
FileInputFormat.addInputPath(job, inPath);
clustersOut = new Path(outPath, Cluster.CLUSTERS_DIR + iteration);
priorPath = clustersOut;
FileOutputFormat.setOutputPath(job, clustersOut);
job.setJarByClass(ClusterIterator.class);
if (!job.waitForCompletion(true)) {
throw new InterruptedException("Cluster Iteration " + iteration + " failed processing " + priorPath);
}
ClusterClassifier.writePolicy(policy, clustersOut);
FileSystem fs = FileSystem.get(outPath.toUri(), conf);
iteration++;
if (isConverged(clustersOut, conf, fs)) {
break;
}
}
Path finalClustersIn = new Path(outPath, Cluster.CLUSTERS_DIR + (iteration - 1) + Cluster.FINAL_ITERATION_SUFFIX);
FileSystem.get(clustersOut.toUri(), conf).rename(clustersOut, finalClustersIn);
}
So, they have a loop in which they run MR jobs. Is this a good approach? I know that Oozie is used for DAGs, and can be used with another components, such Pig, but should I consider using it for something like this?
What if I want to run clustering algorithm multiple times, let's say for clustering (using specific driver), should I do that in a loop, or using Oozie.
Thanks
If you are looking to run map reduce jobs only then you can consider following ways
chain MR jobs using Map reduce job Control API.
http://hadoop.apache.org/docs/r2.5.0/api/org/apache/hadoop/mapreduce/lib/jobcontrol/JobControl.html
Submit multiple MR jobs from a single driver class.
Job job1 = new Job( getConf() );
job.waitForCompletion( true );
if(job.isSuccessful()){
//start another job with different Mapper.
//change config
Job job2 = new Job( getConf() );
}
If you have a complex DAG or involving multiple ecosystem tools like hive,pig then Oozie suits well.
I'd like to generate alarms on my Java desktop application :
alarms set with a specific date/time which can be in 5 minutes or 5 months
I need to be able to create a SWT application when the alarm is triggered
I need this to be able to work on any OS. The software users will likely have Windows (90% of them), and the rest Mac OS (including me)
the software license must allow me to use it in a commercial program, without requiring to open source it (hence, no GPL)
I cannot require the users to install Cygwin, so the implementation needs to be native to Windows and Unix
I am developing using Java, Eclipse, SWT and my application is deployed from my server using Java Web Start. I'm using Mac OS X.6 for developing.
I think I have a few options:
Run my application at startup, and handle everything myself;
Use a system service.
Use the cron table on Unix, and Scheduled Tasks on Windows
Run at startup
I don't really like this solution, I'm hoping for something more elegant.
Refs: I would like to run my Java program on System Startup on Mac OS/Windows. How can I do this?
System service
If I run it as a system service, I can benefit from this, because the OS will ensure that my software:
is always running
doesn't have/need a GUI
restarts on failure
I've researched some resources that I can use:
run4j — CPL — runs on Windows only, seems like a valid candidate
jsvc — Apache 2.0 — Unix only, seems like a valid candidate
Java Service Wrapper — Various — I cannot afford paid licenses, and the free one is a GPL. Hence, I don't want to/can't use this
My questions in the system service options are:
Are there other options?
Is my planned implementation correct:
at the application startup, check for existence of the service
if it is not installed:
escalate the user to install the service (root on Unix, UAC on Windows)
if the host OS is Windows, use run4j to register the service
if the host OS is Unix, use jsvc to register the service
if it is not running, start it
Thus, at the first run, the application will install the service and start it. When the application closes the service is still running and won't need the application ever again, except if it is unregistered.
However, I think I still miss the "run on startup" feature.
Am I right? Am I missing something?
cron / Task Scheduler
On Unix, I can easily use the cron table without needing the application to escalate the user to root. I don't need to handle restarts, system date changes, etc. Seems nice.
On Windows, I can use the Task Scheduler, even in command-line using At or SchTasks. This seems nice, but I need this to be compatible from XP up to 7, and I can't easily test this.
So what would you do? Did I miss something? Do you have any advice that could help me pick the best and most elegant solution?
Bicou: Great that you shared your solution!
Note that the "schtasks.exe" has some localization issues, if you want to create a daily trigger with it, on an English Windows you'd have to use "daily", on a German one (for example) you'd have to use "täglich" instead.
To resolve this issue I've implemented the call to schtasks.exe with the /xml-option, providing a temporary xml-file which I create by template.
The easiest way to create such a template is to create a task "by hand" and use the "export"-function in the task management GUI tool.
Of the available options you have listed, IMHO Option 3 is better.
As you are looking only for an external trigger to execute the application, CRON or Scheduled tasks are better solutions than other options you have listed. By this way, you remove a complexity from your application and also your application need not be running always. It could be triggered externally and when the execution is over, your application will stop. Hence, unnecessary resource consumption is avoided.
Here's what I ended up implementing:
public class AlarmManager {
public static final String ALARM_CLI_FORMAT = "startalarm:";
public static SupportedOS currentOS = SupportedOS.UNSUPPORTED_OS;
public enum SupportedOS {
UNSUPPORTED_OS,
MAC_OS,
WINDOWS,
}
public AlarmManager() {
final String osName = System.getProperty("os.name");
if (osName == null) {
L.e("Unable to retrieve OS!");
} else if ("Mac OS X".equals(osName)) {
currentOS = SupportedOS.MAC_OS;
} else if (osName.contains("Windows")) {
currentOS = SupportedOS.WINDOWS;
} else {
L.e("Unsupported OS: "+osName);
}
}
/**
* Windows only: name of the scheduled task
*/
private String getAlarmName(final long alarmId) {
return new StringBuilder("My_Alarm_").append(alarmId).toString();
}
/**
* Gets the command line to trigger an alarm
* #param alarmId
* #return
*/
private String getAlarmCommandLine(final long alarmId) {
return new StringBuilder("javaws -open ").append(ALARM_CLI_FORMAT).append(alarmId).append(" ").append(G.JNLP_URL).toString();
}
/**
* Adds an alarm to the system list of scheduled tasks
* #param when
*/
public void createAlarm(final Calendar when) {
// Create alarm
// ... stuff here
final long alarmId = 42;
// Schedule alarm
String[] commandLine;
Process child;
final String alarmCL = getAlarmCommandLine(alarmId);
try {
switch (currentOS) {
case MAC_OS:
final String cron = new SimpleDateFormat("mm HH d M '*' ").format(when.getTime()) + alarmCL;
commandLine = new String[] {
"/bin/sh", "-c",
"crontab -l | (cat; echo \"" + cron + "\") | crontab"
};
child = Runtime.getRuntime().exec(commandLine);
break;
case WINDOWS:
commandLine = new String[] {
"schtasks",
"/Create",
"/ST "+when.get(Calendar.HOUR_OF_DAY) + ":" + when.get(Calendar.MINUTE),
"/SC ONCE",
"/SD "+new SimpleDateFormat("dd/MM/yyyy").format(when.getTime()), // careful with locale here! dd/MM/yyyy or MM/dd/yyyy? I'm French! :)
"/TR \""+alarmCL+"\"",
"/TN \""+getAlarmName(alarmId)+"\"",
"/F",
};
L.d("create command: "+Util.join(commandLine, " "));
child = Runtime.getRuntime().exec(commandLine);
break;
}
} catch (final IOException e) {
L.e("Unable to schedule alarm #"+alarmId, e);
return;
}
L.i("Created alarm #"+alarmId);
}
/**
* Removes an alarm from the system list of scheduled tasks
* #param alarmId
*/
public void removeAlarm(final long alarmId) {
L.i("Removing alarm #"+alarmId);
String[] commandLine;
Process child;
try {
switch (currentOS) {
case MAC_OS:
commandLine = new String[] {
"/bin/sh", "-c",
"crontab -l | (grep -v \""+ALARM_CLI_FORMAT+"\") | crontab"
};
child = Runtime.getRuntime().exec(commandLine);
break;
case WINDOWS:
commandLine = new String[] {
"schtasks",
"/Delete",
"/TN \""+getAlarmName(alarmId)+"\"",
"/F",
};
child = Runtime.getRuntime().exec(commandLine);
break;
}
} catch (final IOException e) {
L.e("Unable to remove alarm #"+alarmId, e);
}
}
public void triggerAlarm(final long alarmId) {
// Do stuff
//...
L.i("Hi! I'm alarm #"+alarmId);
// Remove alarm
removeAlarm(alarmId);
}
}
Usage is simple. Schedule a new alarm using:
final AlarmManager m = new AlarmManager();
final Calendar cal = new GregorianCalendar();
cal.add(Calendar.MINUTE, 1);
m.createAlarm(cal);
Trigger an alarm like this:
public static void main(final String[] args) {
if (args.length >= 2 && args[1] != null && args[1].contains(AlarmManager.ALARM_CLI_FORMAT)) {
try {
final long alarmId = Long.parseLong(args[1].replace(AlarmManager.ALARM_CLI_FORMAT, ""));
final AlarmManager m = new AlarmManager();
m.triggerAlarm(alarmId);
} catch (final NumberFormatException e) {
L.e("Unable to parse alarm !", e);
}
}
}
Tested on Mac OS X.6 and Windows Vista. The class L is an helper to System.out.println and G holds my global constants (here, my JNLP file on my server used to launch my application).
You can also try using Quartz http://quartz-scheduler.org/ . It has a CRON like syntax to schedule jobs.
I believe your scenario is correct. Since services are system specific things, IMHO you should not user a generic package to cover them all, but have a specific mechanism for every system.