I am trying to write a process, which reads the cron expression from a set of records stored in database and runs a job (execute a program if that expression triggers in the next one hour). The records with cron expression in the database can have different triggering times (like Friday or hourly etc).
Example of the table with cron expressions.
----------
0 0 12 * * ? , 12Noon, AJOB
----------
0 11 11 11 11 ? , Nov 11, BJOB
----------
0 15 10 * * ? , EveryDay 10: 15, XJOB
----------
Users can update this crons in the table.
What's the best way to design this kind of application?
The major problem I see here is the following: let's say I ran my job every 1 hour and take the records which are scheduled in the next one hour and run my job, This looks all good when the application is up. If the application is down for 2 hours we might miss some jobs which need to triggered at that time.
How to write this kind of application keeping in mind that the application can fail, but we should not miss any crons during the downtime?
The cron API also has nextTriggerTime, but much less support on previous trigger time.
Since Akka is one of the question tags, I assume you're looking for some Akka based solution.
For cron-like job scheduler in JVM, maybe go straight with Quartz (as suggested by Ashiq), it is Java and should be straightforward to integrate into a Scala+Akka project. Also, take a look at akka-quartz-scheduler. It integrates with Akka well and it provides the cron utils.
In terms of the application:
Maintain a checkpoint table in DB or disk, whichever fits better, to indicate "if cron job ID=X has been run"
Your app loads all cron expressions into memory on startup, trigger quartz scheduler on all of them. If the app doesn't crash, it'll keep running.
Every time your app runs a cron job successfully, update the checkpoint table.
If your app crashes and restarts, lookup the checkpoint table first to understand where it has been, and scheduler the yet-to-finished jobs.
You can use Quartz which is a job scheduling library that can be integrated to Java. It can store and trigger cron jobs and simple jobs. It also has misfire policies which you can utilise in case any of the triggers are missed.
Related
I do not know really what I am looking for and I am not asking for the best solution. I would like to ask for possible solutions to do what I am asking for.
I am building an application using Spring Boot.
I have a database with words in it. I am using these words on a site to search for products and would like to run them over and over again with some time between each search.
So basically the application ask the database for all words that has not been searched for within a specific time (let say 5 minutes). The words that I get in my query response I send out to a KafkaQueue and they are then processed by workers. As soon as they have been used by a worker I make an update in the database with the current time.
So I make a search in the database like every 1 minute (or more often) to find the ones that has not been used for 5 min and then runs them again.
This gives me a lot of connections to the database and I was thinking if there is a better solution. The workers are also saving other data to other tables in the database to.
It is like 80 - 90 words that are turned over like every 5 minutes.
I had a thought to pick some of them and then send them to like a scheduler to set them to run in the remaining of the time until 5 minutes have passed.
If I schedule like 20 tasks at the time will this effect the memory a lot?
Currently I am using postgresql but maybe this is not the best DB for this kind of execution?
I will be able to remove and/or add new words so I do not know if it is possible to even use a in-memory database for the words.
Sounds like a scheduler from SpringBoot is what you need. Look into using the #Scheduled annotation. The following link should provide you with everything Scheduled Annotation .
I ended up using JobRunr which I was able to schedule different delays for each task I set up. So far it has been really smoth using it and it has been working out fine for me.
Scheduled annotation in Spring Boot works really fine to but not for me when I wanted to have more flexibility in setting up different delays for each task.
I have a requirement to send SOAP message to lot of devices everyday at a certain time. I will get the time from a tomcat parameter in web.xml. Something like;
<context-param>
<param-name>DailyTime</param-name>
<param-value>04:00</param-value>
</context-param>
I must create a separate thread that sends the messages. Time will be in 24-hrs format.
The problem is, as a starter i have no idea where to start or how to do it. Can you guys please point me in the right direction or give me some tips, which will help me greatly.
Thank You Everyone :)
You have several options. The two I've used most in the past are:
1) Schedule a cron job to run at the time(s) you want, and have it call an executable java class / jar file.
2) Use a scheduler library like Quartz
Regarding #1 - this assumes you're using a *nix system. If you're using Windows, you can schedule tasks through the Task Scheduler.
Regarding #2 - this gives you more flexibility on the conditions of running a task/job. For example, you could schedule a job to run every 1 minute, but not to start a new job until any existing job is complete.
Anecdotal remark from a version of Quartz circa 2006 - on WebSphere, it seems that my quartz jobs were getting executed by some background thread that made jobs take hours which should have only taken a few seconds. But that was almost a decade ago, and certainly quartz (and hopefully websphere) have vastly improved.
We're developing a web app and are coming to the end of development, and the client we're working with has suddenly sprung the fact on us that we will need to be able to handle load balancing.
We have batch jobs running which would need to run on both servers, but we don't want them to overlap. They are selecting rows from the database, processing the objects, and merging them back into the database. One of these jobs MUST run at the same time each day, though the others run every n minutes. We have about a week at most to get something working, and it'll become technical debt for us.
My question is: what quick and dirty hacks exist to get this working properly? We have a SQLServer 2008 instance and are running Java EE 6 on JBoss 5, which will be load balanced between two servers. We're using Spring 3 and JPA2 backed by Hibernate, and using the stock spring scheduler to schedule and run the jobs. Help me Obi Wan Kenobi; you're my only hope!
on jboss5 u need to use Scheduler API as the simplest solution - the implmentation is built on top of quartz and generally you would user clustered configuration like described here
http://quartz-scheduler.org/documentation/quartz-2.x/configuration/ConfigJDBCJobStoreClustering
Almost 10 years after this question was asked, I had the same need and the "quickest and dirty-ess" solution for me was a load balancer using shared file system without any master.
Each worker locks->picks the jobs from the shared file system, independent of other workers. To balance load, each worker sleeps X seconds between each job polling iteration, where X is proportional to load on the worker (in my case load is count of processes started by worker in the background). Thus high load sleeper gives higher probability to other workers to pick up the next job. Worker loops are running under supervisor (linux).
My use case was execution of sparklyr client-mode jobs on Spark/Hadoop cluster without overloading the edge nodes. It was implemented as a bash script within few hours and then scaled to 3 hosts, and has been stable for some months now - till there is time to invest in a better solution.
I've created a small java program and I want to launch it everyday at 1 o'clock.
I can add it windows task plannifier and it works very well but I want to do it with java.
The java timer task seems to be not good.
I heard about Quartz and when I try their it seems to be complicated for me or I don't find the simple example or tutorial.
Can anyone know some good tutorial or example code easier than the Quartz's site.
Or redirect me to some other site.
Both Quartz and the built in Timer class are not built to start your whole application. They are built to run some specified tasks according to some schedule as long as your application is running.
To actually start your application at a specified time, an external resource will be necessary (unless you want your application to run at all times and only do some activity ever so often).
For that purpose the Windows Task Scheduler is sufficient.
Quartz
Quartz is a full-featured, open source job scheduling service that can be integrated with, or used along side virtually any Java EE or Java SE application.
The basic terminology are (at very basic view):
Scheduler : You can think of this as the core container or something that is the base of quartz.
Job : You can think this as the task we need to do , out simple java Class
Trigger : Something that will make Job to run on scheduler, there are two types of trigger with quartz
Simple Trigger (you can configure it with delay between execution, delay for first execution . . and many such params)
Cron-Trigger : Here you can configure trigger with cron expression.
Also See
Quartz cook book
Quartz : Best practices
small java program and I want to launch it everyday at 1 o'clock
cronjob in unix
Scheduling a Job For a Specific Time Every Day
The basic usage of cron is to execute a job in a specific
time as shown below.This will execute the sample_java_program
everyday at 1am.
30 01 * * * java /home/suresh/sample_java_program
* 30 – 30th Minute
* 01 – 01 AM
* * – every Day
* * – every Month
* * – Every day of the week
Scheduling in windows this link might help u .
cron4j is another one
When you say:
Both Quartz and the built in Timer class are not built to start your whole application.
So i could never launch my whole java program with Quartz or Timer.
it only launches some speicfic task while my prg is running?
so it's better to keep Windows task Scheduler?
ok thank you
I'm developing an application which requires Contents of the database to be written to an ms-excel file at the end of each day. I've written the code for copying the contents into ms-excel file but Now how to proceed further? Whether threads are to be used to check for the completion of 24 hours or there's some other mechanism? Please provide me some guidance.
If you need to facility to run things at set times during the day, you should consider the Quartz Scheduler. It might be overkill, but it's very capable.
For example, you can use its CronTrigger to configure a job to run on a schedule defined by a cron expression, e.g. 0 23 55 * * ? (or something like that) would run your job at 5 to midnight every night (see examples).
Quartz recently got a boost to its future and fortunes by being acquired by the Terracotta folks. Hopefully it'll get some real active development now.
I agree with the others that using something like crontab would be better. However, if you can't do that, I would use the java.util.concurrent package added in Java 1.5. The class you would need is ScheduledThreadPoolExecutor. Specifically, the scheduleAtFixedRate() method.
I think that from the design perspective it is better to use crontab on linux platform or task scheduler on windows platform. It will keep your java program small, and simple. While the solution with thread waiting for the specific time seems simple it will add one serious concern - you will have to monitor its health.
In addition - I would suggest to carefully plan logs your job is writing each time it is run. It is important to have logs for both successful and unsuccessful runs.
It makes sense to make separate file for such logs.
One more case to be considered - what to do if database was not available exactly in the time when job run? Is it acceptable to wait another 24 hours?