What works now
In my Java Spring Boot application I'm using Quartz Scheduler v2.2.2 (http://www.quartz-scheduler.org) to dynamically schedule execution of a job. The job store is Quartz managed transactional one with Oracle DB configured. Works.
What needs to work
The triggers that "expired" (finished their last execution) are removed from the database by default. However what I need is that they stay persisted with a COMPLETED status. They should not fire any executions any more, but the Scheduler (http://quartz-scheduler.org/api/2.2.0/org/quartz/Scheduler.html) object should be able to retrieve them. Is there a way to disable trigger automatic removal (even if it's a dirty hack)?
Related
I have to run a schduler(any schduled jobs) where i have to fetch 300,000-400,000 in average twice daily from database and apply business logics one by one where each process requests to thirdparty which takes 3-4 seconds to respond.
what are alternates of spring batch processing to process such huge data in efficient ways?
Note: fetched data are not static, data may vary everyday.
May be #Scheduled annotation can help you out.
You can use the Spring Batch schedular. It execute spring batch jobs periodically on fixed schedule using some cron expression passed to Spring TaskScheduler
To configure, batch job scheduling is done in two steps:
Enable scheduling with #EnableScheduling annotation.
Create method annotated with #Scheduled and provide recurrence details using cron job. Add the job execution logic inside this method.
Here is the link of an example : https://howtodoinjava.com/spring-batch/job-scheduler-example/
I had added quartz scheduler code earlier in our application and now I have removed that code.
The job was scheduled to run every hour.
But now I have removed that code and deployed new war which does not have that code.
But now also I can see logs like below every one hour,
10:00:00.001 [org.springframework.scheduling.quartz.SchedulerFactoryBean#0_Worker-4]
How to kill these threads?
If you are making use of quartz jdbcstore, you need to delete the trigger and job detail records from database.
We run several spring batch jobs within tomcat in the same web application that serves up our UI. Lately we have been adding many more jobs and we are noticing that when we patch our app, several jobs may get stuck in a STARTING or STARTED status. Many of those jobs ensure that another job is not running before they start up, so this means after we patch the server, some of our jobs are broken until we manually run SQL to update the statuses of the jobs to ABANDONED or STOPPED.
I have read here that JobScope and StepScope jobs don't play nicely with shutting down.
That article suggests not using JobScope or StepScope but I can't help but think that this is a solved problem where people must be doing something when the application exits to prevent this problem.
Are there some best practices for handling this scenario? What are you doing in your applications?
We are using spring-batch version 3.0.3.RELEASE
I will provide you an idea on how to solve this scenario. Not necessarily a spring-batch solution.
Everytime I need to add jobs in an application I do as this:
Create a table to control the jobs (queue, priority, status, etc.)
Create a JobController class to manage all jobs
All jobs are defined by the status R-running, F-Finished, Q-Queue (you can add more as you need like aborted, cancelled, etc) (the jobs control these statuses)
The jobController must be loaded only once, you can define it as a spring bean for this
Add a boolean attribute to JobController to inform if you already checked the jobs when you instantiate it. Set it to false
Check if there are jobs with the R status which means that in the last stop of the server they were running so you update every job with this R status to Q and increase their priority so it will get executed first after a restart of the server. This check is inside the if for that boolean attribute, after the check set it to true.
That way every time you call the JobController for the first time and there are unfinished jobs from a server crash you will be able to set then all to a status where it can be executed again. And this check will happens only once since you will be checking that boolean attribute.
A thing that you should be aware of is caution with your jobs priority, if you manage it wrong you may run into a starvation problem.
You can easily adapt this solution to spring-batch.
Hope it helps.
So I started to tinker around with JDBCJobStore in Quartz. Firstly, I could not find a single good resource on how to configure it from scratch. After looking for it for a while and singling out a good resource for beginners, I downloaded the sample application at Job scheduling with Quartz. I have a few doubts regarding it.
How does JDBCJobStore capture jobs.? I mean in order for the job to get stored in the database does the job have to run manually once.? Or will JDBCJobStore automatically detect the jobs and their details..?
How does JDBCJobStore schedule the jobs.? Does it hit the database at a fixed interval like a heartbeat to check if there are any scheduled jobs.? Or does it keep the triggers in the memory while the application is running.?
In order to run the jobs will I have to manually specify the details of the job like like name and group and fetch the trigger accordingly.? Is there any alternative to this.?
On each application restart how can I tell the scheduler to start automatically..? Can it be specified somehow.?
If you are using servlet/app server you can start it during startup:
http://quartz-scheduler.org/documentation/quartz-2.2.x/cookbook/ServletInitScheduler
If you are running standalone you have to initialize it manually i think.
You can read more about JobStores here:
http://quartz-scheduler.org/documentation/quartz-2.2.x/tutorials/tutorial-lesson-09
And about jobs and triggers:
http://quartz-scheduler.org/documentation/quartz-2.2.x/tutorials/tutorial-lesson-02
http://quartz-scheduler.org/documentation/quartz-2.2.x/tutorials/tutorial-lesson-03
http://quartz-scheduler.org/documentation/quartz-2.2.x/tutorials/tutorial-lesson-04
I guess that quartz checks jobs based on time interval to proper work in clusters and distributed systems.
I am using spring batch. I have an ETL process that writes records to a DB and after it completes the ETL process, it will also write a FLAG to PROCESS_COMPLETE table.
Now, I'd like my spring job to trigger once when both the below conditions are true
It is past 5 PM and
The FLAG has been written in PROCESS_COMPLETE table
Appreciate if someone can suggest how to achieve the above using spring batch.
I'd recommend using Quartz for this. The actual triggering the start of a job is not Spring Batch's responsibility. Using Quartz you can create a custom trigger that will fire when both the time and database conditions are met.