I have to run a schduler(any schduled jobs) where i have to fetch 300,000-400,000 in average twice daily from database and apply business logics one by one where each process requests to thirdparty which takes 3-4 seconds to respond.
what are alternates of spring batch processing to process such huge data in efficient ways?
Note: fetched data are not static, data may vary everyday.
May be #Scheduled annotation can help you out.
You can use the Spring Batch schedular. It execute spring batch jobs periodically on fixed schedule using some cron expression passed to Spring TaskScheduler
To configure, batch job scheduling is done in two steps:
Enable scheduling with #EnableScheduling annotation.
Create method annotated with #Scheduled and provide recurrence details using cron job. Add the job execution logic inside this method.
Here is the link of an example : https://howtodoinjava.com/spring-batch/job-scheduler-example/
Related
I have a spring batch job which is run on scheduler. Also I have created a controller which can be involved to trigger the job.
But the issue is, I am not able to identify whether the job is triggered from controller or scheduler. In job configuration I need to give different sql query as per to the job is invoked from controller or scheduler.
How this can be achieved?
Here is scenario,
I am working on one web application,where I am using java, spring, oracle. My requirement is schedule some job where user should specify execution time by some UI and job should be persistent once schedule even if server restarted or application redeployed.
From last few days I am going through lots of stackoverflow post regarding quartz persistence scheduler, but here is tricky part I can not execute any update or insert statement from my java code, I am only allowed to do all db stuff in plsql procedure and call that procedure from java code.
So, the problem is I can't go for quartz persistence implementation directly.It will very great full if anyone tell me right approach to deal with it.
in short how I can implement my own table structure and plsql procedure to make job persistence,which I can load once on application/server start and also allow user to update trigger time and other parameters.
Thanks
What you are looking is for the Quartz JdbcJobStore which stores the jobs and track their execution. Please read the following:
For configuring the JdbcJobStore read this (TX) or this (CMT)
For configuring the data sources read this
#Gan, If you want, you can customize quartz persistance by overriding JobStoreCMT class of Quartz and using that as your job store class in quartz.properties as value for property org.quartz.jobStore.class.
#============================================================================
# Configure JobStore
#============================================================================
org.quartz.jobStore.misfireThreshold = 60000
org.quartz.jobStore.class = <package>.CustomJobStoreCMT
Has "spring batch" ability to limit run jobs without manual check job status? Job can be different or instances of one job. Need something like configurable property.
No. The only way is to manually check via JobExplorer interface or directly query jobs metadata tables
I am looking at Spring Batch 2.0 to implement a pipeline process. The process is listening to some event, and needs to perform a set of transformation steps base on the event type and its content.
Spring batch seem to be a great fit. However, going through the documentation, every example have them job and its steps configured in xml. Does the framework support creating jobs during run-time and configuring the steps dynamically?
the job configuration itself is set before the job runs, but it is possible to create a flexible job configuration with conditional flows
you can't just change the job configuration while the job runs, but between jobs its easy to replace the configuration
Addon for Michael answer:
Do you want to create a flow from beginning to end completely dynamically or you want to have some dynamics at certain point?
As Spring Batch instantiates jobs (will all internals) from XML configuration, that means all necessary beans have setters/getters and you can create the Job from empty page. This is long and bug-prone way (you need to create FlowJob as in JobParserJobFactoryBean goes, then SimpleFlow then StepState then TaskletStep as in SimpleStepFactoryBean and bind them together).
I think the alternative to XML flows could be your coded logic. For String Batch it will look as one step, but with custom implementation and subflow. See <tasklet ref="myCleverTasklet" /> example in Example Tasklet Implementation.
I am using spring batch. I have an ETL process that writes records to a DB and after it completes the ETL process, it will also write a FLAG to PROCESS_COMPLETE table.
Now, I'd like my spring job to trigger once when both the below conditions are true
It is past 5 PM and
The FLAG has been written in PROCESS_COMPLETE table
Appreciate if someone can suggest how to achieve the above using spring batch.
I'd recommend using Quartz for this. The actual triggering the start of a job is not Spring Batch's responsibility. Using Quartz you can create a custom trigger that will fire when both the time and database conditions are met.