I want to run a spring batch job which has set of steps and finally I want to send a notification to redis containing the status of the Job execution. Let's say if all the steps are executed, I should send "Pass". If there was any execution or any error, I want to pass "Fail". So my last step will be notification to redis updating the status regardless of it finished fine or got an exception.
My question is:
Can I achieve this in Spring Batch?
Can I use notification
function as a last step or should I use any specific method for
this?
How can I get the status of jobs?
I know I can get the job status like :
JobExecution execution = jobLauncher.run(job, params);
System.out.println("Exit Status : " + execution.getStatus());
But I call the job in command-line like java -jar app.jar ----spring.batch.job.names=myjobnamehere so that I do not use a JobExecution object.
You can use a JobExecutionListener for that. In the afterJob method, you have a reference to the JobExecution from which you can get the status of the job and send the notification as required.
You can find an example in the getting started guide (See JobCompletionNotificationListener).
Related
We have a java/spring application which runs on EKS pods and we have records stored in MongoDB collection.
STATUS: READY,STARTED,COMPLETED
Application needs to pick the records which are in READY status and update the status to STARTED. Once the processing of the record is completed, the status will be updated to COMPLETED
Once the record is STARTED, it may take few hours to complete, until then other pods(other instance of the same app) should not pick this record. If some exception occurs, the app changes the status to READY so that other pods(or the same pod) can pick the READY record for processing.
Requirement: If the pod crashes when the record is processing(STARTED) but crashes before changing the status to READY/COMPLETED, the other pod should be able to pick this record and start processing again.
We have some solution in mind but trying to find the best solution. Request you to help me with some best approaches.
You can use a shutdown hook from spring:
#Component
public class Bean1 {
#PreDestroy
public void destroy() {
## handle database change
System.out.println(Status changed to ready);
}
}
Beyond that, that kind of job could run better in a messaging architecture, using SQS for example. Instead of using the status on the database to handle and orchestrate the task, you can use an SQS, publish the message that needs to be consumed (the messages that were in ready state) and have a poll of workers consuming messages from this SQS, if something crashes or the pod of this workers needs to be reclaimed, the message goes back to SQS and can be consumed by another pod.
I have streaming job running which will run forever and will execute the query on Kafka topic, I am going through DataProc Documentation for submitting a job via Java, here is the link
// Submit an asynchronous request to execute the job.
OperationFuture<Job, JobMetadata> submitJobAsOperationAsyncRequest =
jobControllerClient.submitJobAsOperationAsync(projectId, region, job);
Job response = submitJobAsOperationAsyncRequest.get();
For the above line of code I am not able to get the response , the above code keeps on running ? Is it because it's streaming job and it's running forever ?
How I can get a response ? So to end user I can provide some job information like URL where they can see their Jobs or any monitoring dashaborad
The OperationFuture<Job, JobMetadata> class has a getMetadata() method which returns a com.google.api.core.ApiFuture<JobMetadata>. You can get the job metadata before the job finishes by calling jobMetadataApiFuture.get().
See more details about the OperationFuture.getMetadata() method and the ApiFuture class.
When I upload a file to s3 bucket a event is triggered and a AWS batch job is started. Is there any way to check the status of AWS batch job in my java code. I have to perform some operation when the status of AWS batch job is SUCCEEDED.
You have the choice of using the ListJobs / DescribeJobs APIs to poll for status.
ListJobsResult listJobs(ListJobsRequest listJobsRequest) Returns a
list of AWS Batch jobs.
You must specify only one of the following items:
A job queue ID to return a list of jobs in that job queue
A multi-node parallel job ID to return a list of that job's nodes
An array job ID to return a list of that job's children
You can filter the results by job status with the jobStatus parameter.
If you don't specify a status, only RUNNING jobs are returned.
Or you can listen for the CloudWatch Events which are emitted as jobs transition from one state to another if you prefer an event-driven architecture.
ListJobsRequest
For solving this problem, I have created separate thread callable thread where looped until status of Job is SUCCEDED and FAILED. Extracted the job status based on job id using describe job API.
class ReturnJobStatus implements Callable<String>
{
public String Callable()
{
while(!(jobStatus.equals("SUCCEEDED") || (jobStatus.equals("FAILED")))
{
// extracts job status using describeJob API after passing jobId
Thread.currentThread().sleep(2000);
}
return jobStatus;
}
I am trying to add a retry logic to a step in java. Using timers, I am able to repeat this step after certain time. However, in case the step doesn't get executed successfully, timer waits till the execution stops. My requirement is that I need to stop execution of this step after say 30s and then retry this logic.
Below is what I am trying to do.
Description of the step: I will call method processRequest by providing a Json input. This step gets executed successfully Everytime and returns a job Id. I am polling a url to get response to the provided job Id. This step sometimes doesn't get executed successfully or takes lot of time to return response. I want to terminate this task of polling and retry to call processRequest again which generates a new job Id and this has to be sent for polling. This logic has to get executed 5 times after waiting for 30s. (Please ignore all the syntax errors if any. These are taken care of in the actual code)
Json request={operation:resize};
String JOBID=processRequest(request);
String response=http://pollingUrl/JOBID;
Can I do this without extending my class to Thread class or please suggest if there's anything else to achieve the same.
I am looking for something like this:
String response="";
Int count=5;
While (count>0)
{
String JOBID=processRequest(request);
String response=http://pollingUrl/JOBID; // terminate step after 30s because this doesnt get terminated automatically
If response !="";
Return response; // return response and stop the while loop.
}
How would it be possible to stop a specific running job in Kettle?
I'm using the following code:
KettleEnvironment.init();
JobMeta jobmeta = new JobMeta(C://Users//Admin//DBTOOL//EDW_Testing_Tool - 1.8(VersionUpgraded)//data-integration//Regress_bug//Start_Validation.kjb,
null);
Job job = new Job(null, jobmeta);
job.initializeVariablesFrom(null);
job.setVariable("Internal.Job.Filename.Directory", Constants.JOB_EXECUTION_KJB_FILE_PATH);
job.setVariable("jobId", jobId.toString());
job.getJobMeta().setInternalKettleVariables(job);
job.stopAll();
How would I ensure that the job which I want to stop is getting stopped and it is not executed after setting the flag?
I'm using rest api to stop the job and i'm not able to get the job Object.
if i'm using CarteSingleton and store the object in map i'm not able to execute the job it gives driver error could not connect to database(eg:-jtds) url not working.