I config a Job to execute every 3 hours day time, below is cron config:
#On("0 0 10-20/3 * * ?")
But it didn't work
This is my play staus output:
Requests execution pool:
~~~~~~~~~~~~~~~~~~~~~~~~
Pool size: 20
Active count: 0
Scheduled task count: 876
Queue size: 0
I think I got the answer:
#On("0 0 10-20/3 * * ?")
didn't means the job will run 4 times(10, 13, 16, 19), play will wait for the first job until it ends, and then wait 3 hours to run next job.
so, if this job spend 10 hours, the job will only execute once per day.
Related
We have 4 cron jobs of Quartz each triggering at
0 */5 * ? * * - 5 mins
0 0 */1 ? * * - 1 hr
0 */1 * ? * * - 1 min
0 */3 * ? * * - 3 mins
All the jobs works fine in local and single cluster environment.
but when we switch to more than one pod the job with 3 mins trigger starts to misbehave and goes on to trigger at every 2 hour or randomly.
We are using Springboot and Quartz scheduler and standard Kubernetes Helm chart deployment in our production.
Cron triggers are defined in the application.properties. any reason or pointers to make it consistent would be great
I am working with Spring-boot and Oracle sql developer.
I want to implement something like 'user administrating scheduler'.
User can add or remove scheduler on web page.
If user adds 'scheduler running every 3 minutes', db table may be...
s_id | s_cron | s_detail
sid000001 | 0 0/3 * * * ? | do job 1
and spring scheduler must do 'job 1' in every 3 minutes.
And if, another user also adds 'scheduler running every 1 minutes',
s_id | s_cron | s_detail
sid000001 | 0 0/3 * * * ? | do job 1
sid000002 | 0 0/1 * * * ? | do job 2
and spring scheduler must do 'job 1' in every 3 minutes, and must do 'job 2' in every 1 minutes simultaneously.
The problem is : how can I make it in Spring-boot?
Spring service must add/remove dynamically(or automatically) scheduler with db data added/removed in server running.
Please give me some hands.
If you want to dynamically schedule tasks you can do it without spring by using ExecutorService in particular ScheduledThreadPoolExecutor
Runnable task = () -> doSomething();
ScheduledExecutorService executor =
Executors.newScheduledThreadPool(Runtime.getRuntime().availableProcessors());
// Schedule a task that will be executed in 120 sec
executor.schedule(task, 120, TimeUnit.SECONDS);
// Schedule a task that will be first run in 120 sec and each 120sec
// If an exception occurs then it's task executions are canceled.
executor.scheduleAtFixedRate(task, 120, 120, TimeUnit.SECONDS);
// Schedule a task that will be first run in 120 sec and each 120sec after
the last
execution
// If an exception occurs then it's task executions are canceled.
executor.scheduleWithFixedDelay(task, 120, 120, TimeUnit.SECONDS);
I have Quartz running in a cluster and I do get jobs running periodically. The job is started in one machine and the others will hold until next execution time.
What I want now is to delay the job invocation if the previous invocation isn't finished yet. For instance:
10:00 - instance invocation#1
10:06 - invocation#1 finished
10:10 - instance invocation#2
10:13 - invocation#2 finished
10:20 - instance invocation#3
10:31 - invocation#3 finished // took longer than expected
10:31 - instance invocation#4 // start delayed
10:35 - invocation#4 finished
Even this would be acceptable:
10:00 - instance invocation#1
10:06 - invocation#1 finished
10:10 - instance invocation#2
10:13 - invocation#2 finished
10:20 - instance invocation#3
10:31 - invocation#3 finished // took longer than expected
10:40 - instance invocation#4 // waits for next timed invocation
10:44 - invocation#4 finished
I'm using cron expression like triggers and it is triggered once each 10 minutes (0 0/10 * * *).
Annotating your job with #DisallowConcurrentExecution should do the trick.
I am trying to create a Flowable which emits events respecting backpressure to avoid memory issues, while running each stage of transformation in parallel for efficiency. I have created a simple test program to reason about the behavior of the different steps of my program and when events are being emitted vs. waiting on different stages.
My program is as follows:
public static void main(String[] args) throws ExecutionException, InterruptedException {
Stream<Integer> ints = IntStream.range(0, 1000).boxed().collect(Collectors.toList())
.stream().map(i -> {
System.out.println("emitting:" + i);
return i;
});
Flowable<Integer> flowable = Flowable.fromIterable(() -> ints.iterator());
System.out.println(String.format("Buffer size: %d", flowable.bufferSize()));
Long count = flowable.onBackpressureBuffer(10)
.buffer(10)
.flatMap(buf -> {
System.out.println("Sleeping 500 for batch");
Thread.sleep(500);
System.out.println("Got batch of events");
return Flowable.fromIterable(buf);
}, 1)
.map(x -> x + 1)
.doOnNext(i -> {
System.out.println(String.format("Sleeping : %d", i));
Thread.sleep(100);
System.out.println(i);
})
.count()
.blockingGet();
System.out.println("count: " + count);
}
When I run this, I get output that respects backpressure as expected, where a batch of events is emmited up to the size in buffer, then they are flatmapped, and finally some action is taken where they are printed one-by-one:
Buffer size: 128
emitting:0
emitting:1
emitting:2
emitting:3
emitting:4
emitting:5
emitting:6
emitting:7
emitting:8
emitting:9
Sleeping 500 for batch
Got batch of events
Sleeping : 1
1
Sleeping : 2
2
Sleeping : 3
3
Sleeping : 4
4
Sleeping : 5
5
Sleeping : 6
6
Sleeping : 7
7
Sleeping : 8
8
Sleeping : 9
9
Sleeping : 10
10
emitting:10
emitting:11
emitting:12
emitting:13
emitting:14
emitting:15
emitting:16
emitting:17
emitting:18
emitting:19
Sleeping 500 for batch
Got batch of events
Sleeping : 11
11
Sleeping : 12
12
Sleeping : 13
However if I attempt to parallelize the different stages of operation here by adding some calls to .observeOn(Schedulers.computation()) then it seems like my program no longer respects backpressure. My code now looks like:
public static void main(String[] args) throws ExecutionException, InterruptedException {
Stream<Integer> ints = IntStream.range(0, 1000).boxed().collect(Collectors.toList())
.stream().map(i -> {
System.out.println("emitting:" + i);
return i;
});
Flowable<Integer> flowable = Flowable.fromIterable(() -> ints.iterator());
System.out.println(String.format("Buffer size: %d", flowable.bufferSize()));
Long count = flowable.onBackpressureBuffer(10)
.buffer(10)
.observeOn(Schedulers.computation())
.flatMap(buf -> {
System.out.println("Sleeping 500 for batch");
Thread.sleep(500);
System.out.println("Got batch of events");
return Flowable.fromIterable(buf);
}, 1)
.map(x -> x + 1)
.observeOn(Schedulers.computation())
.doOnNext(i -> {
System.out.println(String.format("Sleeping : %d", i));
Thread.sleep(100);
System.out.println(i);
})
.observeOn(Schedulers.computation())
.count()
.blockingGet();
System.out.println("count: " + count);
}
And my output is the following, where all of my events are emitted upfront instead of respecting the backpressure and buffers specified by the various stages of execution:
Buffer size: 128
emitting:0
emitting:1
emitting:2
emitting:3
emitting:4
emitting:5
emitting:6
emitting:7
emitting:8
emitting:9
emitting:10
Sleeping 500 for batch
emitting:11
emitting:12
... everything else is emitted here ...
emitting:998
emitting:999
Got batch of events
Sleeping 500 for batch
Sleeping : 1
1
Sleeping : 2
2
Sleeping : 3
3
Sleeping : 4
4
Sleeping : 5
Got batch of events
Sleeping 500 for batch
5
Sleeping : 6
6
Sleeping : 7
7
Sleeping : 8
8
Sleeping : 9
9
Sleeping : 10
Got batch of events
Sleeping 500 for batch
10
Sleeping : 11
11
Sleeping : 12
12
Sleeping : 13
13
Sleeping : 14
14
Sleeping : 15
Got batch of events
Sleeping 500 for batch
15
Sleeping : 16
16
Sleeping : 17
17
Sleeping : 18
18
Sleeping : 19
19
Sleeping : 20
Got batch of events
Sleeping 500 for batch
20
Sleeping : 21
21
Sleeping : 22
22
Sleeping : 23
23
Sleeping : 24
24
Sleeping : 25
Got batch of events
Sleeping 500 for batch
25
Pretend my stages of batching are calling out to external services, but that I want them to run in parallel because of latency. I also want to have control of the number of items in memory at a given time because the number of items emitted initially could be highly variable, and the stages operating on batches run much slower than the initial emission of events.
How can I have my Flowable respect backpressure across a Scheduler? Why does it seem to only disrespect backpressure when I sprinkle in calls to observeOn?
How can I have my Flowable respect backpressure across a Scheduler
Actually, applying onBackpressureBuffer makes the source above it disconnect from any backpressure applied by downstream as it is an unbounded-in operator. You don't need it because Flowable.fromIterable (and by the way, RxJava has a range operator) supports and honors backpressure.
Why does it seem to only disrespect backpressure when I sprinkle in calls to observeOn?
In the first example, there is a natural backpressure happening called call-stack blocking. RxJava is synchronous by default and most operators don't introduce asynchrony, just like none do in the first example.
observeOn introduces an asynchronous boundary thus in theory, stages can run in parallel with each other. It has a default 128 element prefetch buffer which can be adjusted via one of its overloads. In your case, however, buffer(10) will actually amplify the prefetch amount to 1280 which may still lead to the complete consumption of your 1000 element long source in one go.
I am running spark job on hadoop cluster, and the job is failing at few times with the exception :
exception : Failing Oozie Launcher, Main class [org.apache.oozie.action.hadoop.JavaMain], main() threw exception, begin > end in range (begin, end): (1494159709088, 1494159706071)
the job ran successfully on the rerun.
After searching on google, It might be Clock skew between the Oozie server host and launcher host.
Is there a way i can check if there is clock skew ? or how can i check the time on all the nodes whether they are in sync or not.
Thanks
ntptime command output :
ntp_gettime() returns code 0 (OK)
time dcb9b19b.a2328f64 Sun, May 7 2017 14:45:47.633, (.633584090),
maximum error 434990 us, estimated error 815 us, TAI offset 0
ntp_adjtime() returns code 0 (OK)
modes 0x0 (),
offset 176.871 us, frequency -25.666 ppm, interval 1 s,
maximum error 434990 us, estimated error 815 us,
status 0x2001 (PLL,NANO),
time constant 10, precision 0.001 us, tolerance 500 ppm,
ntpstat command output :
synchronised to NTP server (174.68.168.57) at stratum 3
time correct to within 77 ms
polling server every 1024 s