How to start batch jobs using DataMinder scheduler? - java

We need to run a batch job each night at 03:00. We have a few processes each built with some tasks.
The idea is to let the processes run each night and download some data files and update other systems.
What I don't understand is how to implement a scheduler (DMScheduler)
that will start a process every night at a specific time.
There exits general information about plugin development but nothing
specific about schedulers as far as I can see.
Can someone explain how DataMinder schedulers work and how to implement one as above?
Perhaps some implementation examples?

The 2 scheduler methods bellow work with "machine" time (as opposed to "human" time):
getNextRunTimeInMilliseconds(long nowInMilliseconds)
setLastStartedTimeInMilliseconds(long lastStartedTimeInMilliseconds)
Please see discussion about "machine" and "human" time at https://docs.oracle.com/javase/tutorial/datetime/iso/instant.html
Basically "machine" time is a time stamp in time without any relation to dates, time zones.
"Human" time is a specific date and time in a specific time zone.
This means that in this case you need to transform between the "machine" time and "human" time.
Calls to method getNextRunTimeInMilliseconds(nowInMilliseconds) may be called multiple times by DataMinder and is just a check
to see if next run time has changed because, for example, plugin/scheduler parameters may have changed.
When a process has been started and finished running the method
setLastStartedTimeInMilliseconds(lastStartedTimeInMilliseconds)
is called with the last time the process was started and run.
Let's take an example:
Here are some date/times and corresponding time stamps:
2016-08-06 03:00:00 = 1470445200000
2016-08-07 03:00:00 = 1470531600000
Assume the process was last run "6:th of August 2016, 03:00 CET". This corresponds to time stamp (as Java long) 1470445200000.
When
setLastStartedTimeInMilliseconds(1470445200000)
is called with above value you need to transfer it to "human" time
"6:th of August 2016, 03:00 CET" and then figure out what the next run time should be.
In this case the next runtime in "human" time would be the day after, that is "7:th of August 2016, 03:00 CET" and this corresponds to
timestamp 1470531600000.
The
getNextRunTimeInMilliseconds(nowInMilliseconds)
should return the new value
of 1470531600000 which tells DataMinder to run the processes on "7:th of August 2016, 03:00 CET". And when the process
has run a call will be made to
setLastStartedTimeInMilliseconds(lastStartedTimeInMilliseconds)
and the scheduler must again figure
out what the next run time should be based on the received lastStartedTimeInMilliseconds.
Hope it make things more clear.

Related

At application startup change all time calls to be from same time but change the year to 1991

Is it possible each time the application starts it can run at a time in the past for demo purposes. i.e all time calls need to return a time in the past!
Run a demo that starts on 1991, I do not want to change the system time. So each time the app starts it sets its time to 1991.
In the project I have a lot of calls of System.currentTimeMillis() but I don't want for every line of code where currentTimeMillis is used to subtract it there I would like maybe overriding System.currentTimeMillis() and internally subtracting the amount of years between now and 1991 and then question is: "Is it possible?".

How can a task be scheduled in Java to occur at midnight, even if a leap second has just occurred?

First, definition of "occur at midnight" is that when task is run, new DateTime() or similar will show 00:00:00 or later for the time portion when converted to a human readable format. Important point is that it must not show 23:59:59 of the previous day.
A common way to achieve this would be to calculate how many milliseconds are between now and the desired point in time, and then use a ScheduledExecutorService to execute the task at the correct time. However, when a leap second is inserted this will result in the task running a second early (or a few milliseconds early depending on how the leap second is 'smeared' and when you scheduled the task):
Runnable task = ...
long numberOfMillisUntilMidnight = ...
ScheduledExecutorService executor = ...
// task runs too early when leap seconds are inserted
executor.schedule(task, numberOfMillisUntilMidnight, TimeUnit.MILLISECONDS);
The reason is that executor.schedule() is based on System.nanoTime() which obviously ignores the leap seconds. I guess what I need is some scheduler based on "run at this time" rather than "run after this amount of time".
For those who are interested, the reason the task must run at midnight related to the fact that all events in my system must be categorized according to which day they occurred on, and in so far as is possible, this needs to be in sync with another system. Of course it would be better if the other system stamped each event with what day it is, but we are not there.
I guess what I need is some scheduler based on "run at this time" rather than "run after this amount of time"
That would be the all-singing, all-dancing solution. But:
First, definition of "occur at midnight" is that when task is run, new DateTime() or similar will show 00:00:00 or later for the time portion...Important point is that it must not show 23:59:59 of the previous day.
(my emphasis)
The simple way is always add a second, or even two. So it'd be 00:00:01 (00:00:00 or later) in the common case, and 00:00:00 (not 23:59:59) in the leap second case.
Based on the resulting discussions it seems clear that, in general, it is unwise to rely on your scheduler to run a task at the "correct" time if "wall time" is important to you. This is also true when running daily tasks at the same "wall time" across daylight savings shifts, although unlike the leap second case, the daylight savings case is well supported by existing tools (by Quartz for example).
Instead I think the best approach for such "wall time sensitive" processes is that when the task is run, check the system clock at that point. If your schedule was inaccurate for whatever reason (leap seconds are not the only time your system clock is adjusted relative to the elapsed time measured by System.nanoTime()) and the time has not yet been reached, then do nothing and reschedule the task for the correct time. This approach would also work for schedules that respond to daylight savings changes but as mentioned above this is already supported by common tools.
This approach was inspired by the comment by Jonathon Reinhart above. Rescheduling rather than sleeping though seems better.
Assuming that your concrete ScheduledExecutorService-implementation relies on System.nanoTime() (as you said) and taking into account your requirement/configuration that the initial delay parameter of the method schedule(...) counts the elapsed milliseconds until next midnight including a possible leap second,
I suggest you to use a leap-second-aware solution. An example using my library Time4J shows how to calculate the delay parameter:
Moment now = SystemClock.currentMoment(); // should not be called during a leap second
Moment nextMidnight =
now.with(
PlainTime.COMPONENT.setToNext(PlainTime.midnightAtStartOfDay()).in(
Timezone.ofSystem().with(
GapResolver.NEXT_VALID_TIME.and(OverlapResolver.EARLIER_OFFSET)
)
)
);
long delayInMilliseconds = SI.NANOSECONDS.between(now, nextMidnight) / 1_000_000;
This code will also choose the earliest valid local time after midnight in case of daylight-saving-change (standard-Java would rather push the time forward by the size of the gap possibly resulting in a local time later than first valid time). For most zones, this is only relevant if choosing an arbitrary start time after midnight (dependent on business requirements).
What so ever, you should also care about connecting your systems to the same NTP-clock. Either you rely on OS-NTP-configuration, or you can use a Java-based clock connecting to NTP (Time4J offers here a solution, too).
If your chosen clock is just doing arbitrary jumps (i.e. if someone has manually adjusted it or in case of bad NTP-configuration) then rescheduling the task after having checked the local walltime again is probably safer. However, I still think that calculating the delay parameter by Time4J-code above is a good idea because the chance to match midnight is higher than just to run the task and rechecking the local time.
You could also combine both approaches: exact calculation of delay AND check/reschedule.

Android running thread every 1 Second to Increment Unix Timestamp

I need to running thread every one second. But when application killed, the thread must be still alive.
My thread task is used for increment Unix Timestamp (that synchronized when the first time application running from our server time) by one every second. I need to create this task because in some device, date time can changed unpredictable (maybe low on battery, hard reset, dropped or something else).
My Activity must be get that Unix Timestamp value when it needed.
From SO, Alarm Manager is not a good choice,
I would recommend you not to use an AlarmManager for 30 seconds, as some have suggested. Because 30 seconds is too short. it will drain the battery. For AlarmManager use a minimum 1 minute with RTC.
Other people suggest using Timer Task or ScheduledExecutorService, what the best thread to fit my need?
Thanks.
You would never achieve that. Any process could be killed by System. And task running every seconds is horrible (like AlarmManager said).
One idea is: save your server time and device time such as SystemClock.elapsedRealtime() . (do not use System.currentTimeMillis() for this purpose. ... this is display time for user and can be changed by user or something).
When you need time later, get elapsedRealtime() again and compare with stored elapsedRealtime(), and add this diff to stored server time. You will get desired time.
Or simply ask current time to your server , depends on needs :).
If you want to care hard reset I think that you should have database on your server to manage the first time when user launches app.

Skipped Jobs because of DST

I have to schedule a few jobs using Java Timer using JDK1.4 without using any third-party API.
If the Daylight Saving Time (DST) change is from 2 am to 3 am , what should be the expected behavior for Jobs scheduled between the DST transition time i.e 2 am and 3 am?
Should the jobs be simply ignored as the time between 2 am and 3 am never appears on the clock
Should they be run immediately at 3 am.
Any other expected behavior ?
I think that many Enterprise applications cannot afford to skip the jobs. How should one proceed ?
Don't store/run/etc. anything using DST. It's... difficult to maintain.
UTC
Store all dates in the database in UTC. Perform all time calculations in UTC. Maintain a single standard non-changing time measurement for all business processes. Only when displaying a result to a user (showing on a screen, printing on a report, etc.) do you then localize the time to whatever that user would expect.
Basically consider all back-end logic to be in UTC, and at the interface level there would be a kind of translation layer between localized time and "system" time.
The answer by David is correct.
ScheduledExecutorService
In addition, I suggest using the ScheduledExecutorService rather than Timer.
Be sure to read up on including all the JavaDoc. Search StackOverflow to find discussion and example code. Specifically you must trap for all Exceptions or else the service will halt.

java Calendar, Date, and Time management for a multi-timezone application

I am designing a scheduling web app.
I expect events to be added by users in several different timezones and locales. The challenge is correct presentation of these events.
So, as an example:
if a user is in EST timezone and is looking at a webinar event that was added by another user in PST, I want to convert the actual PST time of event into a local time for viewer. So, if an event is scheduled for 2 PM PST, then it should shown as 5 PM EST.
I also want to be careful that performance does not take a hit if there are thousands of events which may require a conversion from actual event time to the viewer's local time.
All thoughts and comments are appreciated.
TIA
In general, scheduling future events is a complex subject. You have to make a distinction in the context of what is going to be scheduled:
Does the event occur at a specific universal instant in time? If so, you should record the event time in terms of UTC.
For example, a task that runs every 24 hours would be scheduled by UTC time and not by the local time. It might start off at some local midnight, but as daylight saving time changes take effect it could be running at 23:00 or 01:00 by the local clock.
However, if the event is scheduled by human beings, it is likely to be in terms of a local time, so you should record it that way.
For example, a meeting that occurs at 08:00 Eastern Time will always occur at that local time. In the winter, that would be 13:00 UTC, and in the summer it would be at 12:00 UTC.
So in this context, you cannot record the scheduled start time in terms of UTC. This is a very common mistake, as there is a ton of advice on the Internet that says "always store using UTC", which would be wrong in this scenario.
Instead, you should store two values - the local time such as 08:00 and its IANA time zone identifier, such as America/New_York. You may also need to store a recurrence pattern or specific date depending on how the event is scheduled.
Consider using Joda Time instead of Java's Calendar or Date classess. It will save you from many headaches. Make sure you read the Joda Time documentation and understand how it works.
Joda Time has all of the functions you will need for converting between one time zone and another - which I believe was the primary concern of your question.
Be sure to have a procedure in place for updating the time zone data regularly. Updates are pushed out multiple times a year, as the governments of the world make changes to the legal definitions of their time zones. You cannot just deploy it once and forget about it.
Also be sure you understand that conversion from local time to a specific UTC moment is not a perfect function due to daylight saving time. If an event is scheduled during an invalid or ambiguous local time, you should have a strategy for detecting and dealing with that in your application. You might just apply some assumptions, or you might want to go out of your way to ask the user what to do.
For example, if I schedule an event at 2:00 AM Eastern Time every day, then on March 10th 2013, that time does not exist. Should the event occur at 3:00 AM? Or should it not occur at all?
Another example, if I schedule an event at 1:00 AM Eastern Time every day, then on November 3rd, 2013, that time occurs twice. Should the event happen at the first (daylight time) instance? Or at the second (standard time) instance? Or both? Should I assume one or the other, or should I ask the user which they mean?
Only you can decide what to do, as it is your application. But ignoring the problem will likely lead to errors.
Once an event has passed, you can record it in UTC if you wish, or record it with the full local date time and offset. Either are acceptable. This works just fine for singular past events, just not for recurring future ones.

Categories