What is The Z at the end of Date in Elasticsearch - java

Why in Elasticsearch we have the 'Z'
at the end of the date field?
For instance:
2016-05-16T00:00:00.000Z
What does it mean?
Is this something useful for anything?
Is it harmful?
Can I get rid of it?
What about joda time?

What does it mean?
The 'Z' means UTC.
Reference:
https://www.w3.org/TR/NOTE-datetime
Try not to store local dates. If you want to find a good thread on dates and why you should use UTC check this thread.
Is this something useful for anything?
It is very useful, storing all dates in UTC allows for easy conversion of dates from locale to locale. It also allows for date comparison and date visualisations to be consistent.
Is it harmful?
No.
Can I get rid of it?
If you wish? I wouldn't recommend doing it... It's a bit like removing the currency field from a transaction. The transaction doesn't make any sense.
Another example with dates:
I rang my friend in Country X (+6 hours ahead) at 1pm December 24th 2016.
For him it was 7pm.
So we have two local date times.
One for me:
1pm December 24th 2016 in London
One for my friend:
7pm December 24th 2016 in Country X
If I delete the 'in ...' Part these two become become two instances of time 5 hours apart of each other. Which means we couldn't have possibly spoken on the phone? Right?
No, because they are the same instance in time, across two locales.
What about joda time?
What about it?...
I hope this helps.

From RFC3339 about Date and Time on the Internet:
Numeric offsets are calculated as "local time minus UTC". So the
equivalent time in UTC can be determined by subtracting the offset
from the local time. For example, 18:50:00-04:00 is the same time as
22:50:00Z. (This example shows negative offsets handled by adding
the absolute value of the offset.)
So, a date with Z at the end is the date&time in UTC. And it should be the equivalent of 2016-05-16T00:00:00.000Z-00:00.
The presence of the timezone offset or not is just a matter of the date format being used in Elasticsearch's field definition.

Related

Do I have a way to avoid re-creating lots of String objects?

My project uses Javascript and Java (Android) for the client and Java for the backend.
When I started working on my project, I stored dates as days from epoch (long) and all was good. I then found out that my project doesn't work well with timezones. Suddenly dates were +1 -1 days off. Depending on the client's location in the world.
After a short investigation, I saw that the foolproof way to avoid it was to store the dates as String yyyy-MM-ddT00:00 so when using the Javascript's new Date(dateStr), it creates it correctly and all was good. Ofcourse I could store the dates as yyyy-MM-dd and just send it to the client as yyyy-MM-ddT00:00 but that won't solve the question I have.
After that, I was wondering whether Java (backend) is handled correctly. I use LocalDate when I want to "play" with dates and LocalDate.parse doesn't like yyyy-MM-ddT00:00 format, instead it works with yyyy-MM-dd so whenever I needed dates, I did LocalDate.parse(dateStr.substring(0,10)). LocalDateTime does work with yyyy-MM-ddT00:00 but I don't need the time part and it had its own issues, which I don't remember what they were at the moment.
So now I have a lot of String manipulation (inside loops) that actually creates more String objects. One can say it's not that much of a stress and I shouldn't pay attention to that but I want to make sure I'm not missing something and maybe there's another way (maybe silly enough that I've missed) to overcome this.
Thanks
Update: The events are stored from a different source and only the date itself is important so if an event happened on 2020-06-17, this is the date all users should see, no matter where they are.
I'm using new Date(dateStr) in Javascript. If dateStr is 2020-06-17, the date object uses the client's timezone and the date might be +-1 depending on the client's timezone. If dateStr is 2020-06-17T00:00 then the date object is created as expected no matter where the client is located.
Assuming the above, which I hope is clearer now, creating String objects over and over again is a memory stress that I should consider or is it something Java handles with no problem and I shouldn't worry about this?
My question was closed and I was told to edit it to be more focused. After editing my question, how can I re-open my question to answers?
As you have discovered, storing dates in terms of days since some epoch only works if everyone who uses your system is using the same time zone. If two different users in different time zones have a different idea about the date on which some event occurred (e.g., the person in New York says that the system crashed on Sunday night, but the the person in Hong Kong says it crashed on Monday morning), then you have to store the time zone in which the event occurred in order to show the date of that event accurately.
But if that's the situation you're in, why not just store the time zone along with the date? There's no compelling reason to combine the date and timezone into a string.
When you parse a ISO-formatted timestamp into a LocalDate using only the first 10 characters, be aware that you're losing the time zone information. Implicitly the LocalDate that you get is in the time zone of the original timestamp. So if the original timestamp is New York time, and you take the date part and add 1 day, then you'll get the next day in the New York time zone. But if you then take the date from a second timestamp, you can't compare it to the date you got from the first timestamp, in terms of determining if it represents the "same day." You can only test for "same day" if both dates are implicitly in the same time zone.
UPDATE
After reading your additional comments, I realize that what's happening is this. You have a date stored in your database, like 2020-06-15. You send that to the UI as the string '2020-06-15' and then do new Date('2020-06-15') and then you're surprised when you render the date in the UI and get June 14!
This is the transformation that happens:
The string '2016-06-15' gets parsed into a JavaScript Date representing midnight UTC on the June 15.
When you render the date, it gets converted into a string using the browser's local time zone, which (if you're in the United States) will give you June 14, because at midnight UTC on June 15 it's still June 14 in all time zones west of Greenwich.
You discovered that if you make the string "2020-06-15T00:00" that it works, because now JavaScript uses the browser's local time zone to parse the string. In other words, this string means midnight local time, not UTC, on June 15. So now the sequence is:
'2020-06-15T00:00' gets parsed using the local time zone and becomes June 15 4:00AM UTC.
When you render the date, it gets converted back to local time and is rendered as June 15.
The easiest way to avoid all this messiness is just to send the regular date string '2020-06-15' to the UI and render it using DateTimeFormat, specifying the time zone as UTC:
new Intl.DateTimeFormat('en-US', {timeZone: 'UTC'}).format(d)
Since dates in JavaScript are always UTC, and you're asking DateTimeFormat to output the date in UTC, no date shift occurs.
You could also use the Date methods getUTCFullYear, getUTCMonth, etc. to get the date components and format them however you like.
Once you're no longer sending dates back and forth with "T00:00" appended, you can just use LocalDate on the Java side.
Don't spend even a second worrying about the time required to manipulate strings. Think about the incredible amount of string manipulation that is necessary to build even a simple web page. A few more strings here and there isn't going to make a difference.

CST/CDT time zone change issue

We are storing time in like '22-NOV-17 05.33.51.937000000 PM' format with server default timezone CST. We have half an our time comparison in many places. So CST to CDT and CDT to CST are facing issues because on retrieval time for database we can not identify the time zone. So it is breaking our time comparison on CST to CDT and CDT to CST time changes.
We can not change our storing logic like store with timezone and store in UTC timezone because it will breaking our existing logic in many places.
So is there any way to identity date timezone like CST or CDT, stored in database with '22-NOV-17 05.33.51.937000000 PM' format.
We are storing time in like '22-NOV-17 05.33.51.937000000 PM' format
does not make sense with your comment
We are storing as a timestamp in database
In Oracle databases, a TIMESTAMP does not have a format - it is stored in the database as 11 bytes representing year (2 bytes), month, day, hours, minutes, seconds (1 byte each) and fractional seconds (4 bytes). It is only when whatever interface you are using (SQL/Plus, SQL Developer, Toad, Java, PHP, etc.) to talk to the database decides to show it to you, the user, that that interface will format it as a string (but the database will just keep it as bytes without any format).
Assuming you are using SQL/Plus or SQL Developer then you can find the default format using:
SELECT value FROM NLS_SESSION_PARAMETERS WHERE parameter = 'NLS_TIMESTAMP_FORMAT';
And change the default format using:
ALTER SESSION SET NLS_TIMESTAMP_FORMAT = 'YYYY-MM-DD HH24:MI:SSXFF9';
Or for TIMESTAMP WITH TIME ZONE
ALTER SESSION SET NLS_TIMESTAMP_TZ_FORMAT = 'YYYY-MM-DD HH24:MI:SSXFF9 TZR';
So is there any way to identity date timezone like CST or CDT, stored in database with '22-NOV-17 05.33.51.937000000 PM' format.
No, without any other meta-data that could identify the source of the timestamp and indicate which location it came from (i.e. is there another column that links to the user who entered the data that could be mapped to a physical location and so a time zone) then it is impossible to determine which time zone it is from.
You will either need to:
change your database column to TIMESTAMP WITH TIME ZONE and store the time zone; or
convert all the values to the same time zone when you are storing them.
I am assuming by CST and CDT you mean North American Central Standard Time and Central Daylight Time such as observed in Rainy River, Chicago and Mexico (the city) among other places. More on this ambiguity later.
For 99.977 % of all times it is fairly easy to know whether they are standard time or daylight saving time. Only times from the two hours around the transition from DST to standard time are ambiguous, and as said in the comments, there is no way to know from the time stamp which is the right way to resolve this ambiguity.
java.time
This answer will take you as far into the future as possible without taking you away from Java 7. You can still use java.time, the modern Java date and time API also known as JSR-310. It has been backported to Java 6 and 7 in the ThreeTen Backport, so it’s a matter of getting this and adding it to your project (just until one day you upgrade to Java 8 or later).
I am taking your word for your date-time string format. What we can do with it:
DateTimeFormatter storedFormatter = new DateTimeFormatterBuilder()
.parseCaseInsensitive()
.appendPattern("d-MMM-uu hh.mm.ss.SSSSSSSSS a")
.toFormatter(Locale.US);
ZoneId zone = ZoneId.of("America/Mexico_City");
String storedTime = "22-NOV-17 05.33.51.937000000 PM";
LocalDateTime dateTime = LocalDateTime.parse(storedTime, storedFormatter);
// First shot -- will usually be correct
ZonedDateTime firstShot = ZonedDateTime.of(dateTime, zone);
System.out.println(firstShot);
This prints:
2017-11-22T17:33:51.937-06:00[America/Mexico_City]
You can see that it picked an offset of -06:00, which means that the time is in standard time (CDT is -05:00).
Since your month abbreviation is in all uppercase, I needed to tell the formatter to parse case insensitively. If America/Mexico_City time zone is not appropriate for you, pick a better one, for example America/Rainy_River or America/Chicago.
Ambiguous times in fall
I once had to parse a log file containing date-times without indication of standard time and summer time (DST). Since we assumed time would always move forward, we failed at the transition to standard time, and one hour of the log file was lost. In this case we might have solved it using the information that times were in summer time until the leap backward by an hour, from there they were in standard time. You may want to think about whether something similar will be possible for you.
Other options include just taking DST time every time — this is what the above code will do — or taking an average and living with the error thus introduced.
We can at least detect the ambiguous times:
ZoneOffset standardOffset = ZoneOffset.ofHours(-6);
ZoneOffset dstOffset = ZoneOffset.ofHours(-5);
// check if in fall overlap
ZonedDateTime standardDateTime
= ZonedDateTime.ofLocal(dateTime, zone, standardOffset);
ZonedDateTime dstDateTime
= ZonedDateTime.ofLocal(dateTime, zone, dstOffset);
if (! standardDateTime.equals(dstDateTime)) {
System.out.println("Ambiguous, this could be in CST or CDT: " + dateTime);
}
Now if the string was 29-OCT-17 01.30.00.000000000 AM, I get the message
Ambiguous, this could be in CST or CDT: 2017-10-29T01:30
ZonedDateTime.ofLocal() will use the provided offset for resolving the ambiguity if it is a valid offset for the date-time and zone.
Non-existing times in the spring
Similarly we can detect if your date-time falls in the gap where the clock is moved forward in the transition to DST:
// Check if in spring gap
if (! firstShot.toLocalDateTime().equals(dateTime)) {
System.out.println("Not a valid date-time, in spring gap: " + dateTime);
}
This can give a message like
Not a valid date-time, in spring gap: 2018-04-01T02:01
I suggest you can safely reject such values. They cannot be correct.
Avoid the three letter time zone abbreviations
CST may refer to Central Standard Time (in North and Central America), Australian Central Standard Time, Cuba Standard Time and China Standard Time. CDT may mean Central Daylight Time or Cuba Daylight Time. The three and four letter abbreviations are not standardized and are very often ambiguous. Prefer time zone IDs in the region/city format, for example America/Winnipeg.

Finding which timezone names are at 8am

I have a user table in MySQL with a time zone column in Europe/Paris format. My goal is to find user for whom it's currently 8am from my Java app.
Just using Europe/Paris won't work because Europe/Brussels and Europe/Amsterdam have the same time zone and DST. So I would like to find all possible names for a specific time.
I also can't just use the GMT+1 offset as the daylight saving times will shift it during the summer and it will not switch for everyone on the same date.
I checked the TimeZone classes and couldn't find this.
In general:
Get the current time in UTC.
For each time zone, convert to that zone's local time.
If the local time is the time you're looking for, then you have a match.
Query for all users with matching time zones.
A few points:
You could do this either in MySQL, or in Java. You didn't really say which you were looking for.
You could optimize by limiting the time zones to test. You'd have to ahead-of-time determine the standard and daylight offsets of each zone, then use this as a filter.
When you check for "is it 8AM?" - you probably don't want to do an exact equality check, because it will only be 8:00:00.000 for one millisecond. Instead you probably want to test against a range of values, such as time >= 8:00 AND time < 9:00

Printing milliseconds of Java epoch returns 18000000?

In Joda, if I print
DateTime(GregorianChronology.getInstance())
.withYear(1970)
.withMonthOfYear(1)
.withDayOfMonth(1)
.withHourOfDay(0)
.withMinuteOfHour(0)
.withSecondOfMinute(0)
.withMillisOfSecond(0).getMillis();
I see 18000000 (this also happens to be 1/4th of MILLIS_PER_DAY, FWIW).
What I don't understand is that if the milliseconds represents the offset from the epoch which is defined as Jan-1970-01-01, then shouldn't the milliseconds be 0?
The epoch is Jan-1970-01-01 GMT. The instance you have, obviously has a different DateTimeZone. In fact it lookds like you're at GMT+5. (18000000 millis = 5 hours)
I believe the issue is related to the way Java dates include the time zone as part of there calculations.
For me, this means I'm +10 hours ahead of the epoc.
Try creating a Date/Time value that is set to 0 GMT.
The "epoch" is a specific and universal instant, a point in the universe time (like, say the moment in which the Apollo XI landed on the moon). This reference point can be represented differently in different countries (and a martian could also represent it with his own calendar). For example, for the people in England (GMT), that's the moment in which the hands of their clocks marked "00:00:00" and their (Gregorian) calendars marked "1/1/1970"; but that's just an example.
The line
DateTime(GregorianChronology.getInstance()).withYear(1970).withMonthOfYear(1)
.withDayOfMonth(1).withHourOfDay(0).withMinuteOfHour(0)
.withSecondOfMinute(0).withMillisOfSecond(0)
gives you the instant in which the clocks and the calendars in your country marked "00:00:00 1970-01-01". That's, in general, a different instant.

How to determine if a timestamp is within working hours?

Given a any unix timestamp (i.e. 1306396801) which translates to 26.05.2011 08:00:01, how can I determine if this is within a given timeframe (i.e. 08:00:00 and 16:00:00)?
This needs to work for any day. I just want to know if this timestamp is within the given time-interval, on any future (or past) day, the date is unimportant. I don't care if it is on the 25th or 26th, as long as it is between 08:00 and 16:00.
I am on the lookout for a java solution, but any pseudo code that works will be ok, I'll just convert it.
My attempts so far has been converting it to a java Calendar, and reading out the hour/min/sec values and comparing those, but that just opened up a big can of worms. If the time interval I want it between is 16.30, I can't just check for tsHour > frameStartHour && tsMin > frameStartMin as this will discard any timestamps that got a minute part > 30.
Thank you for looking at this :)
To clarify.
I am only using and referring to UTC time, my timestamp is in UTC, and the range I want it within is in UTC.
I think I understand what you want. You want to test for any day, if it's between 8am and 4pm UTC. Take the timestamp mod 24*3600. This will give you the number of seconds elapsed in the day. Then you just compare that it's between 8*3600 and 16*3600. If you need to deal with timezones, things get more complicated.
Given your timestamp (in seconds) and the desired time zone, Jodatime gives you the hour which leads you to a simple integer range check.
new org.joda.time.DateTime(timestamp*1000L, zone).getHourOfDay()
With java.util.* its more difficult.
If I understood you correctly, you only need to normalize your dates to some common value. Create three instances of Calendar - one with your time, but day, month, and year set to zero, and two with start and end of your timeframe, other fields also zeroed. Then you can use Calendar.after() and Calendar.before() to see if the date is within the range.
Your unix timestamp is an absolute time. Your time frame is relative. You need some kind of time zone information in order to solve this problem. I just answered some of this for PostgreSQL a few minutes ago. Hopefully that article is of use.
Convert the beginning of your range to a unix timestamp, and the end of your range to a unix tmestamp, then it's a simple integer check.

Categories