We Keep Coding

Timezone conversion in Grails leads to wrong dates

I'm about to deal with time zones in Grails (Java). Here Java 7 is used and Grails 2.3.7.
I have a WebApp where each user is assigned a timeZoneID. If a user enters a date, it only consists of day, month and year. I want to set the time automatically.
The date entered by the user (e.g. 01.10.2018, german format) should be saved in the DB (MySQL) in UTC format.
When the date is displayed to the user, it is formatted according to the user's time zone.
Many timeZoneIDs work fine with my code (Europe/Berlin, Hont_Kong, ....), but America/New_York for example doesn't and I don't understand why.
The code to parse and save a date is as follows:
//endDate is 31.10.2018
def format = messageService.getMessage(code: 'default.date.short.format')
//--> dd.MM.yyyy for DE and MM/dd/yy for EN
println("Use format: " + format)
SimpleDateFormat sdf = new SimpleDateFormat(format);
//set timezone (America/New_York)
sdf.setTimeZone(TimeZone.getTimeZone(user.timeZoneID))
//parse endDate
Date parsedEndDate = sdf.parse(endDate)
//create a calendar instance (e.g. America/New_York)
Calendar calendarEnd = Calendar.getInstance(TimeZone.getTimeZone(user.timeZoneID));
//set time
calendarEnd.setTime(parsedEndDate);
//set hour/minute automatically
calendarEnd.set(Calendar.HOUR_OF_DAY, 23)
calendarEnd.set(Calendar.MINUTE, 59)
//at this point it should be 31.10.2018, 23:59 (german format, timezone America/New_York)
//Convert to UTC before saving date in DB (MySQL)
calendarEnd.setTimeZone(TimeZone.getTimeZone('UTC'))
//save the date
def obj = new Foo(date:calendarEnd).save(flush:true)
The code inside my view (gsp) to display a date is as follows:
<g:formatDate
timeZone="${user.timeZoneID}"
date="${fooInstance?.calendarEnd}"
format="${message(code: 'default.date.format', default: 'MM/dd/yyyy, hh:mm a')}"/>
Inside the DB I get 2018-11-01 00:59:00
Inside my view (GSP) it results in 31.10.2018, 19:59, instead of 31.10.2018, 23:59
Thank you very much for your help.

The problem is in convert step:
//Convert to UTC before saving date in DB (MySQL)
calendarEnd.setTimeZone(TimeZone.getTimeZone('UTC'))
Because you are just changing the time zone so it is using the given time and date as if it is the UTC time zone.
Java 1.7 and before are somewhat unwieldy in regards to the Time API so a lot of people use Joda Time.
Otherwise you can use the advice from this question resulting in something like:
calendarEnd.add(Calendar.MILLISECOND, TimeZone.getTimeZone('UTC').getOffset(parsedEndDate.getTime())
This is not tested and could be wrong as the offset calculation might be diffrent

How to disable conversion to UTC when storing OffsetDateTime in PostgreSQL

Given this table:
CREATE TABLE a(
t TIMESTAMP WITH TIME ZONE
);
And this simple JDBC code snippet:
DriverManager.getConnection(
"jdbc:postgresql://localhost:5432/dbname", "user", "password"
).use { connection ->
val nowSomeTimeZone = OffsetDateTime.now(ZoneOffset.of("+4"))
connection.prepareStatement("insert into a(t) values (?)").use { insertStmt ->
insertStmt.setObject(1, nowSomeTimeZone)
insertStmt.executeUpdate()
}
connection.createStatement().use { stmt ->
stmt.executeQuery("select * from a").use { resultSet ->
resultSet.next()
val t = resultSet.getObject(1, OffsetDateTime::class.java)
println("$nowSomeTimeZone -> $t")
}
}
}
Somewhere inside the JDBC stack an automatic conversion from +04:00 to UTC must be happening, because this is the println output:
2018-08-30T10:35:33.594+04:00 -> 2018-08-30T06:35:33.594Z
What's even more weird, when I look into the table using the psql console client, it shows me the timestamp in yet another time zone (which is my local time zone):
$ psql -h localhost -U username
dbname=> select * from a;
t
----------------------------
2018-08-30 08:35:33.594+02
Why does this conversion happen, and how can I disable it?

Disabling the conversion is not possible, because the PostgreSQL server strips the time zone information and stores timestamps always in UTC, even when you are explicitly using the type TIMESTAMP WITH TIME ZONE.
Quoting the PostgreSQL documentation:
For timestamp with time zone, the internally stored value is always in UTC (Universal Coordinated Time, traditionally known as Greenwich Mean Time, GMT). An input value that has an explicit time zone specified is converted to UTC using the appropriate offset for that time zone. If no time zone is stated in the input string, then it is assumed to be in the time zone indicated by the system's TimeZone parameter, and is converted to UTC using the offset for the timezone zone.
Moreover, the documentation states:
We do not recommend using the type time with time zone (though it is supported by PostgreSQL for legacy applications and for compliance with the SQL standard). PostgreSQL assumes your local time zone for any type containing only date or time.
The weird behaviour described in the question is because
The JDBC driver always returns the timestamp in UTC
The psql console client converts the timestamp to the user's local time zone before displaying it, in this case German Time (+02:00)
Thanks to #RobbyCornelissen for the insights.

MySQL client vs server time zone

I'm running into an issue where MySQL stores different date time values than the client passes. The server runs in UTC and the client in a different time zone. Somehow MySQL seems to convert date time values between the client and server time zone even though the SQL types DATE, TIME and TIMESTAMP all have no time zone. No other database I tested so far has this behaviour.
The following code can be used to reproduce the issue. When the server runs in UTC the code only works when the client also runs in UTC.
try (Connection connection = this.dataSource.getConnection();
PreparedStatement preparedStatement = connection.prepareStatement(
"SELECT ? = DATE '1988-12-25', ? = TIME '15:09:02', ? = TIMESTAMP '1980-01-01 23:03:20'")) {
preparedStatement.setDate(1, java.sql.Date.valueOf("1988-12-25"));
preparedStatement.setTime(2, java.sql.Time.valueOf("15:09:02"));
preparedStatement.setTimestamp(3, java.sql.Timestamp.valueOf("1980-01-01 23:03:20"));
try (ResultSet resultSet = preparedStatement.executeQuery()) {
while (resultSet.next()) {
System.out.println(resultSet.getBoolean(1));
System.out.println(resultSet.getBoolean(2));
System.out.println(resultSet.getBoolean(3));
}
}
}
I'm using
MySQL 5.7.14
mysql-connector-java 6.0.5
Oracle Java 1.8.0_131
My JDBC URL is just jdbc:mysql://host:port/database
edit
My reasoning why time zones should not play a role here and no time zone conversion should happen is two fold. Firstly on the SQL level TIMESTAMP is an alias for TIMESTAMP WITHOUT TIME ZONE which strongly implies that unlike TIMESTAMP WITH TIME ZONE it values have no time zone. In other words values are not instants in time but rather in local date time values.
Secondly that java.sql.Timestamp is in JVM time zone is merely artefact of being a subclass of java.util.Date. (I am aware that java.util.Date has no time zone). The Javadoc of java.sql.Timestamp of makes it quite clear that relationship is only for implementation purposes.
I feel both of these assertions are confirmed by the fact that in Java SE 8 / JDBC 4.2 java.sql.Timestamp is mapped to java.time.LocalDateTime and not java.time.ZonedDateTime or java.time.OffsetDateTime.
edit 2
I do not understand why TIMESTAMP values are subject to time zone conversion. Unlike TIMESTAMP WITH TIME ZOONE These are "local" values and do not have an associated time zone and should therefore have no time zone conversion applied to them.

I agree with the comment that timezone handling under MySQL JDBC can be confusing, but in this case ...
MySQL stores different date time values than the client passes
... is not quite correct. It interprets and/or displays the string representation of the same datetime value in the context of the server timezone.
First you need to understand that java.sql.Timestamp#valueOf creates a Timestamp value in the local timezone under which the Java Virtual Machine is running. So, for my machine which is on "Mountain Time" in Canada (UTC-7 for Standard Time):
System.out.printf("Local (client) timezone is %s%n", TimeZone.getDefault().getID());
java.sql.Timestamp tStamp = java.sql.Timestamp.valueOf("1980-01-01 23:03:20");
SimpleDateFormat sdfLocal = new SimpleDateFormat("yyyy-MM-dd HH:mm:ss z");
SimpleDateFormat sdfUTC = new SimpleDateFormat("yyyy-MM-dd HH:mm:ss z");
sdfUTC.setCalendar(Calendar.getInstance(TimeZone.getTimeZone("UTC")));
System.out.println("tStamp = java.sql.Timestamp.valueOf(\"1980-01-01 23:03:20\")");
System.out.printf(" ... which is %s%n", sdfLocal.format(tStamp));
System.out.printf(" ... which is %s%n", sdfUTC.format(tStamp));
prints
Local (client) timezone is America/Edmonton
tStamp = java.sql.Timestamp.valueOf("1980-01-01 23:03:20")
... which is 1980-01-01 23:03:20 MST
... which is 1980-01-02 06:03:20 UTC
Now, when we pass that Timestamp value to a PreparedStatement and send it off to a MySQL server that is using the UTC timezone, the server interprets and displays string literal representations of that MySQL TIMESTAMP as UTC:
String connUrl = "jdbc:mysql://localhost/mydb";
try (Connection conn = DriverManager.getConnection(connUrl, myUid, myPwd)) {
String sql = "SELECT CAST((TIMESTAMP ?) AS CHAR) AS foo";
try (PreparedStatement ps = conn.prepareStatement(sql)) {
ps.setTimestamp(1, tStamp);
try (ResultSet rs = ps.executeQuery()) {
rs.next();
System.out.printf("String representation of TIMESTAMP value at server: %s%n",
rs.getString(1));
}
}
}
producing
String representation of TIMESTAMP value at server: 1980-01-02 06:03:20
If the MySQL server had been running under Toronto time (UTC-5 for Standard Time) the output would have been ...
String representation of TIMESTAMP value at server: 1980-01-02 01:03:20
... not because the MySQL TIMESTAMP value is different, but because the string representation of that value in the context of the MySQL server timezone is different.

I guess it is like Gord Thompson and spencer7593 stated, that the timezone of the server and that of the connection-string of the client differ and a conversion between timezones has to be made.
I also learned something while reading your edits, as in that the java.sql.Timestamp contains no timezone.
As spencer7593 stated: "The server is interpreting the literal in the context of the server timezone".
My two approaches here would be:
Connect to server (UTC) with your timezone in the connection-string(e.g. &serverTimezone=Europe/Zurich).
Alternatively pass an instance of an Calendar with your local timezone (Calendar.getInstance()) to PreparedStatement.setTimestamp.

Spring Data MongoDB - when saving LocaDate/LocalDateTime the value is being set to yesterday

I've got a problem with saving Java8 Date API values to MongoDB database. Whenever a new LocalDate instance (LocalDate.now()) is being saved, as a result we are getting value with yesterdays date with time set to 23:00 PM.
Example:
dt.getDate().toString()
is giving me "2017-03-17"
but when i look into the database i have a value like this:
"dt" : ISODate("2017-03-16T23:00:00.000Z")
My time zone is UTC+01:00

MongoDB saves the date in UTC time.
LocalTime is your wall clock time.
When you pass it to MongoDb, spring will convert the LocalTime to Instant(UTC time) by using your System Zone.
Consider
LocalDateTime localDateTime = LocalDateTime.parse("2017-03-17T00:00:00")
Something like this happpens
Instant instant = localDateTime.atZone(ZoneId.systemDefault()).toInstant();
This is where your local time is changed to UTC instant by applying offset (+01:00) from your local time.
Output(in UTC) : 2017-03-16T23:00:00.000Z

Retrieving UTC DATETIME field from MySQL in Java when server timezone is not UTC

I'm trying to write code to interoperate with a third-party-developed database using Java and MySQL. This database has a field that stores a time stamp in a DATETIME field as a UTC date. The timezone for the server on which both the database and client run is set to a non-UTC zone (Europe/London), so by default the timestamp is read back incorrectly as if it were a local time. I'm trying to write code to read it back as UTC.
I have read several similar questions here, but none of them have an answer that works for me:
MySQL - how to store time with correct timezone? (from Java)
How to store a java.util.Date into a MySQL timestamp field in the UTC/GMT timezone?
Date in UTC in mysql
How do I set the time zone of MySQL?
Unfortunately, I cannot change any server settings, so I have tried using the connection's "time_zone" variable to set the database server to use UTC and the optional Calendar parameter to ResultSet.getTimestamp to retrieve the date, but this has no effect on the result. Here is my code:
private static final Calendar UTCCALENDAR = Calendar.getInstance (TimeZone.getTimeZone (ZoneOffset.UTC));
public Date getDate ()
{
try (Connection c = dataSource.getConnection ();
PreparedStatement s = c
.prepareStatement ("select datefield from dbmail_datefield where physmessage_id=?"))
{
fixTimeZone (c);
s.setLong (1, getPhysId ());
try (ResultSet rs = s.executeQuery ())
{
if (!rs.next ()) return null;
return new Date (rs.getTimestamp(1,UTCCALENDAR).getTime ()); // do not use SQL timestamp object, as it fucks up comparisons!
}
}
catch (SQLException e)
{
throw new MailAccessException ("Error accessing dbmail database", e);
}
}
private void fixTimeZone (Connection c)
{
try (Statement s = c.createStatement ())
{
s.executeUpdate ("set time_zone='+00:00'");
}
catch (SQLException e)
{
throw new MailAccessException ("Unable to set SQL connection time zone to UTC", e);
}
}
The database field I'm trying to read has a value stored in it as follows:
mysql> select * from dbmail_datefield where physmessage_id=494539;
+----------------+--------+---------------------+
| physmessage_id | id | datefield |
+----------------+--------+---------------------+
| 494539 | 494520 | 2015-04-16 10:30:30 |
+----------------+--------+---------------------+
But unfortunately, the result comes out as BST not UTC:
java.lang.AssertionError: expected:<Thu Apr 16 11:30:30 BST 2015> but was:<Thu Apr 16 10:30:30 BST 2015>

Your client getDate() code looks correct as far as it goes. I think you also need to get the MySQL Connector/J JDBC driver to treat the dates stored in the table as UTC dates, to avoid a spurious time zone conversion. This means setting the effective server time zone, in addition to the client session time zone and Calendar used for JDBC getTimestamp calls as you're doing.
Take a look at the values you got in your failed assertion, and which direction the error is in:
expected:<Thu Apr 16 11:30:30 BST 2015> but was:<Thu Apr 16 10:30:30 BST 2015>
What you got back was 10:30 BST, which is 9:30 GMT. This is consistent with the database treating that 10:30 in the table as a BST value and spuriously converting it to GMT for you, before you parse it as a GMT date. That's the opposite direction of a GMT value being spuriously converted to BST.
This may be a JDBC-specific issue, because JDBC requires that time times be converted to the local zone. (Where the MySQL C API doesn't, probably because C's classic time types are not zone-aware the way Java's are.) And it needs to know what zone it's converting from, as well. The MySQL TIMESTAMP type is always stored as UTC. But that's not stated for the DATETIME type. I think that implies that MySQL is going to interpret DATETIME column values as being in the server's time zone. Which you mentioned as being set to BST, and that's consistent with the direction of the shift shown in your assertion error message.
The time_zone session variable you set is telling the MySQL server what your client machine's time zone is, but it doesn't affect what the server thinks its own time zone is. That can be overridden with the serverTimezone JDBC connection property. On your connection, set the serverTimezone to UTC, and make sure useLegacyDatetimeCode is off. (And look through the other zone-related properties if that doesn't work.) See if that gets your dates to come through as UTC with the same calendar field values as in the database.
Be aware that this is going to change the interpretation of other DATETIME values in your database: they're all going to look like UTC dates now (in the context of your JDBC connection). Whether that's correct is going to depend on how they were populated initially. While your client code will have the behavior you want, I don't know if this system as a whole can be made to behave fully consistently without setting the server's time zone to UTC at the server level. Basically, if it doesn't have its zone set to UTC, it's not fully configured for the behavior you want, and you're kludging around it.

Maybe you can use JodaTime as follows;
private static final Calendar UTCCALENDAR = Calendar.getInstance (TimeZone.getTimeZone (ZoneOffset .UTC));
public Date getDate ()
{
try (Connection c = dataSource.getConnection ();
PreparedStatement s = c
.prepareStatement ("select datefield from dbmail_datefield where physmessage_id=?"))
{
s.setLong (1, getPhysId ());
try (ResultSet rs = s.executeQuery ())
{
if (!rs.next ()) return null;
DateTime dt = new LocalDateTime(rs.getTimestamp(1,UTCCALENDAR).getTime ()).toDateTime(DateTimeZone.forID("Europe/London"));
return dt.toDate(); }
}
catch (SQLException e)
{
throw new MailAccessException ("Error accessing dbmail database", e);
}
}
EDIT:
java.util.Date is not TimeZone agnostic. The method toDateTime takes care of TimeZone and DST so you don't care about it
The following code:
public static void main(String[] args) {
// 29/March/2015 1:05 UTC
DateTime now = new DateTime(2015, 3,29,1,5,DateTimeZone.UTC);
// Pre DST 29/March/2015 0:30 UTC
DateTime preDst = new DateTime(2015, 3,29,0,30,DateTimeZone.UTC);
System.out.println("1:05 UTC:"+now);
System.out.println("0:30 UTC:"+preDst);
DateTimeZone europeDTZ = DateTimeZone.forID("Europe/London");
DateTime europeLondon = now.toDateTime(europeDTZ);
System.out.println("1:05 UTC as Europe/London:"+europeLondon);
DateTime europeLondonPreDst = preDst.toDateTime(europeDTZ);
System.out.println("0:30 UTC as Europe/London:"+europeLondonPreDst);
}
Will print:
1:05 UTC:2015-03-29T01:05:00.000Z
0:30 UTC:2015-03-29T00:30:00.000Z
1:05 UTC as Europe/London:2015-03-29T02:05:00.000+01:00
0:30 UTC as Europe/London:2015-03-29T00:30:00.000Z
If you can see JodaTime takes care of DST.

Your best bet, in my view, is to tell MySQL to use GMT and handle all local time issues in your application code, not your database. The values in the database would always be GMT, full stop, which is unambiguous. As you say, with daylight savings time (summer time) adjustments, you can end up with the same value in your database for what is, to us humans, two different times.
This also makes the database portable. If you move to North America and start using MySQL set to (say) Central time, all of a sudden the values in your database seem to have moved several hours. I had that issue with a database I inherited which was using the server's local time, when I moved it from the east coast of the U.S. to the west coast, not having thought to check whether MySQL was slaved to the machine's zone...
long t = 1351382400000; // the timestamp in UTC
String insert = "INSERT INTO my_table (timestamp) VALUES (?)";
PreparedStatement stmt = db.prepareStatement(insert);
java.sql.Timestamp date = new Timestamp(t);
stmt.setTimestamp(1, date);
stmt.executeUpdate();
.....
TimeZone timezone = TimeZone.getTimeZone("MyTimeZoneId");
Calendar cal = java.util.Calendar.getInstance(timezone);
String select = "SELECT timestamp FROM my_table";
// some code omitted....
ResultSet rs = stmt.executeQuery();
while (rs.next()) {
java.sql.Timestamp ts = rs.getTimestamp(1);
cal.setTimeInMillis(ts.getTime());
System.out.println("date in db: " + cal.getTime());
}

If you want to use timezone you can read column as UTC.
ZonedDateTime zdt = ZonedDateTime.of(rs.getTimestamp(1).toLocalDateTime(), ZoneOffset.UTC);
Next you can change to whatever timezone you want using:
zdt = zdt.withZoneSameInstant(ZoneId.of(
TARGET_ZONE));
If you want only to read Date and do not care about zones at all use only:
LocalDateTime ldt = rs.getTimestamp(1).toLocalDateTime()
You will obtain LocalDateTime without timezone.
If you have to return java.util.Date use:
Date.from(ldt.atZone(ZoneOffset.UTC).toInstant());

Don't think about converting or adapting time zone. Don't think about the TZ the mysql uses to store your timestamps or anythink like that. Those things are already handled.
There are three things that you must handle: INPUT, OUTPUT and bugs.
INPUT
When a user enters a date (in a form) without an explicit time zone you have to know what TZ did he intend to use. You can use a SimpleDateFormat object with time zone set to solve this. You don't have to convert the input date, you have to 'interpret' it correctly. Once you have a correctly interpreted Date or timestamp you are done with input.
Input is not only user input, includes configuration files too.
OUTPUT
The same here. Forget about what TZ have your Date objects and timestamps have none, they are just milliseconds since epoch. You have to format your dates to the TZ the user expects so he understand them.
Bugs
You may have bugs in your code related to TZ, but libraries may have them too!!
I noticed mysql java driver failed to communicate the client timezone to the server.
This command s.executeUpdate ("set time_zone='+xx:yy'"); is the workaround but you are using it wrong. You have to tell the server with it the TZ the client is using, before both inserting and querying. The variable is stored in the session. Maybe you may automatize it on your connection pool config.
This is needed so the server know what TZ the client need to use to read or write. This is not dependent on server TZ. It does not mean "store this date in UTC", it does mean "this date I am giving to you is UTC" and "Send me result sets in UTC". No matter you are using Date class with it's internal TZ, the driver screws it up, you would need to set that session variable.
By default it assumes client TZ is the same as server TZ so you shouldn't need to worry about it as you said they are the same.