We Keep Coding

How to converter a TIMESTAMP_MICROS_STRINGIFIER Parquet using Java [duplicate]

There has been changes in Java Date & Time API Since Java 9.
LocalDateTime now has microseconds precision.
Java 9 has a fresh implementation of java.time.Clock capable of capturing the current moment in resolution finer than milliseconds (three digits of decimal fraction).
We get the time in microseconds from our backend service.
System.currentTimeMillis > 1565245051795 > 2019-08-08T06:17:31.795
Service.getTime > 1565245051795306 > 2019-08-08T06:17:31.795306
In order to construct a LocalDateTime to be used in our application, we do
long timeMicros = service.getTime();
long timeMillis = timeMicros / 1000;
LocalDateTime ldt = Instant.ofEpochMilli(timeMillis).atZone(ZoneId.systemDefault()).toLocalDateTime();
For querying the service we need time microseconds again, then we do
long timeMillis = dateTime.atZone(ZoneId.systemDefault()).toInstant().toEpochMilli();
long timeMicros = timeMillis * 1000;
The problem is we do not get back the time microseconds precision.
Is it possible to create an Instant with microsecond precision?
We are now using Java 11. I noticed this change when one of our JUnit tests failed because of the increased microsecond precision.
For the JUnit test I found a workaround:
private static final LocalDateTime START = LocalDateTime.now().truncatedTo(ChronoUnit.MILLIS);
I'm not sure if this is a workaround or an actual solution, but adding the last three microseconds digits from the timestamp seems to work.
long micros = 306L; //TODO get the last three digits from the timeMicros
ldt.plus(micros, ChronoUnit.MICROS));

long timeMicros = 1_565_245_051_795_306L;
Instant i = Instant.EPOCH.plus(timeMicros, ChronoUnit.MICROS);
System.out.println(i);
Output is:
2019-08-08T06:17:31.795306Z
Edit: Rather than dividing and multiplying to convert microseconds to milliseconds and/or seconds I preferred to use the built-in support for microseconds. Also when explicitly adding them to the epoch feels a little hand-held.
You already know how to convert Instant to LocalDateTime, you’ve shown it in the question, so I am not repeating that.
Edit:
Do you have a solution to get the timeMicros back from the Instant?
There are a couple of options. This way the calculation is not so complicated, so I might do:
long microsBack = TimeUnit.SECONDS.toMicros(i.getEpochSecond())
+ TimeUnit.NANOSECONDS.toMicros(i.getNano());
System.out.println(microsBack);
1565245051795306
To be more in style with the first conversion you may prefer the slightly shorter:
long microsBack = ChronoUnit.MICROS.between(Instant.EPOCH, i);
Edit: Possibly nit-picking, but also to avoid anyone misunderstanding: LocalDateTime has had nanosecond precision always. Only the now method had millisecond precision on Java 8. I read somewhere that from Java 9 the precision varies with the platform, but you are right, microsecond precision seems typical.

Java millis time with hourly resolution

How do I get java time millis in UTC ignoring the minutes and seconds.
For instance :
If it is October 10 2019, 1:10:59 AM , it should get the Time or millis for
October 10 2019, 1 AM.

Summary:
Instant
.now()
.truncatedTo(
ChronoUnit.HOURS
)
.toEpochMilli()
1570600800000
java.time, the modern Java date and time API has got exactly the method you need: many of the classes have a truncatedTo method for needs like yours.
Instant now = Instant.now();
System.out.println("Rough milliseconds: " + now.toEpochMilli());
Instant currentWholeHour = now.truncatedTo(ChronoUnit.HOURS);
System.out.println("Milliseconds ignoring minutes and seconds: "
+ currentWholeHour.toEpochMilli());
When running this snippet just now the output was:
Rough milliseconds: 1570604053787
Milliseconds ignoring minutes and seconds: 1570600800000
I know very well that the first line is what you asked not to have. I only included it for you to see the difference.
The truncation happens in UTC. If you are in a time zone whose offset is not a whole number of hours from UTC, the results may not be as you had expected. Examples of such time zones include Asia/Kathmandu, America/St_Johns some of the year also Australia/Lord_Howe.
Link: Oracle tutorial: Date Time

You can use LocalDate#atTime:
LocalDate.now().atTime(LocalDateTime.now().getHour(), 0, 0);
This will give you current date with hour and minutes and seconds set to 0.
And to get milliseconds in UTC:
LocalDate.now().atTime(LocalDateTime.now().getHour(), 0, 0).toInstant(ZoneOffset.UTC).toEpochMilli();
Jon Skeet notices, that calling now might give unexpected results in corner cases. To be sure, we can call it once and then convert it to LocalDate with mentioned solution:
var currentTime = LocalDateTime.now();
var currentDate = currentTime.toLocalDate();
Or the other way around - get LocalDate first and use LocalDate#atStartOfDay.

Given that you're interested in UTC milliseconds, and there are a whole number of milliseconds per hour, you can do this with simple arithmetic. For most calendrical computations I really wouldn't recommend that, but in this case I think it's the simplest approach. Something like this:
private static final long MILLISECONDS_PER_HOUR = TimeUnit.HOURS.toMillis(1);
// Injecting a clock makes the method testable. You can use Clock.systemUTC()
// for the system clock.
public static long truncateMillisToHour(Clock clock) {
long millisSinceEpoch = clock.millis();
// Truncate to the nearest hour
long hoursSinceEpoch = millisSinceEpoch / MILLISECONDS_PER_HOUR;
// Then multiply up again
return hoursSinceEpoch * MILLISECONDS_PER_HOUR;
}
Note that if the clock is for before the epoch, this will round up to the nearest hour, but if you're taking the genuine "current time" then that's unlikely to be a problem.
(I wrote this answer before seeing Ole V.V.'s answer with truncatedTo, which is a very nice approach.)

Writing and reading LocalDateTime from file most efficiently

What is the fastest way to write LocalDateTime instance to file and then read it from file and convert it back to LocalDateTime object?
I used to save milliseconds and then convert it to Date object. It looked pretty fast but now I am dealing with Java 8's LocalDateTime and it's not clear what would be most efficient way to save and retrieve it from file.
I don't think using DateTimeFormater is a good idea as it requires more resources to convert it to String and then parse the String.
Time zone is not relevant.

If you want to save the milliseconds and timezone is not important, you can use the java.time.Instant class - with only the LocalDateTime there's no way to get the milliseconds, because this class has no timezone/offset information.
// get the current date
Instant instant = Instant.now();
// get milliseconds (equivalent to java.util.Date.getTime())
long millis = instant.toEpochMilli();
// get Instant from milliseconds
Instant instant = Instant.ofEpochMilli(millis);
If you have a LocalDateTime, though, you can easily convert it to an Instant:
LocalDateTime d = LocalDateTime.now();
Instant instant = d.atOffset(ZoneOffset.UTC).toInstant();
This code obviously assumes that the values in the LocalDateTime correspond to an UTC date and time. To convert the Instant back to a LocalDateTime:
LocalDateTime d = LocalDateTime.ofInstant(instant, ZoneOffset.UTC);
PS: have you measured your system's performance to know if "fast" is really an issue? Anyway, I'm doing things the "standard" way (the most straightforward way based on what the API offers), is that what you want?
Maybe you can think that creating an Instant as an "intermediary" object makes things less "fast" (but you'll need to measure that anyway). If that's the case, you can get the millis from the LocalDateTime directly (assuming that it corresponds to a date and time in UTC):
// get the current date
LocalDateTime d = LocalDateTime.now();
// get milliseconds value
long millis = d.toEpochSecond(ZoneOffset.UTC) * 1000 + d.get(ChronoField.MILLI_OF_SECOND);
// get LocalDateTime from millis
LocalDateTime d = LocalDateTime.ofEpochSecond(millis / 1000, (int) (millis % 1000) * 1000000, ZoneOffset.UTC);
It's important to note that java.time classes have nanoseconds precision, so getting the milliseconds makes you lose this precision.
If you don't want to lose the nanoseconds precision and don't necessarily need to work with the millis value, you can store 2 different numbers (epoch day and nano of day):
// get the current date
LocalDateTime d = LocalDateTime.now();
// get values from LocalDateTime
long epochDay = d.toLocalDate().toEpochDay();
long nanoOfDay = d.toLocalTime().toNanoOfDay();
// save both values to file
// retrieve the LocalDateTime from the values
LocalDateTime d = LocalDateTime.of(LocalDate.ofEpochDay(epochDay), LocalTime.ofNanoOfDay(nanoOfDay));
This doesn't require the conversion to UTC, but it requires 2 numbers instead of one. You might think that creating a LocalDate and a LocalTime makes things slower, but these 2 objects are always created internally by LocalDateTime (in all cases).
Not sure, though, if all this math is "faster" enough than using an Instant. It's a matter of testing to see which one is best for your case.
But for me, the most "efficient" in terms of clarity and code-easy-to-maintain is using an Instant (or using the last approach of epoch day and nano of day). And unless you're dealing with millions of records, I'm not sure if that'll really be a performance issue.
I've made a simple test (run each case above 10 million times), and the last approach (using epoch day and nano of day) seems to be the fastest. But the difference is less than 1 second. Only by running 2 billion times I've got a 20-seconds difference, so if you're dealing with this many records, maybe it's worth it.
Regarding other resources (memory usage, CPU, I/O), I didn't check. But anyway, performance issues are very specific to each environment: depending on how the system is designed, how the system's parts/modules/components interact with each other, and many other factors, you can have different bottlenecks in each situation.
In the end, you'll have to test each approach and see which one performs best in your system. Or you can conclude that it doesn't make a significant difference (and for cases with less than a couple of millions records, maybe it doesn't - but you'll only know after benchmarking it).

Compare LDAP date to epoch

I'm trying to calculate LDAP accountExpires.
The given value is LDAP date - nanoseconds since 01/01/1601 00:00.
What is the best way to test if it is indeed after new Date()?

The best way probably depends on your precision requirements. I suggest
private static final Instant ldapEpoch = LocalDateTime.of(1601, Month.JANUARY, 1, 0, 0)
.atOffset(ZoneOffset.UTC)
.toInstant();
and then
long ldapTime = 131_428_662_140_000_000L;
Instant convertedTime = ldapEpoch.plusMillis( ldapTime / 10_000L );
System.out.println(convertedTime.isAfter(Instant.now()));
With my example LDAP time value this produces an Instant of 2017-06-25T12:10:14Z and prints false because the time is not after the current time.
Since you mentioned new Date() in the question, I assumed that the precision of Date would suffice for you, that is, milliseconds. I would really have loved to do ldapEpoch.plusNanos(ldapTime * 100) to keep the full precision, but this overflows the Java long data type and therefore gives an incorrect result. If you need the full precision, … Edit: as suggested by Basil Bourque in a comment, slice off the fractional second, work in whole seconds, then add back your fractional second:
Instant convertedTime = ldapEpoch.plusSeconds( ldapTime / 10_000_000L )
.plusNanos( ldapTime % 10_000_000L * 100L );
(The way I had first presented works too, gives the same result; but the edited version may be more natural to readers who know the Java date & time API (and may also perform a slight bit better, but that’s hardly critical).)
Why I wanted to multiply by 100? The LDAP, Active Directory & Filetime Timestamp Converter I found says “The timestamp is the number of 100-nanoseconds intervals (1 nanosecond = one billionth of a second) since Jan 1, 1601 UTC.”
Beware that in 1601 not everyone agreed about calendars, so January 1 that year is ambiguous. Most computer software assumes the Gregorian calendar, so I guess the definition of LDAP time does too, it’s not something I know.

Current time in microseconds in java

On a Unix system, is there a way to get a timestamp with microsecond level accuracy in Java? Something like C's gettimeofday function.

No, Java doesn't have that ability.
It does have System.nanoTime(), but that just gives an offset from some previously known time. So whilst you can't take the absolute number from this, you can use it to measure nanosecond (or higher) precision.
Note that the JavaDoc says that whilst this provides nanosecond precision, that doesn't mean nanosecond accuracy. So take some suitably large modulus of the return value.

tl;dr
Java 9 and later: Up to nanoseconds resolution when capturing the current moment. That’s 9 digits of decimal fraction.
Instant.now()
2017-12-23T12:34:56.123456789Z
To limit to microseconds, truncate.
Instant // Represent a moment in UTC.
.now() // Capture the current moment. Returns a `Instant` object.
.truncatedTo( // Lop off the finer part of this moment.
ChronoUnit.MICROS // Granularity to which we are truncating.
) // Returns another `Instant` object rather than changing the original, per the immutable objects pattern.
2017-12-23T12:34:56.123456Z
In practice, you will see only microseconds captured with .now as contemporary conventional computer hardware clocks are not accurate in nanoseconds.
Details
The other Answers are somewhat outdated as of Java 8.
java.time
Java 8 and later comes with the java.time framework. These new classes supplant the flawed troublesome date-time classes shipped with the earliest versions of Java such as java.util.Date/.Calendar and java.text.SimpleDateFormat. The framework is defined by JSR 310, inspired by Joda-Time, extended by the ThreeTen-Extra project.
The classes in java.time resolve to nanoseconds, much finer than the milliseconds used by both the old date-time classes and by Joda-Time. And finer than the microseconds asked in the Question.
Clock Implementation
While the java.time classes support data representing values in nanoseconds, the classes do not yet generate values in nanoseconds. The now() methods use the same old clock implementation as the old date-time classes, System.currentTimeMillis(). We have the new Clock interface in java.time but the implementation for that interface is the same old milliseconds clock.
So you could format the textual representation of the result of ZonedDateTime.now( ZoneId.of( "America/Montreal" ) ) to see nine digits of a fractional second but only the first three digits will have numbers like this:
2017-12-23T12:34:56.789000000Z
New Clock In Java 9
The OpenJDK and Oracle implementations of Java 9 have a new default Clock implementation with finer granularity, up to the full nanosecond capability of the java.time classes.
See the OpenJDK issue, Increase the precision of the implementation of java.time.Clock.systemUTC(). That issue has been successfully implemented.
2017-12-23T12:34:56.123456789Z
On a MacBook Pro (Retina, 15-inch, Late 2013) with macOS Sierra, I get the current moment in microseconds (up to six digits of decimal fraction).
2017-12-23T12:34:56.123456Z
Hardware Clock
Remember that even with a new finer Clock implementation, your results may vary by computer. Java depends on the underlying computer hardware’s clock to know the current moment.
The resolution of the hardware clocks vary widely. For example, if a particular computer’s hardware clock supports only microseconds granularity, any generated date-time values will have only six digits of fractional second with the last three digits being zeros.
The accuracy of the hardware clocks vary widely. Just because a clock generates a value with several digits of decimal fraction of a second, those digits may be inaccurate, just approximations, adrift from actual time as might be read from an atomic clock. In other words, just because you see a bunch of digits to the right of the decimal mark does not mean you can trust the elapsed time between such readings to be true to that minute degree.

You can use System.nanoTime():
long start = System.nanoTime();
// do stuff
long end = System.nanoTime();
long microseconds = (end - start) / 1000;
to get time in nanoseconds but it is a strictly relative measure. It has no absolute meaning. It is only useful for comparing to other nano times to measure how long something took to do.

As other posters already indicated; your system clock is probably not synchronized up to microseconds to actual world time. Nonetheless are microsecond precision timestamps useful as a hybrid for both indicating current wall time, and measuring/profiling the duration of things.
I label all events/messages written to a log files using timestamps like "2012-10-21 19:13:45.267128". These convey both when it happened ("wall" time), and can also be used to measure the duration between this and the next event in the log file (relative difference in microseconds).
To achieve this, you need to link System.currentTimeMillis() with System.nanoTime() and work exclusively with System.nanoTime() from that moment forward. Example code:
/**
* Class to generate timestamps with microsecond precision
* For example: MicroTimestamp.INSTANCE.get() = "2012-10-21 19:13:45.267128"
*/
public enum MicroTimestamp
{ INSTANCE ;
private long startDate ;
private long startNanoseconds ;
private SimpleDateFormat dateFormat ;
private MicroTimestamp()
{ this.startDate = System.currentTimeMillis() ;
this.startNanoseconds = System.nanoTime() ;
this.dateFormat = new SimpleDateFormat("yyyy-MM-dd HH:mm:ss.SSS") ;
}
public String get()
{ long microSeconds = (System.nanoTime() - this.startNanoseconds) / 1000 ;
long date = this.startDate + (microSeconds/1000) ;
return this.dateFormat.format(date) + String.format("%03d", microSeconds % 1000) ;
}
}

You could maybe create a component that determines the offset between System.nanoTime() and System.currentTimeMillis() and effectively get nanoseconds since epoch.
public class TimerImpl implements Timer {
private final long offset;
private static long calculateOffset() {
final long nano = System.nanoTime();
final long nanoFromMilli = System.currentTimeMillis() * 1_000_000;
return nanoFromMilli - nano;
}
public TimerImpl() {
final int count = 500;
BigDecimal offsetSum = BigDecimal.ZERO;
for (int i = 0; i < count; i++) {
offsetSum = offsetSum.add(BigDecimal.valueOf(calculateOffset()));
}
offset = (offsetSum.divide(BigDecimal.valueOf(count))).longValue();
}
public long nowNano() {
return offset + System.nanoTime();
}
public long nowMicro() {
return (offset + System.nanoTime()) / 1000;
}
public long nowMilli() {
return System.currentTimeMillis();
}
}
Following test produces fairly good results on my machine.
final Timer timer = new TimerImpl();
while (true) {
System.out.println(timer.nowNano());
System.out.println(timer.nowMilli());
}
The difference seems to oscillate in range of +-3ms. I guess one could tweak the offset calculation a bit more.
1495065607202174413
1495065607203
1495065607202177574
1495065607203
...
1495065607372205730
1495065607370
1495065607372208890
1495065607370
...

Use Instant to compute microseconds since Epoch:
val instant = Instant.now();
val currentTimeMicros = instant.getEpochSecond() * 1000_000 + instant.getNano() / 1000;

a "quick and dirty" solution that I eventually went with:
TimeUnit.NANOSECONDS.toMicros(System.nanoTime());
UPDATE:
I originally went with System.nanoTime but then I found out it should only be used for elapsed time, I eventually changed my code to work with milliseconds or at some places use:
TimeUnit.MILLISECONDS.toMicros(System.currentTimeMillis());
but this will just add zeros at the end of the value (micros = millis * 1000)
Left this answer here as a "warning sign" in case someone else thinks of nanoTime :)

If you're interested in Linux:
If you fish out the source code to "currentTimeMillis()", you'll see that, on Linux, if you call this method, it gets a microsecond time back. However Java then truncates the microseconds and hands you back milliseconds. This is partly because Java has to be cross platform so providing methods specifically for Linux was a big no-no back in the day (remember that cruddy soft link support from 1.6 backwards?!). It's also because, whilst you clock can give you back microseconds in Linux, that doesn't necessarily mean it'll be good for checking the time. At microsecond levels, you need to know that NTP is not realigning your time and that your clock has not drifted too much during method calls.
This means, in theory, on Linux, you could write a JNI wrapper that is the same as the one in the System package, but not truncate the microseconds.

Java support microseconds through TimeUnit enum.
Here is the java doc:
Enum TimeUnit
You can get microseconds in java by this way:
long microsenconds = TimeUnit.MILLISECONDS.toMicros(System.currentTimeMillis());
You also can convert microseconds back to another time units, for example:
long seconds = TimeUnit.MICROSECONDS.toSeconds(microsenconds);

If you intend to use it for realtime system, perhaps java isnt the best choice to get the timestamp. But if youre going to use if for unique key, then Jason Smith's answer will do enough. But just in case, to anticipate 2 item end up getting the same timestamp (its possible if those 2 were processed almost simultaneously), you can loop until the last timestamp not equals with the current timestamp.
String timestamp = new String();
do {
timestamp = String.valueOf(MicroTimestamp.INSTANCE.get());
item.setTimestamp(timestamp);
} while(lasttimestamp.equals(timestamp));
lasttimestamp = item.getTimestamp();

LocalDateTime.now().truncatedTo(ChronoUnit.MICROS)

Here is an example of how to create an UnsignedLong current Timestamp:
UnsignedLong current = new UnsignedLong(new Timestamp(new Date().getTime()).getTime());