Recently I am being challenged by quite an "easy" problem. Suppose that there is sentences (saved in a String), and I need to find out if there is any date in this String. The challenges is that the date can be in a lot of different formats. Some examples are shown in the list:
June 12, 1956
London, 21st October 2014
13 October 1999
01/11/2003
Worth mentioning that these are contained in one string. So as an example it can be like:
String s = "This event took place on 13 October 1999.";
My question in this case would be how can I detect that there is a date in this string. My first approach was to search for the word "event", and then try to localize the date. But with more and more possible formats of the date this solution is not very beautiful. The second solution that I tried is to create a list for months and search. This had good results but still misses the cases when the date is expressed all in digits.
One solution which I have not tried till now is to design regular expressions and try to find a match in the string. Not sure how much this solution might decrease the performance.
What could be a good solution that I should probably consider? Did anybody face a similar problem before and what solutions did you find?
One thing is for sure that there are no time, so the only interesting part is the date.
Using the natty.joestelmach.com library
Natty is a natural language date parser written in Java. Given a date expression, natty will apply standard language recognition and translation techniques to produce a list of corresponding dates with optional parse and syntax information.
import com.joestelmach.natty.*;
List<Date> dates =new Parser().parse("Start date 11/30/2013 , end date Friday, Sept. 7, 2013").get(0).getDates();
System.out.println(dates.get(0));
System.out.println(dates.get(1));
//output:
//Sat Nov 30 11:14:30 BDT 2013
//Sat Sep 07 11:14:30 BDT 2013
You are after Named Entity Recognition. I'd start with Stanford NLP. The 7 class model includes date, but the online demo struggles and misses the "13". :(
Natty mentioned above gives a better answer.
If it's only one String you could use the Regular Expression as you mentioned. Having to find the different date format expressions. Here are some examples:
Regular Expressions - dates
In case it's a document or a big text, you will need a parser. You could use a Lexical analysis approach.
Depending on the project using an external library as mentioned in some answers might be a good idea. Sometimes it's not an option.
I've done this before with good precision and recall. You'll need GATE and its ANNIE plugin.
Use GATE UI tool to create a .GAPP file that will contain your
processing resources.
Use the .GAPP file to use the extracted Date
annotation set.
Step 2 can be done as follows:
Corpus corpus = Factory.newCorpus("Gate Corpus");
Document gateDoc = Factory.newDocument("This event took place on 13 October 1999.");
corpus.add(gateDoc);
File pluginsHome = Gate.getPluginsHome();
File ANNIEPlugin = new File(pluginsHome, "ANNIE");
File AnnieGapp = new File(ANNIEPlugin, "Test.gapp");
AnnieController =(CorpusController) PersistenceManager.loadObjectFromFile(AnnieGapp);
AnnieController.setCorpus(corpus);
AnnieController.execute();
Later you can see the extracted annotations like this:
AnnotationSetImpl ann = (AnnotationSetImpl) gateDoc.getAnnotations();
System.out.println("Found annotations of the following types: "+ gateDoc.getAnnotations().getAllTypes());
I'm sure you can do it easily with the inbuilt annotation set Date. It is also very enhancable.
To enhance the annotation set Date create a lenient annotation rule in JAPE say 'DateEnhanced' from inbuilt ANNIE annotation Date to include certain kinds of dates like "9/11" and use a Chaining of Java regex on R.H.S. of the 'DateEnhanced' annotations JAPE RULE, to filter some unwanted outputs (if any).
Related
In Java the Locale defines things that are related how people want to see things (like currency formats, the name of the months and when a week starts).
When parsing the name of a Month (with a DateTimeFormatter) it starts to become tricky.
If you use Locale.US or Locale.ENGLISH then September has the short form Sep.
If you use Locale.UK then September also has the short form Sep in Java 11 ... but when you try Java 17 then it has Sept (because of changes at the Unicode CLDR end for which I asked if this was correct).
The effect is that my tests started failing when trying to build with Java 17.
The reason my current code uses Locale.UK instead of Locale.ENGLISH is because in Java Locale.ENGLISH is actually not just English but also the non-ISO American way of defining a week (they use Sunday as the first day of the week). I want to have it the ISO way.
Simply:
WeekFields.ISO = WeekFields.of(Locale.UK) = WeekFields[MONDAY,4]
WeekFields.of(Locale.ENGLISH) = WeekFields.of(Locale.US) = WeekFields[SUNDAY,1]
So starting with Java 17 I have not yet been able to find a built in Locale that works correctly.
In my mind I have to take either the Locale.ENGLISH and change the WeekFields or take the Locale.UK and change the shortname of the month September to what I need.
My question is how do I do this (in Java 17)?
Or is there a better way to fix this?
Update 1:
I already got feedback from the people at Unicode indicating that the change for en_GB to use Sept instead of Sep is a bugfix because that is the way it should be abbreviated in the UK.
So it seems I will need not just a parser that accepts "Sep" but one that will accept a mix of "Sept" and "Sep" for English.
Update 2:
I have tweaked my code that in case of a parse exception it will try to change what is assumed to be the input ("Sep") into what the currently selected locate likes to have. This does not cover all cases, it covers enough cases for my specific situation.
For those interested: my commit.
I found a way of handling this by using SPI.
I'm documenting it here as a possibility that may work for others (it does not work for my context).
As an experiment I created a class:
package nl.basjes.parse.httpdlog.dissectors.locale;
import java.util.Locale;
import java.util.spi.CalendarDataProvider;
import static java.util.Calendar.MONDAY;
public class CalendarDataProviderISO8601 extends CalendarDataProvider {
public static final Locale ENGLISH_ISO = new Locale("en", "", "ISO");
#Override
public int getFirstDayOfWeek(Locale locale) {
return MONDAY;
}
#Override
public int getMinimalDaysInFirstWeek(Locale locale) {
return 4;
}
#Override
public Locale[] getAvailableLocales() {
return new Locale[]{ENGLISH_ISO};
}
}
and a file ./src/main/resources/META-INF/services/java.util.spi.CalendarDataProvider with
nl.basjes.parse.httpdlog.dissectors.locale.CalendarDataProviderISO8601
Because this is just a variant over the regionless "English" it will take everything from "English" and put the above class over it.
Although this works I cannot use it.
The problem is that although http://openjdk.java.net/jeps/252 describes The default lookup order will be CLDR, COMPAT, SPI, the current reality is that the SPI has been removed from this list in this change because of deprecating the Extension Mechanism.
So to use this construct the class must be in the classpath at startup and the commandline option -Djava.locale.providers=CLDR,COMPAT,SPI must be passed to the JVM.
Given that my library ( https://github.com/nielsbasjes/logparser/ ) is also used in situations (like Apache Flink/Beam/Drill/Pig) where classes are shipped in a more dynamic way (serialized and transported to an already running JVM) to multiple machines this construct cannot be used.
I currently do not know of a dynamic way of doing something like this in Java.
I'm using the en_GB locale, but a similar issue may also affect other en_XX locales.
Under Java 15 the following code works:
LocalDate.parse("10-Sep-17", DateTimeFormatter.ofPattern("dd-MMM-yy", Locale.UK));
Under Java 16 it gives: DateTimeParseException: Text '10-Sep-17' could not be parsed at index 3
After spending a long time in the debugger I have traced this to this commit: 8251317: Support for CLDR version 38
This commit changes the abbreviated form of September in make/data/cldr/common/main/en_GB.xml from Sep to Sept for both the context-sensitive and standalone forms. None of the other months are touched, remaining as 3 characters.
I have verified that this is indeed a genuine change between CLDR versions 37 and 38, although I'm not sure when we Brits switched to using 4 letters for our 3-letter abbreviation for September...
Now this is annoying, as it has broken my datafile processing (although I suspect I can fix it by specifying Locale.ENGLISH rather than using the default locale in my code), but I can't decide if it counts as a bug that has been introduced that breaks my reliable 3-character-month match pattern, or whether this is actually meant to be a feature.
The JavaDoc says:
Text: The text style is determined based on the number of pattern letters used. Less than 4 pattern letters will use the short form. ...
and later:
Number/Text: If the count of pattern letters is 3 or greater, use the Text rules above. Otherwise use the Number rules above.
My bad for never having read this carefully enough to spot that textual values are handled differently to numbers, where the number of letters in your pattern sets the width. But this leaves me wondering how you are supposed to specify a fixed number of characters when you output a month, and equally why it can't be permissive and accept the three-character form when parsing rather than throw an exception?
At the end of the day this still feels like a regression to me. My code that has worked reliably for years parsing dates with 3-character months in now, with no warning, fails on all dates in September. Am I wrong to think this feels incorrect?
I wanted to retrieve dates and other temporal entities from a set of Strings. Can this be done without parsing the string for dates in JAVA as most parsers deal with a limited scope of input patterns. But input is a manual entry which here and hence ambiguous.
Inputs can be like:
12th Sep |mid-March |12.September.2013
Sep 12th |12th September| 2013
Sept 13 |12th, September |12th,Feb,2013
I've gone through many answers on finding date in Java but most of them don't deal with such a huge scope of input patterns.
I've tried using SimpleDateFormat class and using some parse() functions to check if parse function breaks which mean its not a date. I've tried using regex but I'm not sure if it falls fit in this scenario. I've also used ClearNLP to annotate the dates but it doesn't give a reliable annotation set.
The closest approach to getting these values could be using a Chain of responsibility as mentioned below. Is there a library that has a set of patterns for date. I can use that maybe?
A clean and modular approach to this problem would be to use a chain,
every element of the chain tries to match the input string against a regex,
if the regex matches the input string than you can convert the input string to something that can feed a SimpleDateFormat to convert it to the data structure you prefer (Date? or a different temporal representation that better suits your needs) and return it, if the regexp doesn't matches the chain element just delegates to the next element in the chain.
The responsibility of every element of the chain is just to test the regex against the string, give a result or ask the next element of the chain to give it a try.
The chain can be created and composed easily without having to change the implementation of every element of the chain.
In the end the result is the same as in #KirkoR response, with a 'bit' (:D) more code but a modular approach. (I prefer the regex approach to the try/catch one)
Some reference: https://en.wikipedia.org/wiki/Chain-of-responsibility_pattern
You could just implement support for all the pattern possibilities you can think of, then document that ... OK, these are all patterns my module supports. You could then throw some RuntimeException for all the other possibilities.
Then ... in an iterative way you can keep running your module over the input data, and keep adding support for more date formats until it stops raising any RuntimeException.
I think that's the best you can do here if you want to keep it reasonably simple.
Yes! I've finally extracted all sorts of dates/temporal values that can be as generic as :
mid-March | Last Month | 9/11
To as specific as:
11/11/11 11:11:11
This finally happened because of awesome libraries from GATE and JAPE
I've created a more lenient annotation rule in JAPE say 'DateEnhanced' to include certain kinds of dates like "9/11 or 11TH, February- 2001" and used a Chaining of Java regex on R.H.S. of the 'DateEnhanced' annotations JAPE RULE, to filter some unwanted outputs.
I can recommend to you very nice implementation of your problem, unfortunetlly in polish: http://koziolekweb.pl/2015/04/15/throw-to-taki-inny-return/
You can use google translator:
https://translate.google.pl/translate?sl=pl&tl=en&js=y&prev=_t&hl=en&ie=UTF-8&u=http%3A%2F%2Fkoziolekweb.pl%2F2015%2F04%2F15%2Fthrow-to-taki-inny-return&edit-text=
The code there looks really nice:
private static Date convertStringToDate(String s) {
if (s == null || s.trim().isEmpty()) return null;
ArrayList<String> patterns = Lists.newArrayList(YYYY_MM_DD_T_HH_MM_SS_SSS,
YYYY_MM_DD_T_HH_MM_SS
, YYYY_MM_DD_T_HH_MM
, YYYY_MM_DD);
for (String pattern : patterns) {
try {
return new SimpleDateFormat(pattern).parse(s);
} catch (ParseException e) {
}
}
return new Date(Long.valueOf(s));
}
mark.util.DateParser dp = new DateParser();
ParsePositionEx parsePosition = new ParsePositionEx(0);
Date startDate = dp.parse("12.September.2013", parsePosition);
System.out.println(startDate);
output: Thu Sep 12 17:18:18 IST 2013
mark.util.Dateparser is a part of library which is used by DateNormalizer PR. So in Jape file, we have to just import it.
I'm looking for a list of common datetime formats used in logs (e.g. webserver, database, etc).
Even better would be a (java) library that can extract date and time from a given string ( < 10KB).
Does anyone know a good one?
this library is likely a good place to start: SimpleDateFormat
The docs contains the an introduction to the standard datetime format strings. But as #Olaf points out, you're going to need to specify what the format is beforehand or there is literally no way differentiate certain dates from one another.
Looks like what you'd want to do is construct a range of date formats that might match, apply all of them to a date string, then see which date is closest to Datetime.now().
Although this doesn't answer your question directly, but Java includes libraries for working with regular expressions. It would be pretty easy to write a library of your own based on that. I've has a lot of success extracting all sorts of data using regular expression. It would certainly be less than 10kb and would require no external dependencies other than the JDK.
I have searched throughout the site but I think I have a slightly different issue and could really do with some help before I either have heart failure or burn the computer.
I dynamically generate a list of month names (in the form June 2011, July 2011) and obviously I want this to be locale sensitive: hence I use the simple date format object as follows:
//the actual locale name is dependent on UI selection
Locale localeObject=new Locale("pl");
// intended to return full month name - in local language.
DateFormat dtFormat = new SimpleDateFormat("MMMM yyyy",localeObject);
//this bit just sets up a calendar (used for other bits but here to illustrate the issue
String systemTimeZoneName = "GMT";
TimeZone systemTimeZone=TimeZone.getTimeZone(systemTimeZoneName);
Calendar mCal = new GregorianCalendar(systemTimeZone); //"gmt" time
mCal.getTime(); //current date and time
but if I do this:
String value=dtFormat.format(mCal.getTime());
this "should" return the localized version of the month name. In polish the word "September" is "Wrzesień" -- note the accent on the n. However all I get back is "Wrzesie?"
What am I doing wrong?
Thanks to all - I accept now that it's a presentation issue - but how can I "read" the result from dtFormat safely - I added some comments below ref using getBytes etc. - this worked in other situations, I just can't seem to get access to the string result without messing it up
-- FINAL Edit; for anyone that comes accross this issue
The answer was on BalusC's blog : http://balusc.blogspot.com/2009/05/unicode-how-to-get-characters-right.html#DevelopmentEnvironment
Basically the DTformat object was returning UTF-8 and was being automatically transformed back to the system default character set when I read it into a string
so this code worked for me
new String(dtFormat.format(mCal.getTime()).getBytes("UTF-8"),"ISO-8859-1");
thank you very much for the assistance
Your problem has nothing to do with SimpleDateFormat - you're just doing the wrong thing with the result.
You haven't told us what you're doing with the string afterwards - how you're displaying it in the UI - but that's the problem. You can see that it's fetching a localized string; it's only the display of the accented character which is causing a problem. You would see exactly the same thing if you had a string constant in there containing the same accented character.
I suggest you check all the encodings used throughout your app if it's a web app, or check the font you're displaying the string in if it's a console or Swing app.
If you examine the string in the debugger I'm sure you'll see it's got exactly the right characters - it's just how they're getting to the user which is the problem.
In my tests, dtFormat.format(mCal.getTime()) returns
październik 2011
new SimpleDateFormat(0,0,localeObject).format(mCal.getTime()) returns:
poniedziałek, 3 październik 2011 14:26:53 EDT