We Keep Coding

How to conduct nested named entity recognition in OpenNLP?

I am currently working on a java web server project, that requires the use of Natural Language processing, specifically Named Entity Recognition (NER).
I was using OpenNLP for java, since it was easy to add custom training data. It works perfectly.
However, I need to also be able to extract entites inside of entities (Nested named entity recognition). I tried doing this in OpenNLP, but I got parsing errors. So my guess is that OpenNLP sadly does not support nested entities.
Here is an example of what I need to parse:
Remind me to [START:reminder] give some presents to [START:contact] John [END] and [START:contact] Charlie [END][END].
If this cannot be achieved with OpenNLP, is there any other Java NLP Library that could do this. If there are no Java libraries at all, are there any NLP libraries in any other language that can do this?
Please help. Thanks!

The short answer is:
This cannot be achieved using openNLP NER which is suitable only for continuous entities because it use a BIO tagging scheme.
I don't know any library in any language capable of do this.
I think you are extending too much the concept of entity, which is habitually associated with persons, places, organizations, gene names etc.
But not with the identification of complex structures within text.
For that purpose you need to think in a more elaborated solution, taking into account the grammatical structure of the sentence, which can be obtained using a parser like the one in OpenNLP, and maybe combine this with the output of the NER process.

For the purpose of Name Entity Recognition (Java based) I use the following:
Apache UIMA
ClearTK
https://github.com/merishav/cleartk-tutorials
You can train models for your use case, I have already trained for NER for person, places, date of birth, profession.
ClearTK gives you a wrapper on MalletCRFClassifier.

Use this python source code (Python 3) https://gist.github.com/ttpro1995/cd8c60cfc72416a02713bb93dff9ae6f
It's create multiple un-nest version of nest data for you.
For input sentence below ( input data must be tokenized first, so there are space between and thing around it)
Remind me to <START:reminder> give some presents to <START:contact> John <END> and <START:contact> Charlie <END> <END> .
It output multiple sentence with different nest level.
Remind me to give some presents to John and Charlie .
Remind me to <START:reminder> give some presents to John and Charlie <END> .
Remind me to give some presents to <START:contact> John <END> and <START:contact> Charlie <END> .
Full source code here for quick copy-paste
import sys
END_TAG = 0
START_TAG = 1
NOT_TAG = -1
def detect_tag(in_token):
"""
detect tag in token
:param in_token:
:return:
"""
if "<START:" in in_token:
return START_TAG
elif "<END>" == in_token:
return END_TAG
return NOT_TAG
def remove_nest_tag(in_str):
"""
với <START:ORGANIZATION> Sở Cảnh sát Phòng cháy , chữa cháy ( PCCC ) và cứu nạn , cứu hộ <START:LOCATION> Hà Nội <END> <END>
:param in_str:
:return:
"""
state = 0
taglist = []
tag_dict = dict()
sentence_token = in_str.split()
## detect token tag
max_nest = 0
for index, token in enumerate(sentence_token):
# print(token + str(detect_tag(token)))
tag = detect_tag(token)
if tag > 0:
state += 1
if max_nest < state:
max_nest = state
token_info = (index, state, token)
taglist.append(token_info)
tag_dict[index] = token_info
elif tag == 0:
token_info = (index, state, token)
taglist.append(token_info)
tag_dict[index] = token_info
state -= 1
generate_sentences = []
for state in range(max_nest+1):
generate_sentence_token = []
for index, token in enumerate(sentence_token):
if detect_tag(token) >= 0: # is a tag
token_info = tag_dict[index]
if token_info[1] == state:
generate_sentence_token.append(token)
elif detect_tag(token) == -1 : # not a tag
generate_sentence_token.append(token)
sentence = ' '.join(generate_sentence_token)
generate_sentences.append(sentence)
return generate_sentences
# generate sentence
print(taglist)
def test():
tstr2 = "Remind me to <START:reminder> give some presents to <START:contact> John <END> and <START:contact> Charlie <END> <END> ."
result = remove_nest_tag(tstr2)
print("-----")
for sentence in result:
print(sentence)
if __name__ == "__main__":
"""
un-nest dataset for opennlp name
"""
# test()
# test()
if len(sys.argv) > 1:
inpath = sys.argv[1]
infile = open(inpath, 'r')
outfile = open(inpath+".out", 'w')
for line in infile:
sentences = remove_nest_tag(line)
for sentence in sentences:
outfile.write(sentence+"\n")
outfile.close()
else:
print("usage: python unnest_data.py input.txt")

How to extract sub-strings for a collection of text?

I extracted text from pdf document. .. I want to extract some particular fields in it using java..
The portion of text ..
US00RE44697E (i9) United States (12) Reissued Patent (10)
Patent Number: RE44,697 E Jones et al. (45) Date of
ReissuedPatent: Jan. 7, 2014 (54) ENCRYPTIONPROCESSORWITH SHARED
MEMORY INTERCONNECT (75) Inventors: David E.Jones, Ottawa
(CA); Cormac M.O'Connell, Carp (CA) (73) Assignee: Mosaid
Technologies Incorporated, Ottawa, Ontario (CA) (21)
Appl.No.: 13/603,137 (22) Filed: Sep. 4, 2012 Related U.S.
Patent Documents Reissue of: (64) Patent No.: Issued:
Appl. No.: Filed: 6,088,800 Jul. 11, 2000
09/032,029 Feb. 27, 1998 (51) Int.CI. G06F 21/00
(2013.01) (52) U.S. CI. USPC .............713/189; 713/190;
713/193; 380/28; 380/33; 380/52 (58) Field of Classification
Search None
Now my mission is to extract fields form it and give to strings.. that is
the text (10) Patent Number: RE44,697 E will be extracted as String pat_no= " RE44,697 E"
the text (54) ENCRYPTIONPROCESSORWITH SHARED
MEMORY INTERCONNECT will be extracted as String title= "ENCRYPTIONPROCESSORWITH SHARED
MEMORY INTERCONNECT"
the extremely irregular text block
(64) Patent No.: Issued: Appl. No.: Filed:
6,088,800 Jul. 11, 2000 09/032,029 Feb. 27, 1998
have to be extracted as
String pat_no_org = "6,088,800";
String issued = "jul.11,2000"
String filed = "feb 27 ,1998"
......
like this..
My Works
First i used the string.split , string.substring , string,indexof and even apache string utils , but none helped.. Because the text are scattered , above methods doesn't helped.. I also tried regular expressions ,but since I very weak in it I can't program .
Please tell me how to achieve my objective using java ?

With regex, I would split it in 3 parts:
1.) (10) Patent Number the regex could look like this:
\(10\)\s*Patent Number:\s*([\w,]+)
as a java string:
"\\(10\\)\\s*Patent Number:\\s*([\\w,]+)"
The matches for the first parenthesized group will be in [1].
\s is a shorthand for [ \t\r\n\f] any kind of white-space.
\w is a shorthand for [A-Za-z0-9_] word-characters, together with , in a character class.
Some characters have special meanings in regex. They have to be escaped with a backslash.
2.) (54) ENCRYPT...
A pattern could look like:
(?s)\(54\)\s*(.*?)\s*(?=\(\d|$\))
as a java string:
"(?s)\\(54\\)\\s*(.*?)\\s*(?=\\(\\d|$\\))"
(?s) The s modifier equals Pattern.DOTALL where the dot matches new-lines too.
(?=\(\d|$\)) a lookahead is used, to match (.*?) lazy any amount of any characters until another ( followed by a digit | or string-end $ (anchor for end) is seen.
3.) For the other desired 3 parts I would try to reflect formatting of the input with the pattern. This requires, that all data is constructed compatible. A pattern could look like this:
(?s)\(64\).*?Filed:\s*([\d,]+)\s*(\w+\.\s*\d+,\s*\d+)\s*\n[\d+][^\n]+\n\s*(\w+\.\s*\d+,\s*\d+)
as a java string:
"(?s)\\(64\\).*?Filed:\\s*([\\d,]+)\\s*(\\w+\\.\\s*\\d+,\\s*\\d+)\\s*\\n[\\d+][^\\n]+\\n\\s*(\\w+\\.\\s*\\d+,\\s*\\d+)"
\n matches a newline.
Matches will be in [1] e.g. 6,088,800, [2] e.g. Jul. 11, 2000 and [3] e.g. Feb. 27, 1998.
For getting started with regex, this is too much information at once :)

Sparql query with java

A have an RDF file and I want to make a select SPARQL query that will select descriptions of a specific author. I tried a query like below but with no success. I am new to SPARQL so if anyone can help I would appreciate it…
MY TRY:
String query = "PREFIX schema: <"+Constants.SCHEMA+"> \n" +
"SELECT ?description \n" +
"WHERE { \n" +
"?review a schema:Review; \n" +
"schema:author \"Judson C.\"; \n" +
"}";
The RDF file looks like this:
<rdf:Description rdf:nodeID="A3">
<schema:ratingValue>5.0</schema:ratingValue>
<rdf:type rdf:resource="http://schema.org/Rating" />
</rdf:Description>
<rdf:Description rdf:nodeID="A4">
< schema:reviewRating rdf:nodeID="A5"/>
<schema:description>OMG Love this place!! But glad I don't have to
stand in line for an hour like back in the day to get my fix!</schema:description>
<schema:datePublished>2013-05-18</schema:datePublished>
<schema:author>Judson C.</schema:author>
<rdf:type rdf:resource="http://schema.org/Review" />
</rdf:Description>
<rdf:Description rdf:nodeID="A6">
<schema:reviewRating rdf:nodeID="A7" />
<schema:description>If you're ever wondering of where to stuff your
face with a good ol' sandwich, you'll definitely have to pop in at
Ike's Place. Not only are the different sandwiches uniquely named,
they're also uniquely flavor profiled. A wonderful experience
indeed!Sandwiches I've personally ordered are the Love Triangle,
Nacho girl, and the Al Bundy. For all you people out there who loved
the tv show, Married with Children, I think you could appreciate the
sentiment of having a sammie named after the 'No Ma'am enthusiast. I
know I did. Ahhh... *tucks hand under pant waistband*All kidding
aside, these sandwiches are pretty legit. You go to the counter and
order whichever sandwich sounds appealing and they can either add
chips or a drink to your meal. Simple. They have a wide range of
sandwiches that are made to please any meat lover, vegetarian, or
vegan palate alike. The huge variety is an A+ in my book. They can
even individualize and customize your sandwich to your liking. This
means that even the pickiest of eaters are welcome!So if you're
interested in a fun place to enjoy a good sandwich, most definitely
give 'em a try!</schema:description>
<schema:datePublished>2013-04-30</schema:datePublished>
<schema:author>Ann S.</schema:author>
<rdf:type rdf:resource="http://schema.org/Review" />
</rdf:Description>

Short Answer
The short answer is that the query isn't syntactically well formed, and that it doesn't return the things you're looking for. The query needs to terminate its triple pattern with a full stop (.), not a semicolon (;). The fixed query should look something like this:
PREFIX schema: <http://schema.org/>
SELECT ?description
WHERE {
?review a schema:Review ;
schema:author "Judson C." .
}
However, this SELECTs the variable ?description which isn't used in the query, so it's always empty. You probably either want to SELECT ?review or else add to the pattern schema:description ?description.
Long Answer
A bit more detail that shows that this works follows. The RDF you provided isn't well formed, so it's hard to test against it. I took what you provided, and wrote it up as N3 to get the following. (RDF/XML is at the end of the answer, too.)
#prefix schema: <http://schema.org/> .
[]
a schema:Rating ;
schema:ratingValue "5.0" .
[]
a schema:Review ;
schema:reviewRating [] ;
schema:description "OMG Love this place!! But glad I don't have to stand in line for an hour like back in the day to get my fix!" ;
schema:datePublished "2013-05-18" ;
schema:author "Judson C." .
[]
a schema:Review ;
schema:reviewRating [] ;
schema:description "If you're ever wondering of where to stuff your face with a good ol' sandwich, you'll definitely have to pop in at Ike's Place. Not only are the different sandwiches uniquely named, they're also uniquely flavor profiled. A wonderful experience indeed!Sandwiches I've personally ordered are the Love Triangle, Nacho girl, and the Al Bundy. For all you people out there who loved the tv show, Married with Children, I think you could appreciate the sentiment of having a sammie named after the 'No Ma'am enthusiast. I know I did. Ahhh... *tucks hand under pant waistband*All kidding aside, these sandwiches are pretty legit. You go to the counter and order whichever sandwich sounds appealing and they can either add chips or a drink to your meal. Simple. They have a wide range of sandwiches that are made to please any meat lover, vegetarian, or vegan palate alike. The huge variety is an A+ in my book. They can even individualize and customize your sandwich to your liking. This means that even the pickiest of eaters are welcome!So if you're interested in a fun place to enjoy a good sandwich, most definitely give 'em a try!" ;
schema:datePublished "2013-04-30" ;
schema:author "Ann S." .
The query, as I mentioned above, has some syntax errors; the triple pattern should end with a ., not a ;, and should be:
PREFIX schema: <http://schema.org/>
SELECT ?description
WHERE {
?review a schema:Review ;
schema:author "Judson C." .
}
This doesn't return anything useful though, because ?description isn't used in the query. Using Jena's ARQ (and writing the query as schema.query) we output with one row (since the pattern matched), but with an unbound variable:
$ /usr/local/lib/apache-jena-2.10.0/bin/arq \
--query schema.query \
--data schema.n3
---------------
| description |
===============
| |
---------------
Here's a query that binds ?description:
PREFIX schema: <http://schema.org/>
SELECT ?description
WHERE {
?review a schema:Review ;
schema:author "Judson C." ;
schema:description ?description .
}
As expected, it gives some useful results:
$ /usr/local/lib/apache-jena-2.10.0/bin/arq \
--query schema2.query \
--data schema.n3
-------------------------------------------------------------------------------------------------------------------
| description |
===================================================================================================================
| "OMG Love this place!! But glad I don't have to stand in line for an hour like back in the day to get my fix!" |
-------------------------------------------------------------------------------------------------------------------
RDF/XML Appendix
Here's the RDF/XML serialization of the data at hand.
<rdf:RDF
xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#"
xmlns:schema="http://schema.org/">
<schema:Review>
<schema:reviewRating rdf:parseType="Resource">
</schema:reviewRating>
<schema:description>If you're ever wondering of where to stuff your face with a good ol' sandwich, you'll definitely have to pop in at Ike's Place. Not only are the different sandwiches uniquely named, they're also uniquely flavor profiled. A wonderful experience indeed!Sandwiches I've personally ordered are the Love Triangle, Nacho girl, and the Al Bundy. For all you people out there who loved the tv show, Married with Children, I think you could appreciate the sentiment of having a sammie named after the 'No Ma'am enthusiast. I know I did. Ahhh... *tucks hand under pant waistband*All kidding aside, these sandwiches are pretty legit. You go to the counter and order whichever sandwich sounds appealing and they can either add chips or a drink to your meal. Simple. They have a wide range of sandwiches that are made to please any meat lover, vegetarian, or vegan palate alike. The huge variety is an A+ in my book. They can even individualize and customize your sandwich to your liking. This means that even the pickiest of eaters are welcome!So if you're interested in a fun place to enjoy a good sandwich, most definitely give 'em a try!</schema:description>
<schema:datePublished>2013-04-30</schema:datePublished>
<schema:author>Ann S.</schema:author>
</schema:Review>
<schema:Review>
<schema:reviewRating rdf:parseType="Resource">
</schema:reviewRating>
<schema:description>OMG Love this place!! But glad I don't have to stand in line for an hour like back in the day to get my fix!</schema:description>
<schema:datePublished>2013-05-18</schema:datePublished>
<schema:author>Judson C.</schema:author>
</schema:Review>
<schema:Rating>
<schema:ratingValue>5.0</schema:ratingValue>
</schema:Rating>
</rdf:RDF>

I suppose the question is really on SPARQL, not java.
Have you tried if works writing:
?review schema:author "Judson C."^^xsd:string
or:
?review schema:author ?author:
FILTER(str(?author)=="Judson C.")

Java String formatting solution

I have a string description of a company, which is nasty written by different users (hand-typed). Here is a example (focus on the dots, spaces, first letters etc..):
XXXX is a Global menagement consulting,Technology services and
outsourcing company, with 257000people serving clients in more than
120 countries.. combining unparalleled experience, comprehensive
capabilities across all industries and business functions,and
extensive research on the worlds most successfull companies, XXXX
collaborates with clients to help them become high-performance
businesses and governments., the company generated net revenues of
US$27.9 Billion for the fiscal year ended 31.07.2012..
Now what i want is to format the string to a bit nicer version like this:
XXXX is a global management consulting, technology services and
outsourcing company, with 257,000 people serving clients in more than
120 countries. Combining unparalleled experience, comprehensive
capabilities across all industries and business functions, and
extensive research on the world’s most successful companies, XXXX
collaborates with clients to help them become high-performance
businesses and governments. The company generated net revenues of
US$27.9 billion for the fiscal year ended Aug. 31, 2012.
My question is: Is there any library with already defined methods which could do all the spelling corrections, unneeded space removal, etc .. ?
So far, I do it be replacing stuff like " ," with ", " and toUpperCase() if the is a "///." in front etc..
desc = desc.replace(" ", " ");
desc = desc.replace("..", ".");
desc = desc.replace(" .", ".");
desc = desc.replace(" ,", ", ");
desc = desc.replace(".,", ".");
desc = desc.replace(",.", ".");
desc = desc.replace(", .", ".");
desc = desc.replace("*", "");
I'm sure there is a cleaner and better version to do this. Using regex maybe??
Any solution would be appreciated.

If I were trying to solve your problem, I would probably read the text 1 char at a time, and format it as you go. For example, in psuedocode...
while (has more chars){
char letter = readChar();
if (letter == ','){
// checking for the ',.' combination
letter = readChar();
if (readChar == '.'){
// write out a '.' only
out.print('.');
}
else {
// it wasn't the ',.' combination, so you need to output both characters, whatever they are
out.print(',');
out.print(letter);
}
}
else if (another letter you want to filter){
// etc.
}
else {
// doesn't match any of the filters, so just output the letter
out.print(letter);
}
}
Basically if you read the text 1 char at a time, you can detect any of your chosen formatting problems as you go, and correct them immediately. This provides a performance improvement, as you're only reading over the text string once (not 8 times, like you are currently doing), and allows you to add as many different/complex formatting changes as you want. The downside, however, is that you need to write the logic yourself rather than relying on in-built functions.

Java regex trying to split string up

Hi I am trying to split this string up (its quite long):
Library Catalogue Log off | Borrower record | Course Reading | Collections | A-Z E-Journal list | ILL Request | Help   Browse | Search | Results List | Previous Searches | My e-Shelf | Self-Issue | Feedback       Selected records:  View Selected  |  Save/Mail  |  Create Subset  |  Add to My e-Shelf  |        Whole set:  Select All  |  Deselect  |  Rank  |  Refine  |  Filter   Records 1 - 15 of 101005 (maximum display and sort is 2500 records)         1 Drower, E. S. (Ethel Stefana), Lady, b. 1879. Lady E.S. Drower’s scholarly correspondence : an intrepid English autodidact in Iraq / edited by 2012. BK Book University Library( 1/ 0) 2 Kowalski, Robin M. Cyberbullying : bullying in the digital age / Robin M. Kowalski, Susan P. Limber, Patricia W. Ag 2012. BK Book University Library( 1/ 0) ... 15 Ambrose, Gavin. Approach and language [electronic resource] / Gavin Ambrose, Nigel Aono-Billson. 2011. BK Book
So that I either get back:
1 Drower, E. S. (Ethel Stefana), Lady, b. 1879. Lady E.S. Drower’s scholarly correspondence : an intrepid English autodidact in Iraq / edited by 2012. BK Book University Library( 1/ 0)
// Or
1 Drower, E. S. (Ethel Stefana), Lady, b. 1879. Lady E.S. Drower’s scholarly correspondence : an intrepid English autodidact in Iraq
This is just an example and the 1 Drower, E. S. ... will not be static. While the input will be different every time (the detail between 1 and 2) the general layout of the string will always be the same.
I have:
String top = ".* (.*)";
String bottom = "\( \d/ \d\)\W*";
Pattern p = Pattern.compile(top); //+bottom
Matcher matcher = p.matcher(td); //td is the input String
String items = matcher.group();
System.out.println(items);
When I run it with top, it is meant to remove all of the headers but all I get back is No match found. bottom is my attempt to split the rest of the string.
I can post all of the input up to number 15 if it is needed. What I need is to split up the input string so that I can work with each individual of the 15 results.
Thanks for your help!

This will provide both inputs for you. It is what you wanted?
String text = "Library Catalogue Log off ..."; \\truncated text
Pattern p = Pattern.compile("((1 Drower.+Iraq).+0\\)).+2 Kowalski");
Matcher m = p.matcher(text);
if (m.find()) {
System.out.println(m.group(1));
System.out.println(m.group(2));
}
Compile and run code here.

First off you need to separate the headers from the result data. Assuming that each time there will be that block of 9 whitespaces you can use this: .*\s{9}(.*)
Next you need to parse the data into rows, this is more difficult because you have no row delimiters. The best you can do is assume that rows are delimited by: a space then one or more digits then another space.
((?<=(?:^|\s))\d+\s.*?(?=(?:$|\s\d+\s)))
If you're planning to try to parse the records into fields then don't bother unless you can change the delimiters!
A little explanation of what each bit does:
(?<=(?:^|\s)) Look behind: Make sure the characters preceding the group is either the start of the string (1st record), or a space (all other records).
\d+\s.*? Capture group: One or more digits followed by a space, then followed by text. This is the only part of the expression that shows up in the output because of the use of non-capturing groups ?: in the assertions.
(?=(?:$|\s\d+\s)) Look ahead: Make sure the characters following the group are either the end of string marker $ or a space followed by 1+ digits, followed by a space (indicating the next record).
This method is works with the fields you provided, but it will break if you have a record that contains the custom delimiter e.g. a book called "My 10 favourite things". There other ways of parsing records that are a little safer, but if that's what you want to do then it's beyond the expectations of regex...