We Keep Coding

Aggregating within Apache Jena

I'm using the Java API of Apache Jena to store and retrieve documents and the words within them. For this I decided to set up the following datastructure:
_dataset = TDBFactory.createDataset("./database");
_dataset.begin(ReadWrite.WRITE);
Model model = _dataset.getDefaultModel();
Resource document= model.createResource("http://name.space/Source/DocumentA");
document.addProperty(RDF.value, "Document A");
Resource word = model.createResource("http://name.space/Word/aword");
word.addProperty(RDF.value, "aword");
Resource resource = model.createResource();
resource.addProperty(RDF.value, word);
resource.addProperty(RSS.items, "5");
document.addProperty(RDF.type, resource);
_dataset.commit();
_dataset.end();
The code example above represents a document ("Document A") consisting of five (5) words ("aword"). The occurences of a word in a document are counted and stored as a property. A word can also occur in other documents, therefore the occurence count relating to a specific word in a specific document is linked together by a blank node. (I'm not entirely sure if this structure makes any sense as I'm fairly new to this way of storing information, so please feel free to provide better solutions!)
My major question is: How can I get a list of all distinct words and the sum of their occurences over all documents?

Your data model is a bit unconventional, in my opinion. With your code, you'll end up with data that looks like this (in Turtle notation), and which uses rdf:type and rdf:value in unconventional ways:
:doc rdf:value "document a" ;
rdf:type :resource .
:resource rdf:value :word ;
:items 5 .
:word rdf:value "aword" .
It's unusual, because usually you wouldn't have such complex information on the type attribute of a resource. From the SPARQL standpoint though, rdf:type and rdf:value are properties just like any other, and you can still retrieve the information you're looking for with a simple query. It would look more or less like this (though you'll need to define some prefixes, etc.):
select ?word (sum(?n) as ?nn) where {
?document rdf:type ?type .
?type rdf:value/rdf:value ?word ;
:items ?n .
}
group by ?word
That query will produce a result for each word, and with each will be the sum of all the values of the :items properties associated with the word. There are lots of questions on Stack Overflow that have examples of running SPARQL queries with Jena. E.g., (the first one that I found with Google): Query Jena TDB store.

Using Lucene, how to index TXT files into different fields?

I am using the NSF data whose format is txt. Now I have indexed these data and can send a query and got several results. But how can I search something in a selected field (eg. title) ? Because all of these NSF data are totally plain txt file. I do not think Lucene can recognize which part of the file is a "title" or something else. Should I firstly transfer the txt files to XML files (with tags telling Lucene which part is "title")? Can Lucene do that? I have no idea how to split the txt files into several fields. Can anyone please give me some suggestions? Thanks a lot!
BTW, every txt file looks like this:
---begin---
Title: Mitochondrial DNA and Historical Demography
Type: Award
Date: August 1, 1991
Number: 9000006
Abstract: asdajsfhsjdfhsjngfdjnguwiehfrwiuefnjdnfsd
----end----

You have to split the text into the several parts. You can use the resulting strings to create a field for each part of the text, i.e. title.
Create your lucene document with the fields like this:
Document doc = new Document();
doc.add(new Field("title", titleString, Field.Store.NO, Field.Index.TOKENIZED));
doc.add(new Field("abstract", abstractString, Field.Store.NO, Field.Index.TOKENIZED));
and so on. After indexing the document you can search in the title like this: title:dna
More complex queries and mixing multiple fields in the query also possible: +title:dna +abstract:"some example text" -number:935353

Adding entities to solr using solrj and schema.xml

I would like to add entities to documents like you can do with the data-config.
At the moment I'm indexing every page of my documents as a single document.
Now :
<solrDoc>
<id>1</id>
<docname>test.pdf</docmname>
<pagenumber>1</pagenumber>
<pagecontent>blablabla</pagecontent>
</solrDoc>
<solrDoc>
<id>2</id>
<docname>test.pdf</docmname>
<pagenumber>2</pagenumber>
<pagecontent>blablabla</pagecontent>
</solrDoc>
As you can see the data related to the document is stored x pages times. I would like to get documents like this:
<doc>
<id>1</id>
<docname>test.pdf</docmname>
<pageEntries> //multivaluefield
<pageEntry><pagenumber>1</pagenumber><pagecontent>blablabla</pagecontent></pageEntry>
<pageEntry><pagenumber>2</pagenumber><pagecontent>blablabla</pagecontent></pageEntry>
</pageEntries>
</doc>
I don't know how to make something like pageEntry. I saw that solr can import entities from databases but I'm wondering how I can do the same? (or something similar)
I'm using solr 3.6.1. The page extraction is done by myself using pdfbox.
Java code:
SolrInputDocument solrDoc = new SolrInputDocument();
solrDoc.setField("id", 1);
solrDoc.setField("filename", "test");
for (int p : pages) {
solrDoc.addField("page", p);
}
for (String pc : pagecont) {
solrDoc.addField("pagecont", pc);
}

If the extraction is performed by you, you can club all the pages and feed it as a single Solr document with the pagenumber & pagecontent being multivalued fields.
You can use the same id for all the pages (with the id not being a primary field in the schema definition) and use Grouping (Field Collapsing) to group the results for the documents.

XPath normalize-space() to return a sequence of normalized strings

I need to use the XPath function normalized-space() to normalize the text I want to extract from a XHTML document: http://test.anahnarciso.com/clean_bigbook_0.html
I'm using the following expression:
//*[#slot="address"]/normalize-space(.)
Which works perfectly in Qizx Studio, the tool I use to test XPath expressions.
let $doc := doc('http://test.anahnarciso.com/clean_bigbook_0.html')
return $doc//*[#slot="address"]/normalize-space(.)
This simple query returns a sequence of xs:string.
144 Hempstead Tpke
403 West St
880 Old Country Rd
8412 164th St
8412 164th St
1 Irving Pl
1622 McDonald Ave
255 Conklin Ave
22011 Hempstead Ave
7909 Queens Blvd
11820 Queens Blvd
1027 Atlantic Ave
1068 Utica Ave
1002 Clintonville St
1002 Clintonville St
1156 Hempstead Tpke
Route 49
10007 Rockaway Blvd
12694 Willets Point Blvd
343 James St
Now, I want to use the previous expression in my Java code.
String exp = "//*[#slot=\"address"\"]/normalize-space(.)";
XPath xpath = XPathFactory.newInstance().newXPath();
XPathExpression expr = xpath.compile(exp);
Object result = expr.evaluate(doc, XPathConstants.NODESET);
But the last line throws an Exception:
Cannot convert XPath value to Java object: required class is org.w3c.dom.NodeList; supplied value has type xs:string
Obvsiously, I should change XPathConstants.NODESET for something; I tried XPathConstants.STRING but it only returns the first element of the sequence.
How can I obtain something like an array of Strings?
Thanks in advance.

Your expression works in XPath 2.0, but is illegal in XPath 1.0 (which is used in Java) - it should be normalize-space(//*[#slot='address']).
Anyway, in XPath 1.0, when normalize-space() is called on a node-set, only the first node (in document order) is taken.
In order to do what you want to do, you'll need to use a XPath 2.0 compatible parser, or traverse the resulting node-set and call normalize-space() on every node:
XPath xpath = XPathFactory.newInstance().newXPath();
XPathExpression expr;
String select = "//*[#slot='address']";
expr = xpath.compile(select);
NodeList result = (NodeList)expr.evaluate(input, XPathConstants.NODESET);
String normalize = "normalize-space(.)";
expr = xpath.compile(normalize);
int length = result.getLength();
for (int i = 0; i < length; i++) {
System.out.println(expr.evaluate(result.item(i), XPathConstants.STRING));
}
...outputs exactly your given output.

It depends on what version of XPath you're using. Check out this post, hopefully it'll answer your question: Is it possible to apply normalize-space to all nodes XPath expression finds? Good luck.

The expression:
//*[#slot="address"]/normalize-space(.)
is syntactically legal (and practically useful) XPath 2.0 expression.
The same expression is not syntactically legal in XPath 1.0 -- it isn't allowed for a location step to be a function call.
In fact, it isn't possible to write a single XPath 1.0 expression the result of whose evaluation is the wanted set of strings.
You need to use in your program a product that implements XPath 2.0 -- such as Saxon 9.x.

As you noted, the XPath 2.0 expression //*[#slot="address"]/normalize-space(.) returns a sequence of strings. This return type is not supported by the JAXP XPathConstants class, because the JAXP interfaces were not designed to support XPath 2.0.
This leaves you with two choices:
Use an XPath 2.0 processor that has native interfaces for XPath 2.0 or that can convert sequences to a return type supported by JAXP
Use only XPath 1.0 expressions. For example, in your case you could simply select the target nodes:
//*[#slot="address"]
And then iterate the resulting nodeset, collecting the results into an array or List.
Note that it's important to distinguish between the processer you're using to evaluate the expression and the interface you're using to initiate the evaluation.

How can one extract rdf:about or rdf:ID properties from triples using SPARQL?

It seemed a trivial matter at the beginning but so far I have not managed to get the unique identifier for a given resource using SPARQL. What I mean is given, e.g., rdf:Description rdf:about="http://..." and then some properties identifying this resource, what I want to do is to first find this very resource and then retrieve all the triples given some URI.
I have tried naïve approaches by writing statements in a WHERE clause such as:
?x rdf:about ?y and ?x rdfs:about ?y
I hope I am being precise.

You're making a classic mistake: confusing RDF (which is what SPARQL queries) with (one of) its serialisation, namely RDF/XML. rdf:about (and rdf:ID, rdf:Description, rdf:resource) are part of RDF/XML, a way RDF is written down. You can play around with the RDF Validator to see what RDF triples result from a piece of RDF/XML.
In your case let's start with:
<?xml version="1.0"?>
<rdf:RDF
xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#"
xmlns:dc="http://purl.org/dc/terms/">
<rdf:Description rdf:about="http://www.example.org/">
<dc:title>Example for Donal Fellows</dc:title>
</rdf:Description>
</rdf:RDF>
Plug that into the validator and you get:
Number Subject Predicate Object
1 http://www.example.org/ http://purl.org/dc/terms/title "Example for Donal Fellows"
(you can also ask for a pictorial representation)
Notice that rdf:about is not present: its value provides the subject for the triple.
How do I do a query to find properties associated with http://www.example.org? Like this:
select * {
<http://www.example.org/> ?predicate ?object
}
You'll get:
?predicate ?object
<http://purl.org/dc/terms/title> "Example for Donal Fellows"
You'll notice that the query is a triple match with variables (?v) in places where we want to find values. We could also ask what predicate links http://www.example.org/ with "Example for..." by asking:
select * {
<http://www.example.org/> ?predicate "Example for Donal Fellows"
}
This pattern matching is the heart of SPARQL.
RDF/XML is a tricky beast, and you might find it easier to work with N-Triples, which is very verbose but clear, or turtle, which is like N-Triples with a large number of shorthands and abbreviations. Turtle is often preferred by the rdf community.
P.S. rdfs:about doesn't exist anywhere.