Trying to load a model from a CIM/XML file acording to IEC 61970 (Common Information Model, for power systems models), I found a problem;
According JAXB´s graphs between elements are provided by #XmlREF #XmlID and these both should be equals to match. But in CIM/RDF the references to a resource through an ID, i.e. rdf:resource="#_37C0E103000D40CD812C47572C31C0AD" contain the "#" character, consequently JAXB is unable to match "GeographicalRegion" vs. "SubGeographicalRegion.Region" when in the rdf:resource atribute the "#" character is present.
Here an example:
<cim:GeographicalRegion rdf:ID="_37C0E103000D40CD812C47572C31C0AD">
<cim:IdentifiedObject.name>GeoRegion</cim:IdentifiedObject.name>
<cim:IdentifiedObject.localName>OpenCIM3bus</cim:IdentifiedObject.localName>
</cim:GeographicalRegion>
<cim:SubGeographicalRegion rdf:ID="_ID_SubGeographicalRegion">
<cim:IdentifiedObject.name>SubRegion</cim:IdentifiedObject.name>
<cim:IdentifiedObject.localName>SubRegion</cim:IdentifiedObject.localName>
<cim:SubGeographicalRegion.Region rdf:resource="#_37C0E103000D40CD812C47572C31C0AD"/>
</cim:SubGeographicalRegion>
I realize you're asking for a solution using JAXB, but I would urge you to consider an RDF-based solution as it is more flexible and robust. You're basically trying to reinvent what RDF parsers already have built in. RDF/XML is a difficult format to parse, it doesn't make much sense to try and hack your own parsing together - especially since files that have very different XML structures can express exactly the same information: this only becomes apparent when looking at the level of the RDF. You may find that your JAXB parser workaround works on one CIM/RDF file but completely fails on another.
So, here's an example of how to process your file using the Sesame RDF API. No inferencing is involved, this just parses the file and puts it in an in-memory RDF model, which you can then manipulate and query from any angle.
Assuming the root element of your CIM file looks something like this:
<rdf:RDF xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#"
xmlns:rdfs="http://www.w3.org/2000/01/rdf-schema#"
xmlns:cim="http://example.org/cim/">
(only a guess of course, but I need prefixes for a proper example)
Then you can do the following, using Sesame's Rio RDF/XML parser:
String baseURI = "http://example.org/my/file";
FileInputStream in = new FileInputStream("/path/to/my/cim.rdf");
Model model = Rio.parse(in, baseURI, RDFFormat.RDFXML);
This creates an in-memory RDF model of your document. You can then simply filter-query over that. For example, to print out the properties of all resources that have _37C0E103000D40CD812C47572C31C0AD as their SubGeographicalRegion.Region:
String CIM_NS = "http://example.org/cim/";
ValueFactory vf = ValueFactoryImpl.getInstance();
URI subRegion = vf.createURI(CIM_NS, "SubGeographicalRegion.Region");
URI res = vf.createURI("http://example.org/my/file#_37C0E103000D40CD812C47572C31C0AD");
Set<Resource> subs = model.filter(null, subRegion, res).subjects();
for (Resource sub: subs) {
System.out.println("resource: " + sub + " has the following properties: ");
for (URI prop: model.filter(sub, null, null).predicates()) {
System.out.println(prop + ": " + model.filter(sub, prop, null).objectValue());
}
}
Of course at this point you can also choose to convert the model to some other syntax format for further handling by your application - as you see fit. The point is that the difference between the identifiers with the leading # and without has been resolved for you by the RDF/XML parser.
This is of course personal opinion only, since I don't know the details of your use case, but I think you'll find that this is quite quick and flexible. I should also point out that although the above solution keeps the entire model in memory, you can easily adapt this to a more streaming (and therefore less memory-intensive) approach if you find your files are too big.
Related
Our C++ software use ITK to write DICOM files. In it we have a Private Tag with LO (Long String) as VR and 2 decimal values as value like 0.3234\0.34223.
The LO choice is inherent to ITK.
In other java application, I use dcm4che3 to read/write them. Since it respects the DICOM protocol, backslash are forbidden, and dcm4che interpret the value as "0.3234" and never reach the second value.
All DICOM viewer applications I use can display this value.
So my question is: Is there a trick in dcm4che to read this complete value as a string "0.3234\0.34223" despite the presence of a backslash?
Below, the code I use:
public DicomInfo uploadFile(MultipartFile file) throws IOException, ParseException {
DicomInfo infos = new DicomInfo();
Attributes attrs = readDicomAttributes(file);
infos.setTags(toAttributesObject(attrs).toString());
}
private static JsonObject toAttributesObject(Attributes targetSeriesAttrs)
{
StringWriter strWriter = new StringWriter();
JsonGenerator gen = Json.createGenerator(strWriter);
JSONWriter writer = new JSONWriter(gen);
writer.write(targetSeriesAttrs);
gen.flush();
gen.close();
return Json.createReader(new
StringReader(strWriter.toString())).readObject();
}
public Attributes readDicomAttributes(MultipartFile file) throws IOException
{
DicomInputStream dis = new DicomInputStream(file.getInputStream());
Attributes dataSet = dis.readDataset(-1, Tag.PixelData);
Attributes fmi = dis.readFileMetaInformation();
dis.close();
fmi.addAll(dataSet);
return fmi;
}
In the JSON I get for this tag:
\"00110013\":{\"vr\":\"LO\",\"Value\":[\"0.4323\"]},
As you can see it is LO and the second part is already lost.
The method I use to get the specific attribute:
attr.getStrings(0x00110013)
send back a table with only one value, 0.4323.
The problem happens during the readDataSet function.
When I open tags with software like free dicom viewer, I have the complete data, so data is here.
Ok I found the source of the problem... It is the addAll fmi.addAll(dataSet);
In dataSet, getStrings works perfectly. In fmi after addAll, the attributes lost the second value.
So my problem is to solve this addAll issue now: dcm4che3 java lib: Attributes.addAll method seems to lost multiple LO values
See answer from Paolo, and please believe us that the backslash is not a violation of the VR. Like he said, the attribute is 2-dimensional, i.e. it has two values of VR LO which are separated by the backslash.
I know a bit about the dcm4che project and the people behind it, and it is nearly unthinkable to me that it is generally incapable of handling this.
I strongly suspect that your problem is related to the fact that your attribute is private. That is, without any additional information to the tag and its value, dcm4che (and any other product) can never know that the attribute's value is encoded as VR LO (Long String).
The default transfer syntax in DICOM is Implicit Little Endian. This means, that the dataset does not convey an explicit information about the VR of the attributes in the dataset. This information is implicitly encoded by the Tag of the attribute, and the data dictionary (DICOM Part 6) must be used to look up the tag and obtain the corresponding VR. Obvioulsy this only works for well-known DICOM tags defined in the standard and fails for private ones.
So one option is to try encoding the dataset in Explicit Little Endian, which means that the VR is part of the attribute's encoding in the dataset. I would expect this to be sufficient.
Mature toolkits (like dcm4che) allow for extending the data dictionary by configuration, that is, you can extend the "official" data dictionary used by the tookit with your custom tag definitions - including the VR. So the tag can be looked up in the case that the VR is not explicitly given in the dataset itself.
Again, I am not an expert for dcm4che, but a quick search at google for "dcm4che private dictionary" yields this promising link.
I am more than confident that you can solve the problem in dcm4che and that you do not have to migrate to a different toolkit.
The solution of this problem is to write
dataSet.addAll(fmi);
return dataSet;
instead of
fmi.AddAll(dataSet);
return fmi;
since the addAll methods lost multiple values of private LO
LO can have multiple values separated by a backslash.
The DICOM standard says that in the VR "LO" the backslash cannot be used in values because it is used to separate the different elements.
In VRs that don't allow multiple elements then the backslash can be used in values.
So dcm4che is wrong here.
I have this piece of information in RDF/XML
<rdf:RDF xmlns:cim="http://iec.ch/TC57/2012/CIM-schema-cim16#" xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#">
<cim:SynchronousMachineTimeConstantReactance rdf:ID="_54302da0-b02c-11e3-af35-080027008896">
<cim:IdentifiedObject.aliasName>GENCLS_DYN</cim:IdentifiedObject.aliasName>
<cim:IdentifiedObject.name>RoundRotor Dynamics</cim:IdentifiedObject.name>
<cim:SynchronousMachineTimeConstantReactance.tpdo>0.30000001192092896</cim:SynchronousMachineTimeConstantReactance.tpdo>
<cim:SynchronousMachineTimeConstantReactance.tppdo>0.15000000596046448</cim:SynchronousMachineTimeConstantReactance.tppdo>
I have learned a little bit about how to read the document but now I want to go farther. I am "playing" with API functions to try to get the values but I am lost (and I think I do not understand properly how JENA and RDF work). So, how can I get the values of each tag?
Greetings!
I would start with the Reading and Writing RDF in Apache Jena documentation, and then read The Core RDF Api. One important step in understanding the RDF Data Model is to seperate any notion of XML from your understanding of RDF. RDF is a graph data model that just so happens to have one serialization which is in XML.
You'll note that xml-specific language like "tags" actually don't show up at all in the discussion unless you are talking about how to serialize/deserialize RDF/XML.
In order to make the data you are looking at more human friendly, I'd suggest writing it out in TURTLE. TURTLE (or TTL) is another serialization of RDF that is much easier to read or write.
The following code will express your data in TURTLE and will be helpful in understanding what you see.
final InputStream yourInputFile = ...;
final Model model = ModelFactory.createDefaultModel();
model.read(yourInputFile, "RDF/XML");
model.write(System.out, null, "TURTLE");
You'll also want to provide minimal working examples whenever submitting questions on the subject area. For example, I had to add some missing end-tags to your data in order for it to be valid XML:
<rdf:RDF xmlns:cim="http://iec.ch/TC57/2012/CIM-schema-cim16#" xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#">
<cim:SynchronousMachineTimeConstantReactance rdf:ID="_54302da0-b02c-11e3-af35-080027008896">
<cim:IdentifiedObject.aliasName>GENCLS_DYN</cim:IdentifiedObject.aliasName>
<cim:IdentifiedObject.name>RoundRotor Dynamics</cim:IdentifiedObject.name>
<cim:SynchronousMachineTimeConstantReactance.tpdo>0.30000001192092896</cim:SynchronousMachineTimeConstantReactance.tpdo>
<cim:SynchronousMachineTimeConstantReactance.tppdo>0.15000000596046448</cim:SynchronousMachineTimeConstantReactance.tppdo>
</cim:SynchronousMachineTimeConstantReactance>
</rdf:RDF>
Which becomes the following TURTLE:
<file:///R:/workspaces/create/git-svn/create-sparql/RDF/XML#_54302da0-b02c-11e3-af35-080027008896>
a cim:SynchronousMachineTimeConstantReactance ;
cim:IdentifiedObject.aliasName "GENCLS_DYN" ;
cim:IdentifiedObject.name "RoundRotor Dynamics" ;
cim:SynchronousMachineTimeConstantReactance.tpdo "0.30000001192092896" ;
cim:SynchronousMachineTimeConstantReactance.tppdo "0.15000000596046448" .
RDF operates at the statement level, so to find out that your _54302da0-b02c-11e3-af35-080027008896 is a cim:SynchronousMachineTimeConstantReactance you would look for the corresponding triples. Jena's Model API (linked to above) will provide you with methods to identify the properties that resources have.
The following will list all statements whose subject is the aforementioned resource:
final Resource s = model.getResource("file:///R:/workspaces/create/git-svn/create-sparql/RDF/XML#_54302da0-b02c-11e3-af35-080027008896");
final ExtendedIterator<Statement> properties = s.listProperties();
while( properties.hasNext() ) {
System.out.println(properties.next());
}
which produces:
[file:///R:/workspaces/create/git-svn/create-sparql/RDF/XML#_54302da0-b02c-11e3-af35-080027008896, http://iec.ch/TC57/2012/CIM-schema-cim16#SynchronousMachineTimeConstantReactance.tppdo, "0.15000000596046448"]
[file:///R:/workspaces/create/git-svn/create-sparql/RDF/XML#_54302da0-b02c-11e3-af35-080027008896, http://iec.ch/TC57/2012/CIM-schema-cim16#SynchronousMachineTimeConstantReactance.tpdo, "0.30000001192092896"]
[file:///R:/workspaces/create/git-svn/create-sparql/RDF/XML#_54302da0-b02c-11e3-af35-080027008896, http://iec.ch/TC57/2012/CIM-schema-cim16#IdentifiedObject.name, "RoundRotor Dynamics"]
[file:///R:/workspaces/create/git-svn/create-sparql/RDF/XML#_54302da0-b02c-11e3-af35-080027008896, http://iec.ch/TC57/2012/CIM-schema-cim16#IdentifiedObject.aliasName, "GENCLS_DYN"]
[file:///R:/workspaces/create/git-svn/create-sparql/RDF/XML#_54302da0-b02c-11e3-af35-080027008896, http://www.w3.org/1999/02/22-rdf-syntax-ns#type, http://iec.ch/TC57/2012/CIM-schema-cim16#SynchronousMachineTimeConstantReactance]
which produces:
I have a RDF file that I am able to read using
Model model = ModelFactory.createDefaultModel();
// use the FileManager to find the input file
InputStream in = FileManager.get().open(args[0]);
if (in == null) {
throw new IllegalArgumentException(
"File: " + args[0] + " not found");
}
// read the RDF/XML file
model.read(in, null);
I also have OWL file which contains the description of the ontology which is used for creating my models. My question is: do I need to read this file (and how?) in order to work with my RDF model correctly?
To make myself clear, I will give ou an example:
I need to know whether one resource has some relationship with other resource (for example Station1 has predicate "isResponsibleFor" Workorder1). How can I do this with Jena?
If I try to use something like resource.hasProperty(ResourceFactory.createProperty("isResponsibleFor")), it returns false (but the property is there!).
Can you direct me to some advanced tutorial on this topic perhaps? I found many tutorials on Papache site etc. but they do not provide me with the information I am looking for. Sorry if the question is not clear, I am quite new to Jena
EDIT: currently, I am searching whether my model contains given statement using this:
public static boolean containsStatement(Model model, String sub,
String pred, String obj) {
// list the statements in the Model
StmtIterator iter = model.listStatements();
// print out the predicate, subject and object of each statement
while (iter.hasNext()) {
Statement stmt = iter.nextStatement(); // get next statement
Resource subject = stmt.getSubject(); // get the subject
Property predicate = stmt.getPredicate(); // get the predicate
RDFNode object = stmt.getObject(); // get the object
if (subject.toString().contains(sub)
&& predicate.toString().contains(pred)
&& object.toString().contains(obj)) {
return true;
}
}
return false;
}
but I am pretty sure that this is highly ineffective approach.. could you suggest me something more elegant and fast? Thanks!
Short answer: no, you don't need the ontology to work with your RDF file, but in many cases it can help your application.
First, you can shorten loading your file:
Model model = FileManager.get().loadModel( args[0] );
Now, in order to work with the relationship between resources, as given by the URI of the property connecting the subject resource to the object, you need the full URI of the predicate. Typically, this will be something like http://example.com/foo#isResponsibleFor. If you just use the short-name of predicate, it won't work - which is what you are finding.
You don't show any examples of your actual RDF data, so I'm going to use a fake namespace. Use your actual namespace in your code. In the meantime:
String NS = "http://example.com/example#";
Property isResponsibleFor = model.getProperty( NS + "isResponsibleFor" );
Resource station = model.getResource( NS + "station1" );
for (StmtIterator i = station.listProperties( isResponsibleFor ); i.hasNext(); ) {
Statement s = i.next();
Resource workorder = s.getResource();
// now you can do something with the work-order resource
}
In your code, you had:
public static boolean containsStatement(Model model, String sub, String pred, String obj)
There are a number of things wrong here. First, it's better if you can write your code in a more object-oriented style, which tends not to use static methods if that can be avoided. Second, don't use strings when you refer to things in a model. Jena has the Resource class to denote resources in a model, and many other RDF-specific classes as well. Use strings for handling input from your user, otherwise convert strings to resources or other RDF objects as soon as you can. Thirdly, I'd advise against exposing the details of your representation via your object's API. containsStatement makes it clear in the API that you are using RDF triples; that's not a detail that callers of the API need to know and it breaks encapsulation. A better API would have methods such as listWorkItems - that relates to the domain, and hides details of the implementation.
Regarding the use of your ontology, there are two specific ways your application can benefit from using your ontology. First, you can automatically generate statements such as:
Property isResponsibleFor = model.getProperty( NS + "isResponsibleFor" );
by using Jena's schemagen vocabulary generator tool. You can use schemagen as part of your build process to ensure that your vocabulary class automatically stays up-to-date as your ontology changes.
Secondly, by using Jena's inference engines, you can use the ontology to infer additional statements about your domain. For example, suppose you have class WidgetOrder, which is a type of WorkItem. Without inference, and without your ontology, if you ask the model object to list all of the WorkItems, it won't list the WidgetOrder resources. However, with the ontology and the reasoner, listing resources of type WorkItem will also return the resources that only have a declared type of WidgetOrder, because the other types can be inferred.
I have about 3200 URLs to small XML files which have some data in the form of strings(obviously).The XML files are displayed(not downloaded) when I go to the URLs. So I need to extract some data from all those XMLs and save it in a single .txt file or XML file or whatever. How can I automate this process?
*Note: This is what the files look like. I need to copy the 'location' and 'title' from all of them and put them in one single file. Using what methodology can this be achieved?
<?xml version="1.0"?>
-<playlist xmlns="http://xspf.org/ns/0/" version="1">
-<tracklist>
<location>http://radiotool.com/fransn.mp3</location>
<title>France, Paris radio 104.5</title>
</tracklist>
</playlist>
*edit: Fixed XML.
It's easy enough with XQuery or XSLT, though the details will depend on how the URLs are held. If they're in a Java List, then (with Saxon at least) you can supply this list as a parameter to the following query:
declare variable urls as xs:string* external;
<data>{
for $u in $urls return doc($u)//*:tracklist
}</data>
The Java code would be something like:
Processor proc = new Processor();
XQueryCompiler c = proc.newXQueryCompiler();
XQueryEvaluator q = c.compile($query).load();
List<XdmItem> urls = new ArrayList();
for (url : inputUrls) {
urls.append(new XdmAtomicValue(url);
}
q.setExternalVariable(new QName("urls"), new XdmValue(urls));
q.setDestination(...)
run();
Have a look at the JSoup library here: http://jsoup.org/
It has facilities for pulling and fixing the contents of a URL, it is intended for HTML though, so I'm not sure it will be good for XML, but it is worth a look.
I need to parse an XML string with MATLAB (caution: without file I/O, so I don't want to write the string to a file and then read them). I'm receiving the strings from an HTTP connection and the parsing should be very fast. I'm mostly concerned about reading the values of certain tags in the entire string
The net is full of death threats about parsing XML with regexp so I didn't want to get into that just yet. I know MATLAB has seamless java integration but I'm not very java savvy. Is there a quick way to get certain values from XML very very rapidly?
For example I want to get the 'volume' information from this string below and write this to a variable.
<?xml version="1.0" encoding="UTF-8" standalone="yes" ?>
<root>
<volume>256</volume>
<length>0</length>
<time>0</time>
<state>stop</state>
....
For what it's worth, below is the Matlab executable Java code to perform the required task, without writing to an intermediate file:
%An XML formatted string
strXml = [...
'<?xml version="1.0" encoding="UTF-8" standalone="yes" ?>' char(10)...
'<root>' char(10) ...
' <volume>256</volume>' char(10) ...
' <length>0</length>' char(10) ...
' <time>0</time>' char(10) ...
' <state>stop</state>' char(10) ...
'</root>' ];
%"simple" java code to create a document from said string
xmlDocument = javax.xml.parsers.DocumentBuilderFactory.newInstance().newDocumentBuilder.parse(java.io.StringBufferInputStream(strXml));
%"intuitive" methods to explore the xmlDocument
nodeList = xmlDocument.getElementsByTagName('volume');
numberOfNodes = nodeList.getLength();
firstNode = nodeList.item(0);
firstNodeContent = firstNode.getTextContent;
disp(firstNodeContent); %Returns '256'
As an alternative, if your application allows it, consider passing the URL directly into your XML parser. Untested java code is below, but that probably also opens up the Matlab built-in xslt function as well.
xmlDocument = javax.xml.parsers.DocumentBuilderFactory.newInstance().newDocumentBuilder.parse('URL_AS_A_STRING_HERE');
Documentation here. Start at the "javax.xml.parsers" package.
There's an entire class of functions for dealing with xml, including xmlread and xmlwrite. Those should be pretty useful for your problem.
I am not familiar with Matlab's APIs at all, but I would point out that using the DOM method outlined by Pursuit will take the most time/memory if you only want specific values out of the XML stream you are getting back over the HTTP connection.
While STAX will give you the fastest parsing approach in Java, using the API can be unwieldy especially if you are not that familiar with Java. You could use SJXP which is an extremely thin abstraction ontop of STAX parsing in Java (disclaimer: I am the author) that allows you to define paths to the elements you want, then you give the parser a stream (your HTTP stream in this case) and it pulls out all the values for you.
As an example, let's say you wanted the /root/state and /root/volume values out of the examples XML you posted, the actual Java would look something like this:
// Create /root/state rule
IRule stateRule = new DefaultRule(Type.CHARACTER, "/root/state") {
#Override
public void handleParsedCharacters(XMLParser parser, String text, Object userObject) {
System.out.println("State is: " + text);
}
}
// Create /root/volume rule
IRule volRule = new DefaultRule(Type.CHARACTER, "/state/volume") {
#Override
public void handleParsedCharacters(XMLParser parser, String text, Object userObject) {
System.out.println("Volume is: " + text);
}
}
// Create the parser with the given rules
XMLParser parser = new XMLParser(stateRule, volRule);
You can do all of that initialization on program start then at some point later when you are processing the stream from your HTTP connection, you would do something like:
parser.parser(httpConnection.getOutputStream());
or the like; then all of your handler code you defined in your rules will get called as the parser runs through the stream of characters from the HTTP connection.
As I mentioned I am not familiar with Matlab and don't know the proper ways to "Matlab-i-fy" this code, but it looks like from the first example you can more or less just use the Java APIs directly in which case this solution will both be faster and use significantly less memory for parsing if that is important than the DOM approach.