We Keep Coding

Java JAXB multithreading unmarshalling

I'm going to create a class, which should unmarshall very huge xml files.
I've implemented general unmarshalling:
public XMLProcessor(XMLFile file) throws JAXBException, IOException, SAXException {
JAXBContext jc = JAXBContext.newInstance(Customers.class);
Unmarshaller unmarshaller = jc.createUnmarshaller();
File xml = new File(file.getFile().getOriginalFilename());
file.getFile().transferTo(xml);
this.customers = (Customers) unmarshaller.unmarshal(xml);
}
It works fine, but it took more than a minute to process 1 million customers XML.
Can i improve perfomance by creating multiple threads and unmarshall a few parts of XML file concurrently?
How should i split my XML file into parts?
Could you show me some sample code for my case?

Although I cannot provide a complete solution yet, I'd like to share with you the approach that I am currently implementing on a similar problem. My XML file structure is like:
<products>
<product id ="p1">
<variant id="v1"></variant>
<variant id="v2"></variant>
</product>
<product id ="p2">
<variant id="v3"></variant>
<variant id="v4"></variant>
</product>
</products>
products and variants may be quite complex, with a lot of attributes, lists etc.
My current approach is to use SAX to extract the XML-stream of a single product entity and then hand this over to a new Unmarshaller Thread (with standard multi-threading operations, limiting to a max thread count, etc.).
However I am still not 100% confident if SAX generates too much overhead (which could eat up the multi-threading benefit). If this is the case, I'll try to read the XML-stream directly, reacting on the open/close-tags for "". A this won't be xml-conform, this is my measure of last resort

how to append the xml string to existing xml file using xstream

I am using xstream to marshal / unmarshal between java object to / from xml, one question is, is there a right solution to solve my problem (using xstream or other advanced method, instead of pure java API).
The existing XML file can grow quite big (more than 200 mb, for example), and I like to append the new xml to this existing XML file, but without unmarshal the existing XML file first, simply append it to the end (before the root element).
Please advice, thanks.

You could load the first XML as org.w3c.dom.Document and import the second one as org.w3c.dom.Element:
Element nodeToImport = DocumentBuilderFactory.newInstance().newDocumentBuilder().parse( secondXmlFile ).getDocumentElement();
dom.importNode( nodeToImport, true );
...
There is a similar example here, but with a new node around the root one: Add an element around root element of given XML file that is stored in org.w3c.dom.Document

JAXB generates unformatted XML using XMLEventWriter

I'm using JAXB in order to generate XML files, and due to a business need I'm currently writing it to the middle of some other XML file using XMLEventWriter:
marshaller.marshal(jaxbElement, xmlEventWriter);
And currently setting some properties like:
marshaller.setProperty(Marshaller.JAXB_FORMATTED_OUTPUT, true);
marshaller.setProperty(Marshaller.JAXB_ENCODING, "utf-8");
marshaller.setProperty(Marshaller.JAXB_FRAGMENT, true);
But, besides having the JAXB_FORMATTED_OUTPUT set to true, my XML is not being formatted.
Does anyone knows what may be the problem?
This only happens when I use the XMLEventWriter...
Thanks in advance.

When you are using an XMLEventWriter as a sink, the JAXB marshaller is only in charge of sending the appropriate XML events to it and the XMLEventWriter may still choose to write out unformatted XML. My advice is to check the configuration of your XMLEventWriter in addition to Marshaller.

Unfortunately, the default XmlEventWriter implementation does not indent. The stax-utils library provides a IndentingXMLEventWriter which might help in these cases.

Jaxp xml parsing & validation without writing to file

I've a JMS messaging app thats reading and writing to MQ queues. The message data is string form and in xml format (minus the normal header markers like the xml version etc). I'm looking at the best ways to read in, write out and validate against an xsd schema however the examples I'm coming across all talk about working with files.
Is there any way (tutorials out there) to take an xml string; read it in and validate it and also do the same for an xml string I create validate and write out without writing to disk?
Would appreciate any pointers.

Check out the javax.xml.validation APIs in Java SE, in particular the Validator class which is used to validate XML content against an XML schema:
http://download-llnw.oracle.com/javase/6/docs/api/javax/xml/validation/package-summary.html

Use a StringReader on the strings, pass the reader to the JAXB methods to read the contents.

thanks folks I managed to sort this one out with the following.
Marshall:
JAXBContext jaxbContext = JAXUtility.getContext(packageLocation);
StringWriter sw = new StringWriter();
Marshaller m = jaxbContext.createMarshaller();
m.setProperty(Marshaller.JAXB_FORMATTED_OUTPUT, Boolean.TRUE);
if (o instanceof UnadvisedDeal) { m.marshal((UnadvisedDeal) o,sw);
UnMarshall:
JAXBContext jaxbContext = JAXUtility.getContext(packageLocation);
Unmarshaller um = jaxbContext.createUnmarshaller();
ud = (UnadvisedDeal) um.unmarshal(new StringReader(sw.toString()));
thanks for the help though

XML to be validated against multiple xsd schemas

I'm writing the xsd and the code to validate, so I have great control here.
I would like to have an upload facility that adds stuff to my application based on an xml file. One part of the xml file should be validated against different schemas based on one of the values in the other part of it. Here's an example to illustrate:
<foo>
<name>Harold</name>
<bar>Alpha</bar>
<baz>Mercury</baz>
<!-- ... more general info that applies to all foos ... -->
<bar-config>
<!-- the content here is specific to the bar named "Alpha" -->
</bar-config>
<baz-config>
<!-- the content here is specific to the baz named "Mercury" -->
</baz>
</foo>
In this case, there is some controlled vocabulary for the content of <bar>, and I can handle that part just fine. Then, based on the bar value, the appropriate xml schema should be used to validate the content of bar-config. Similarly for baz and baz-config.
The code doing the parsing/validation is written in Java. Not sure how language-dependent the solution will be.
Ideally, the solution would permit the xml author to declare the appropriate schema locations and what-not so that s/he could get the xml validated on the fly in a sufficiently smart editor.
Also, the possible values for <bar> and <baz> are orthogonal, so I don't want to do this by extension for every possible bar/baz combo. What I mean is, if there are 24 possible bar values/schemas and 8 possible baz values/schemas, I want to be able to write 1 + 24 + 8 = 33 total schemas, instead of 1 * 24 * 8 = 192 total schemas.
Also, I'd prefer to NOT break out the bar-config and baz-config into separate xml files if possible. I realize that might make all the problems much easier, as each xml file would have a single schema, but I'm trying to see if there is a good single-xml-file solution.

I finally figured this out.
First of all, in the foo schema, the bar-config and baz-config elements have a type which includes an any element, like this:
<sequence>
<any minOccurs="0" maxOccurs="1"
processContents="lax" namespace="##any" />
</sequence>
In the xml, then, you must specify the proper namespace using the xmlns attribute on the child element of bar-config or baz-config, like this:
<bar-config>
<config xmlns="http://www.example.org/bar/Alpha">
... config xml here ...
</config>
</bar-config>
Then, your XML schema file for bar Alpha will have a target namespace of http://www.example.org/bar/Alpha and will define the root element config.
If your XML file has namespace declarations and schema locations for both of the schema files, this is sufficient for the editor to do all of the validating (at least good enough for Eclipse).
So far, we have satisfied the requirement that the xml author may write the xml in such a way that it is validated in the editor.
Now, we need the consumer to be able to validate. In my case, I'm using Java.
If by some chance, you know the schema files that you will need to use to validate ahead of time, then you simply create a single Schema object and validate as usual, like this:
Schema schema = factory().newSchema(new Source[] {
new StreamSource(stream("foo.xsd")),
new StreamSource(stream("Alpha.xsd")),
new StreamSource(stream("Mercury.xsd")),
});
In this case, however, we don't know which xsd files to use until we have parsed the main document. So, the general procedure is to:
Validate the xml using only the main (foo) schema
Determine the schema to use to validate the portion of the document
Find the node that is the root of the portion to validate using a separate schema
Import that node into a brand new document
Validate the brand new document using the other schema file
Caveat: it appears that the document must be built namespace-aware in order for this to work.
Here's some code (this was ripped from various places of my code, so there might be some errors introduced by the copy-and-paste):
// Contains the filename of the xml file
String filename;
// Load the xml data using a namespace-aware builder (the method
// 'stream' simply opens an input stream on a file)
Document document;
DocumentBuilderFactory docBuilderFactory =
DocumentBuilderFactory.newInstance();
docBuilderFactory.setNamespaceAware(true);
document = docBuilderFactory.newDocumentBuilder().parse(stream(filename));
// Create the schema factory
SchemaFactory sFactory = SchemaFactory.newInstance(
XMLConstants.W3C_XML_SCHEMA_NS_URI);
// Load the main schema
Schema schema = sFactory.newSchema(
new StreamSource(stream("foo.xsd")));
// Validate using main schema
schema.newValidator().validate(new DOMSource(document));
// Get the node that is the root for the portion you want to validate
// using another schema
Node node= getSpecialNode(document);
// Build a Document from that node
Document subDocument = docBuilderFactory.newDocumentBuilder().newDocument();
subDocument.appendChild(subDocument.importNode(node, true));
// Determine the schema to use using your own logic
Schema subSchema = parseAndDetermineSchema(document);
// Validate using other schema
subSchema.newValidator().validate(new DOMSource(subDocument));

Take a look at NVDL (Namespace-based Validation Dispatching Language) - http://www.nvdl.org/
It is designed to do what you want to do (validate parts of an XML document that have their own namespaces and schemas).
There is a tutorial here - http://www.dpawson.co.uk/nvdl/ - and a Java implementation here - http://jnvdl.sourceforge.net/
Hope that helps!
Kevin

You need to define a target namespace for each separately-validated portions of the instance document. Then you define a master schema that uses <xsd:include> to reference the schema documents for these components.
The limitation with this approach is that you can't let the individual components define the schemas that should be used to validate them. But it's a bad idea in general to let a document tell you how to validate it (ie, validation should something that your application controls).

You can also use a "resource resolver" to allow "xml authors" to specify their own schema file, at least to some extent, ex: https://stackoverflow.com/a/41225329/32453 at the end of the day, you want a fully compliant xml file that can be validatable with normal tools, anyway :)