We Keep Coding

Converting XML not work UTF-8 XMLOutputter Java

I allready saw other questions about the same problem but i still get an error. Hier is the small part of code where i try to modify exosting xml files. But it modifies some characters in text.
import org.jdom2.Document;
import org.jdom2.JDOMException;
import org.jdom2.input.SAXBuilder;
import org.jdom2.output.Format;
import org.jdom2.output.XMLOutputter;
import java.io.FileOutputStream;
import java.io.IOException;
public class ModyfyXml {
public static void main(String[] args) throws JDOMException, IOException {
try {
SAXBuilder sax = new SAXBuilder();
Document doc = sax.build("F:\\c\\test.xml");
XMLOutputter xmlOutput = new XMLOutputter();
Format format = Format.getPrettyFormat();
format.setEncoding("UTF-8");
xmlOutput.setFormat(format);
xmlOutput.output(doc, (new FileOutputStream("F:\\c\\test2.xml")));
}catch (IOException io) {
io.printStackTrace();
} catch (JDOMException e) {
e.printStackTrace();
}
}}
Hier a small xml file that i try to modify (in this case just copy)
<?xml version="1.0" encoding="utf-8"?><page>
䕶法喇嘛所居此處𡸁仲無妻室亦降神附體
</page>
After program start i get the following:
<?xml version="1.0" encoding="UTF-8"?>
<page>䕶法喇嘛所居此處𡸁仲無妻室亦降神附體</page>
Some chineese characters can't be right transformed

Dang I never noticed this bug in JDOM 2.
You will have the same results with any non-BMP character. You can try with the emoji mania of these latest years and see you get the same results.
It happens because of the escape strategy automatically set for UTF-whatever encodings. What it does is rather wrong.
That will be fixed if you replace the strategy with one that doesn't escape anything beside XML reserved chars:
format.setEscapeStrategy((c) -> false);

Is there a way to add all the values of a key in the Arraylist from xml using Apache Commons Digester?

I have an XML file with the given key-value pair mapping:
<ParentChild>
<Parent>foo</Parent>
<Child>bar</Child>
<Child>bar1</Child>
<Child>bar2</Child>
<Child>bar3</Child>
</ParentChild>
I want to map it to the HashMap with Key Parent as String-Key and it's child as ArrayList Values as in:
HashMap(foo,(bar,bar1,bar2,bar3)) i.e. HashMap<String, ArrayList<String>>()
I am using Apache Commons Digester to get the key and values for other parameters with one key-one value relationship. But I am not sure how to map the List of Values for a single Key using digester.
I have recently started using Apache Commons Digester and any help would be great.
Thank you very much.

Here is a working sample:
import java.io.IOException;
import java.util.ArrayList;
import java.util.HashMap;
import org.apache.commons.digester3.Digester;
import org.xml.sax.SAXException;
public class SampleDigester {
public void run() throws IOException, SAXException {
Digester digester = new Digester();
digester.addObjectCreate("ParentChildren", HashMap.class);
digester.addCallMethod("ParentChildren/ParentChild", "put", 2);
digester.addCallParam("ParentChildren/ParentChild/Parent", 0);
digester.addCallParam("ParentChildren/ParentChild/Child", 1, 0);
digester.addObjectCreate("ParentChildren/ParentChild", ArrayList.class);
digester.addObjectCreate("ParentChildren/ParentChild/Child", StringBuffer.class);
digester.addCallMethod("ParentChildren/ParentChild/Child", "append", 1);
digester.addCallParam("ParentChildren/ParentChild/Child", 0);
digester.addSetNext("ParentChildren/ParentChild/Child", "add");
Object result = digester.parse("file:///path/to//input.xml");
System.out.println(result);
}
public static void main(String[] args) {
try {
new SampleDigester().run();
} catch (Exception e) {
e.printStackTrace();
}
}
}
The input.xml file:
<?xml version="1.0" encoding="UTF-8"?>
<ParentChildren>
<ParentChild>
<Parent>foo</Parent>
<Child>bar</Child>
<Child>bar1</Child>
<Child>bar2</Child>
<Child>bar3</Child>
</ParentChild>
<ParentChild>
<Parent>fooo</Parent>
<Child>barr</Child>
<Child>barr1</Child>
<Child>barr2</Child>
<Child>barr3</Child>
</ParentChild>
</ParentChildren>
And the output: {foo=[bar, bar1, bar2, bar3], fooo=[barr, barr1, barr2, barr3]}
I used this documentation

parsing an XML file with DOM error [duplicate]

I have to parse a bunch of XML files in Java that sometimes -- and invalidly -- contain HTML entities such as —, > and so forth. I understand the correct way of dealing with this is to add suitable entity declarations to the XML file before parsing. However, I can't do that as I have no control over those XML files.
Is there some kind of callback I can override that is invoked whenever the Java XML parser encounters such an entity? I haven't been able to find one in the API.
I'd like to use:
DocumentBuilderFactory dbf = DocumentBuilderFactory.newInstance();
DocumentBuilder parser = dbf.newDocumentBuilder();
Document doc = parser.parse( stream );
I found that I can override resolveEntity in org.xml.sax.helpers.DefaultHandler, but how do I use this with the higher-level API?
Here's a full example:
public class Main {
public static void main( String [] args ) throws Exception {
DocumentBuilderFactory dbf = DocumentBuilderFactory.newInstance();
DocumentBuilder parser = dbf.newDocumentBuilder();
Document doc = parser.parse( new FileInputStream( "test.xml" ));
}
}
with test.xml:
<?xml version="1.0" encoding="UTF-8"?>
<foo>
<bar>Some text — invalid!</bar>
</foo>
Produces:
[Fatal Error] :3:20: The entity "nbsp" was referenced, but not declared.
Exception in thread "main" org.xml.sax.SAXParseException; lineNumber: 3; columnNumber: 20; The entity "nbsp" was referenced, but not declared.
Update: I have been poking around in the JDK source code with a debugger, and boy, what an amount of spaghetti. I have no idea what the design is there, or whether there is one. Just how many layers of an onion can one layer on top of each other?
They key class seems to be com.sun.org.apache.xerces.internal.impl.XMLEntityManager, but I cannot find any code that either lets me add stuff into it before it gets used, or that attempts to resolve entities without going through that class.

I would use a library like Jsoup for this purpose. I tested the following below and it works. I don't know if this helps. It can be located here: http://jsoup.org/download
public static void main(String args[]){
String html = "<?xml version=\"1.0\" encoding=\"UTF-8\"?><foo>" +
"<bar>Some text — invalid!</bar></foo>";
Document doc = Jsoup.parse(html, "", Parser.xmlParser());
for (Element e : doc.select("bar")) {
System.out.println(e);
}
}
Result:
<bar>
Some text — invalid!
</bar>
Loading from a file can be found here:
http://jsoup.org/cookbook/input/load-document-from-file

Issue - 1: I have to parse a bunch of XML files in Java that sometimes -- and
invalidly -- contain HTML entities such as —
XML has only five predefined entities. The —, is not among them. It works only when used in plain HTML or in legacy JSP. So, SAX will not help. It can be done using StaX which has high level iterator based API. (Collected from this link)
Issue - 2: I found that I can override resolveEntity in
org.xml.sax.helpers.DefaultHandler, but how do I use this with the
higher-level API?
Streaming API for XML, called StaX, is an API for reading and writing XML Documents.
StaX is a Pull-Parsing model. Application can take the control over parsing the XML documents by pulling (taking) the events from the parser.
The core StaX API falls into two categories and they are listed below. They are
Cursor based API: It is low-level API. cursor-based API allows the application to process XML as a stream of tokens aka events
Iterator based API: The higher-level iterator-based API allows the application to process XML as a series of event objects, each of which communicates a piece of the XML structure to the application.
STaX API has support for the notion of not replacing character entity references, by way of the IS_REPLACING_ENTITY_REFERENCES property:
Requires the parser to replace internal entity references with their
replacement text and report them as characters
This can be set into an XmlInputFactory, which is then in turn used to construct an XmlEventReader or XmlStreamReader.
However, the API is careful to say that this property is only intended to force the implementation to perform the replacement, rather than forcing it to notreplace them.
You may try it. Hope it will solve your issue. For your case,
Main.java
import java.io.FileInputStream;
import java.io.FileNotFoundException;
import javax.xml.stream.XMLEventReader;
import javax.xml.stream.XMLInputFactory;
import javax.xml.stream.XMLStreamException;
import javax.xml.stream.events.EntityReference;
import javax.xml.stream.events.XMLEvent;
public class Main {
public static void main(String[] args) {
XMLInputFactory inputFactory = XMLInputFactory.newInstance();
inputFactory.setProperty(
XMLInputFactory.IS_REPLACING_ENTITY_REFERENCES, false);
XMLEventReader reader;
try {
reader = inputFactory
.createXMLEventReader(new FileInputStream("F://test.xml"));
while (reader.hasNext()) {
XMLEvent event = reader.nextEvent();
if (event.isEntityReference()) {
EntityReference ref = (EntityReference) event;
System.out.println("Entity Reference: " + ref.getName());
}
}
} catch (FileNotFoundException e) {
e.printStackTrace();
} catch (XMLStreamException e) {
e.printStackTrace();
}
}
}
test.xml:
<?xml version="1.0" encoding="UTF-8"?>
<foo>
<bar>Some text — invalid!</bar>
</foo>
Output:
Entity Reference: nbsp
Entity Reference: mdash
Credit goes to #skaffman.
Related Link:
http://www.journaldev.com/1191/how-to-read-xml-file-in-java-using-java-stax-api
http://www.journaldev.com/1226/java-stax-cursor-based-api-read-xml-example
http://www.vogella.com/tutorials/JavaXML/article.html
Is there a Java XML API that can parse a document without resolving character entities?
UPDATE:
Issue - 3: Is there a way to use StaX to "filter" the entities (replacing them
with something else, for example) and still produce a Document at the
end of the process?
To create a new document using the StAX API, it is required to create an XMLStreamWriter that provides methods to produce XML opening and closing tags, attributes and character content.
There are 5 methods of XMLStreamWriter for document.
xmlsw.writeStartDocument(); - initialises an empty document to which
elements can be added
xmlsw.writeStartElement(String s) -creates a new element named s
xmlsw.writeAttribute(String name, String value)- adds the attribute
name with the corresponding value to the last element produced by a
call to writeStartElement. It is possible to add attributes as long
as no call to writeElementStart,writeCharacters or writeEndElement
has been done.
xmlsw.writeEndElement - close the last started element
xmlsw.writeCharacters(String s) - creates a new text node with
content s as content of the last started element.
A sample example is attached with it:
StAXExpand.java
import java.io.BufferedReader;
import java.io.FileReader;
import java.io.IOException;
import javax.xml.stream.XMLOutputFactory;
import javax.xml.stream.XMLStreamException;
import javax.xml.stream.XMLStreamWriter;
import java.util.Arrays;
public class StAXExpand {
static XMLStreamWriter xmlsw = null;
public static void main(String[] argv) {
try {
xmlsw = XMLOutputFactory.newInstance()
.createXMLStreamWriter(System.out);
CompactTokenizer tok = new CompactTokenizer(
new FileReader(argv[0]));
String rootName = "dummyRoot";
// ignore everything preceding the word before the first "["
while(!tok.nextToken().equals("[")){
rootName=tok.getToken();
}
// start creating new document
xmlsw.writeStartDocument();
ignorableSpacing(0);
xmlsw.writeStartElement(rootName);
expand(tok,3);
ignorableSpacing(0);
xmlsw.writeEndDocument();
xmlsw.flush();
xmlsw.close();
} catch (XMLStreamException e){
System.out.println(e.getMessage());
} catch (IOException ex) {
System.out.println("IOException"+ex);
ex.printStackTrace();
}
}
public static void expand(CompactTokenizer tok, int indent)
throws IOException,XMLStreamException {
tok.skip("[");
while(tok.getToken().equals("#")) {// add attributes
String attName = tok.nextToken();
tok.nextToken();
xmlsw.writeAttribute(attName,tok.skip("["));
tok.nextToken();
tok.skip("]");
}
boolean lastWasElement=true; // for controlling the output of newlines
while(!tok.getToken().equals("]")){ // process content
String s = tok.getToken().trim();
tok.nextToken();
if(tok.getToken().equals("[")){
if(lastWasElement)ignorableSpacing(indent);
xmlsw.writeStartElement(s);
expand(tok,indent+3);
lastWasElement=true;
} else {
xmlsw.writeCharacters(s);
lastWasElement=false;
}
}
tok.skip("]");
if(lastWasElement)ignorableSpacing(indent-3);
xmlsw.writeEndElement();
}
private static char[] blanks = "\n".toCharArray();
private static void ignorableSpacing(int nb)
throws XMLStreamException {
if(nb>blanks.length){// extend the length of space array
blanks = new char[nb+1];
blanks[0]='\n';
Arrays.fill(blanks,1,blanks.length,' ');
}
xmlsw.writeCharacters(blanks, 0, nb+1);
}
}
CompactTokenizer.java
import java.io.Reader;
import java.io.IOException;
import java.io.StreamTokenizer;
public class CompactTokenizer {
private StreamTokenizer st;
CompactTokenizer(Reader r){
st = new StreamTokenizer(r);
st.resetSyntax(); // remove parsing of numbers...
st.wordChars('\u0000','\u00FF'); // everything is part of a word
// except the following...
st.ordinaryChar('\n');
st.ordinaryChar('[');
st.ordinaryChar(']');
st.ordinaryChar('#');
}
public String nextToken() throws IOException{
st.nextToken();
while(st.ttype=='\n'||
(st.ttype==StreamTokenizer.TT_WORD &&
st.sval.trim().length()==0))
st.nextToken();
return getToken();
}
public String getToken(){
return (st.ttype == StreamTokenizer.TT_WORD) ? st.sval : (""+(char)st.ttype);
}
public String skip(String sym) throws IOException {
if(getToken().equals(sym))
return nextToken();
else
throw new IllegalArgumentException("skip: "+sym+" expected but"+
sym +" found ");
}
}
For more, you can follow the tutorial
https://docs.oracle.com/javase/tutorial/jaxp/stax/example.html
http://www.ibm.com/developerworks/library/x-tipstx2/index.html
http://www.iro.umontreal.ca/~lapalme/ForestInsteadOfTheTrees/HTML/ch09s03.html
http://staf.sourceforge.net/current/STAXDoc.pdf

Another approach, since you're not using a rigid OXM approach anyway.
You might want to try using a less rigid parser such as JSoup?
This will stop immediate problems with invalid XML schemas etc, but it will just devolve the problem into your code.

Just to throw in a different approach to a solution:
You might envelope your input stream with a stream inplementation that replaces the entities by something legal.
While this is a hack for sure, it should be a quick and easy solution (or better say: workaround).
Not as elegant and clean as a xml framework internal solution, though.

I made yesterday something similar i need to add value from unziped XML in stream to database.
//import I'm not sure if all are necessary :)
import javax.xml.parsers.DocumentBuilder;
import javax.xml.parsers.DocumentBuilderFactory;
import javax.xml.parsers.ParserConfigurationException;
import javax.xml.xpath.*;
import org.w3c.dom.Document;
import org.xml.sax.InputSource;
import org.xml.sax.SAXException;
//I didnt checked this code now because i'm in work for sure its work maybe
you will need to do little changes
InputSource is = new InputSource(new FileInputStream("test.xml"));
DocumentBuilderFactory dbf = DocumentBuilderFactory.newInstance();
DocumentBuilder db = dbf.newDocumentBuilder();
Document doc = db.parse(is);
XPathFactory xpf = XPathFactory.newInstance();
XPath xpath = xpf.newXPath();
String words= xpath.evaluate("/foo/bar", doc.getDocumentElement());
ParsingHexToChar.parseToChar(words);
// lib which i use common-lang3.jar
//metod to parse
public static String parseToChar( String words){
String decode= org.apache.commons.lang3.StringEscapeUtils.unescapeHtml4(words);
return decode;
}

Try this using org.apache.commons package :
DocumentBuilderFactory dbf = DocumentBuilderFactory.newInstance();
DocumentBuilder parser = dbf.newDocumentBuilder();
InputStream in = new FileInputStream(xmlfile);
String unescapeHtml4 = IOUtils.toString(in);
CharSequenceTranslator obj = new AggregateTranslator(new LookupTranslator(EntityArrays.ISO8859_1_UNESCAPE()),
new LookupTranslator(EntityArrays.HTML40_EXTENDED_UNESCAPE())
);
unescapeHtml4 = obj.translate(unescapeHtml4);
StringReader readerInput= new StringReader(unescapeHtml4);
InputSource is = new InputSource(readerInput);
Document doc = parser.parse(is);

Flat file to XML in JAVA

I have a flat file of fixed length fields like this:
ITEM1234LED Light Set
ITEM1235Ratchet Tie
I'd like to convert it into xml file:
<ITEMS>
<ITEM>
<ITEMID>1234</ITEMID>
<DESCRIPTION>LED Light Set</DESCRIPTION>
</ITEM>
<ITEM>
<ITEMID>1235</ITEMID>
<DESCRIPTION>Ratchet Tie</DESCRIPTION>
</ITEM>
</ITEMS>
Which is the best way to achieve this ?
Thank you.

You can use a simple XMLStreamWriter to create the XML document. No need to create a class for the records. Just extract the ID and the description as strings and push these strings to the XML. This works for large files, too. Neither the input file nor the XML document has to be hold entirely in memory.
import java.io.BufferedOutputStream;
import java.io.BufferedReader;
import java.io.IOException;
import java.io.OutputStream;
import java.nio.charset.StandardCharsets;
import java.nio.file.Files;
import java.nio.file.Path;
import java.nio.file.Paths;
import javax.xml.stream.FactoryConfigurationError;
import javax.xml.stream.XMLOutputFactory;
import javax.xml.stream.XMLStreamException;
import javax.xml.stream.XMLStreamWriter;
public class Items {
private static final int POS_ID = 4;
private static final int POS_DESCR = 8;
public static void main(String[] args) {
// Files for input and output
final Path inFile = Paths.get("items.txt");
final Path outFile = Paths.get("items.xml");
// Unfortunately, XMLStreamWriter doesn't implement AutoCloseable,
// so we cannot use it with try-with-resources.
XMLStreamWriter xmlWriter = null;
try(
// BufferedReader for the input file (assuming UTF-8 for the encoding)
BufferedReader reader = Files.newBufferedReader(
inFile, StandardCharsets.UTF_8);
// BufferedOutputStream, so encoding is handled entirely by
// the XMLStreamWriter.
OutputStream out = new BufferedOutputStream(
Files.newOutputStream(outFile));
)
{
// Use a XMLStreamWriter to create the XML document.
xmlWriter = XMLOutputFactory.newInstance().createXMLStreamWriter(out);
xmlWriter.writeStartDocument();
xmlWriter.writeStartElement("ITEMS");
String line;
while((line = reader.readLine()) != null) {
// Parse the input line with fixed length fields
final String id = line.substring(POS_ID, POS_DESCR);
final String descr = line.substring(POS_DESCR);
xmlWriter.writeStartElement("ITEM");
xmlWriter.writeStartElement("ITEMID");
xmlWriter.writeCharacters(id);
xmlWriter.writeEndElement(); // ITEMID
xmlWriter.writeStartElement("DESCRIPTION");
xmlWriter.writeCharacters(descr);
xmlWriter.writeEndElement(); // DESCRIPTION
xmlWriter.writeEndElement(); // ITEM
}
xmlWriter.writeEndElement(); // ITEMS
xmlWriter.writeEndDocument();
} catch (IOException | XMLStreamException | FactoryConfigurationError e) {
e.printStackTrace();
} finally {
// Cleaning up
if(xmlWriter != null) {
try {
xmlWriter.close();
} catch (XMLStreamException e) {
e.printStackTrace();
}
}
}
}
}

1) Create a Java class, that maps to the data in the flat file, for ex:
public class Item {
private String itemId;
private String description;
/**
* #return the itemId
*/
public String getItemId() {
return itemId;
}
/**
* #param itemId the itemId to set
*/
public void setItemId(String itemId) {
this.itemId = itemId;
}
/**
* #return the description
*/
public String getDescription() {
return description;
}
/**
* #param description the description to set
*/
public void setDescription(String description) {
this.description = description;
}
}
2) Parse the flat file into a list of 'Items' (List of Item objects)
3) Use a good, lightweight framework like 'xStream' and use the appropriate method to serialize Java object to XML file. For example: xStream.toXml(Object obj, Writer out)
PS: This is just a standard way (using well tested frameworks and hence, not re-inventing the wheel), but not the optimal one. Optimally, for performance and less memory footprint, you can parse the flat file and write to a XML file at the same time.

Create java objects to represent your logical data structure.
Parse the flat file and generate the java objects.
Use an XML library (for example JAXB) to serialize that tree of java objects into a file.

You could use any of the following to achieve what you're trying:
JAXB
XSLT
Or you could use this to read a CSV or a flat file and serialize to an XML (as shown in your question)
Hope this helps!

I think what bchetty mentioned is good, but you do not need ANY XML libraries to output XML.
Parse the file into an Collection<> of objects, just like bchetty mentioned. You can use regex, or tools like ANTLR / CookCC / JFlex etc to do the parsing.
Then you can just open a PrintWriter for the file.
PrintWriter out = new PrintWriter (file);
out.println ("<ITEMS>");
for (Item item : Items)
{
out.println (" <ITEM>");
out.println (" <ITEMID>" + item.getItemId() + "</ITEMID>");
out.println (" <DESCRIPTION>" + item.getDescription () + "</DESCRIPTION>");
out.println (" </ITEM>");
}
out.println ("</ITEMS>");
out.close ();

JDOM, umaluts and UTF-8

I have issue with using utf-8 with JDOM parser. My code won't write encoded in utf-8.
Here's code:
import java.io.FileWriter;
import java.io.IOException;
import org.jdom2.Element;
import org.jdom2.output.Format;
import org.jdom2.output.XMLOutputter;
import org.jdom2.Document;
import org.jdom2.JDOMException;
import org.jdom2.input.SAXBuilder;
public class Jdom2 {
public static void main(String[] args) throws JDOMException, IOException {
String textValue = null;
try {
SAXBuilder sax = new SAXBuilder();
Document doc = sax.build("path/to/file.xml");
Element rootNode = doc.getRootElement();
Element application = rootNode.getChild("application");
Element de_DE = application.getChild("de_DE");
textValue = de_DE.getText();
System.out.println(textValue);
application.getChild("de_DE").setText(textValue);
Format format = Format.getPrettyFormat();
format.setEncoding("UTF-8");
XMLOutputter xmlOutput = new XMLOutputter(format);
xmlOutput.output(doc, new FileWriter("path/to/file.xml"));
} catch (IOException io) {
io.printStackTrace();
} catch (JDOMException e) {
e.printStackTrace();
}
}
}
So line System.out.println(textValue); prints text "Mit freundlichen Grüßen", which is exactly what is read from xml file, but when it writing down, I get "Mit freundlichen Gr����"
Here's my xml file
<?xml version="1.0" encoding="UTF-8"?>
<applications>
<application id="default">
<de_DE>Mit freundlichen Grüßen</de_DE>
</application>
</applications>
How will I achieve writing "Mit freundlichen Grüßen" instead of "Mit freundlichen Gr����" ?
Thanks in advance!

This is probably the issue:
xmlOutput.output(doc, new FileWriter("path/to/file.xml"));
FileWriter always uses the platform default encoding. If that's not UTF-8, you've got problems.
I suggest you use a FileOutputStream instead, and let JDOM do the right thing. (Also, I strongly suspect you should keep a reference to the stream so you can close it in a finally block...)

I would totally go for #Jon Skeet answer, but if the code is in the 3rd party library that you cannot modify there are two other options:
Run your JVM with additional environment variable specifying file.encoding:
java -Dfile.encoding=UTF-8 … Jdom2
You can try to modify the Charset.defaultCharset which stores encoding used by FileWritter. Keep in mind that it's a very hacky solution that might break at some point and I would avoid it at all:
Field field = null;
for (Field f : Charset.class.getDeclaredFields()) {
if (f.getName().equals("defaultCharset")) {
field = f;
}
}
field.setAccessible(true);
Field modifiersField = Field.class.getDeclaredField("modifiers");
modifiersField.setAccessible(true);
modifiersField.setInt(field, field.getModifiers() & ~Modifier.FINAL);
field.set(null, StandardCharsets.UTF_8);
One last remark. Both solution might introduce side effects in other parts of your application because you are changing default encoding that might be used by some libraries.