We Keep Coding

Java 'getLocalName()' returns null even with 'setNamespaceAware(true)'

I have the following sample xml file:
<?xml version="1.0" encoding="UTF-8"?>
<root attr1="value1"/>
The following Java sample demonstrates the issue I am facing:
import java.io.FileInputStream;
import java.io.InputStream;
import javax.xml.parsers.DocumentBuilderFactory;
import org.w3c.dom.Attr;
import org.w3c.dom.Document;
import org.w3c.dom.Element;
import org.w3c.dom.NamedNodeMap;
public class XMLClass {
public static void main(String[] args) throws Exception {
// path to xml file
String filename = "src/resources/xmlfile.xml";
DocumentBuilderFactory db = DocumentBuilderFactory.newInstance();
// this only helps for attr1 but not attr2
db.setNamespaceAware(true);
InputStream input = new FileInputStream(filename);
Document doc = db.newDocumentBuilder()
.parse(input);
Element root = doc.getDocumentElement();
// create an additional attribute
root.setAttribute("attr2", "value2");
NamedNodeMap nnm = root.getAttributes();
// The attribute name and value is correct
// for the attr1, however, the name for
// attr2 is null
for (int i = 0; i < nnm.getLength(); i++) {
Attr a = (Attr) nnm.item(i);
String name = a.getLocalName();
String value = a.getValue();
System.out.println("name: " + name + "; value: " + value);
}
System.exit(0);
}
}
The output is:
name: attr1; value: value1
name: null; value: value2
I have searched the Web and the only advise I found was to use setNamespaceAware(true), as I had already done in my code. This ensures that the attribute name for attr1, which is defined in the XML file, is returned correctly by getLocalName(). However, the attribute name attr2, which is set in the code via setAttribute() is null, although the value is correctly retrieved.
What is the reason for this behaviour and what is the proper way to solve my issue?

The documentation for getLocalName() says:
For nodes of any type other than ELEMENT_NODE and ATTRIBUTE_NODE and nodes created with a DOM Level 1 method, such as Document.createElement(), this is always null.
And the documentation for setAttribute says:
To set an attribute with a qualified name and namespace URI, use the setAttributeNS method.
So, setting an attribute value will not set the local name unless you explicitly set a namespaced attribute with setAttributeNS:
root.setAttributeNS(null, "attr2", "value2");

Not all documents are inserted in MongoDB when using the async-driver for Java

I was experimenting with the mongodb-async driver(http://mongodb.github.io/mongo-java-driver/3.0/driver-async/) and noticed odd behaviour. I reproduced the weird behaviour in underlying code:
import com.mongodb.async.SingleResultCallback;
import com.mongodb.async.client.MongoClient;
import com.mongodb.async.client.MongoClients;
import com.mongodb.async.client.MongoCollection;
import com.mongodb.async.client.MongoDatabase;
import org.bson.Document;
public class main {
public static void main(String [] args)
{
MongoClient mongoClient = MongoClients.create();
MongoDatabase database = mongoClient.getDatabase("mongotest");
MongoCollection<Document> collection = database.getCollection("coll");
for(Integer i = 0; i < 100000; i++) {
Document doc = new Document("name"+ i.toString(), "TESTING");
collection.insertOne(doc, new SingleResultCallback<Void>() {
public void onResult(final Void result, final Throwable t) {
System.out.println("Inserted!");
}
});
}
while(true){
}
}
}
I would expect this code to insert 100.000 documents into the collection 'coll' of the mongo-database called "mongotest".
However, when I check the number of elements after running this code, thousands of documents are missing.
When running this statement in the mongodb-shell
db.getCollection("coll").count()
I get 93062 as a result. This number varies for each run but never gets up to 100.000. Can anyone explain why not all objects are properly stored as documents in the MongoDB when I use this code? We tested this on 3 different machines and every machine exposed the same behaviour.
I have the feeling it is a driver-related issue because following up on this I wrote a similar experiment using node.js:
var express = require('express');
var MongoClient = require('mongodb').MongoClient;
var app = express();
var url = 'mongodb://localhost:27017/mongotest';
MongoClient.connect(url, function (err, db) {
for (var i = 0; i < 100000; i++) {
var name = "name" + i;
db.collection("coll").insertOne({
name: name
},function(err,results) {
if(err==null) {
console.log("Sweet");
}
});
}
});
module.exports = app;
This code took longer to run in comparison to the java-code but when the code finishes, 100.000 documents are sitting in the collection as expected.
Can anyone explain why this is not the case with the java-example, and possibly provide a solution?

When did you run db.getCollection("coll").count() to check the insert result?
Maybe the insertion process has not finished when you check the result.
2016-02-19 15:00 edit
I did the same test, and had the same result.
but when I change the following line
Document doc = new Document("name"+ i.toString(), "TESTING");
to
Document doc = new Document("_id", "name"+ i.toString());
It inserted exactly 100000 docoments.

Trying to use jSoup to scrape data from a table

First time poster and fairly new coder, so please go easy on me. I'm trying to use jSoup to scrape data from a table. However, I'm having a couple problems:
1) I'm using NetBeans. I get a "stop" error on Line 30 (Elements tds...) that says cannot find symbol symbol method getElementsByTag. I'm confused because I thought I imported the correct package, and I use the same code a couple lines above and get no error.
2) When I run the code, I get an error that says:
Exception in thread "main" java.lang.NullPointerException
at mytest.JsoupTest1.main(JsoupTest1.java:26)
Which I thought means that a variable with a value of NULL is being used. Did I incorrectly enter the "row" variable in my for loop above?
Here's my code. I truly appreciate any help!
package mytest;
import java.io.IOException;
import org.jsoup.Jsoup;
import org.jsoup.nodes.Document;
import org.jsoup.nodes.Element;
import org.jsoup.select.Elements;
public class JsoupTest1 {
private static Object row;
public static void main(String[] args) {
Document doc = null;
try {
doc = Jsoup.connect( "http://www.fangraphs.com/leaders.aspx?pos=all&stats=bat&lg=all&qual=0&type=8&season=2015&month=0&season1=2015&ind=0&team=18&rost=0&age=0&filter=&players=0" ).get();
}
catch (IOException ioe) {
ioe.printStackTrace();
}
Element table = doc.getElementById( "LeaderBoard1_dg1_ct100" );
Elements rows = table.getElementsByTag( "tr" );
for( Element row:rows ) {
}
Elements tds = row.getElementsByTag( "td" );
for( int i=0; i < tds.size(); i++ ) {
System.out.println(tds.get(i).text());
}
}
}

Welcome to StackOverflow.
This works.
Document doc = null;
try {
doc = Jsoup
.connect(
"http://www.fangraphs.com/leaders.aspx?pos=all&stats=bat&lg=all&qual=0&type=8&season=2015&month=0&season1=2015&ind=0&team=18&rost=0&age=0&filter=&players=0")
.get();
}
catch (IOException ioe) {
ioe.printStackTrace();
}
Element table = doc.getElementById("LeaderBoard1_dg1_ctl00");
Elements rows = table.getElementsByTag("tr");
for (Element row : rows) {
Elements tds = row.getElementsByTag("td");
for (int i = 0; i < tds.size(); i++) {
System.out.println(tds.get(i).text());
}
}
There are three problems with your code.
The id you are using is wrong. Instead of LeaderBoard1_dg1_ct100 use LeaderBoard1_dg1_ctl00. You mistook the l for 1.
The second problem is the Object row. No need for this one. Remove it.
You had the iteration of the rows outside of the for loop. And because you had the Object row variable, no compilation errors where present, thus hiding the problem.

Compare two XML strings ignoring element order

Suppose I have two xml strings
<test>
<elem>a</elem>
<elem>b</elem>
</test>
<test>
<elem>b</elem>
<elem>a</elem>
</test>
How to write a test that compares those two strings and ignores the element order?
I want the test to be as short as possible, no place for 10-line XML parsing etc. I'm looking for a simple assertion or something similar.
I have this (which doesn't work)
Diff diff = XMLUnit.compareXML(expectedString, actualString);
XMLAssert.assertXMLEqual("meh", diff, true);

For xmlunit 2.0 (I was looking for this) it is now done, by using DefaultNodeMatcher
Diff diff = Diffbuilder.compare(Input.fromFile(control))
.withTest(Input.fromFile(test))
.withNodeMatcher(new DefaultNodeMatcher(ElementSelectors.byNameAndText))
.build()
Hope this helps this helps other people googling...

XMLUnit will do what you want, but you have to specify the elementQualifier. With no elementQualifier specified it will only compare the nodes in the same position.
For your example you want an ElementNameAndTextQualifer, this considers a node similar if one exists that matches the element name and it's text value, something like :
Diff diff = new Diff(control, toTest);
// we don't care about ordering
diff.overrideElementQualifier(new ElementNameAndTextQualifier());
XMLAssert.assertXMLEqual(diff, true);
You can read more about it here: http://xmlunit.sourceforge.net/userguide/html/ar01s03.html#ElementQualifier

My original answer is outdated. If I would have to build it again i would use xmlunit 2 and xmlunit-matchers. Please note that for xml unit a different order is always 'similar' not equals.
#Test
public void testXmlUnit() {
String myControlXML = "<test><elem>a</elem><elem>b</elem></test>";
String expected = "<test><elem>b</elem><elem>a</elem></test>";
assertThat(myControlXML, isSimilarTo(expected)
.withNodeMatcher(new DefaultNodeMatcher(ElementSelectors.byNameAndText)));
//In case you wan't to ignore whitespaces add ignoreWhitespace().normalizeWhitespace()
assertThat(myControlXML, isSimilarTo(expected)
.ignoreWhitespace()
.normalizeWhitespace()
.withNodeMatcher(new DefaultNodeMatcher(ElementSelectors.byNameAndText)));
}
If somebody still want't to use a pure java implementation here it is. This implementation extracts the content from xml and compares the list ignoring order.
public static Document loadXMLFromString(String xml) throws Exception {
DocumentBuilderFactory factory = DocumentBuilderFactory.newInstance();
DocumentBuilder builder = factory.newDocumentBuilder();
InputSource is = new InputSource(new StringReader(xml));
return builder.parse(is);
}
#Test
public void test() throws Exception {
Document doc = loadXMLFromString("<test>\n" +
" <elem>b</elem>\n" +
" <elem>a</elem>\n" +
"</test>");
XPathFactory xPathfactory = XPathFactory.newInstance();
XPath xpath = xPathfactory.newXPath();
XPathExpression expr = xpath.compile("//test//elem");
NodeList all = (NodeList) expr.evaluate(doc, XPathConstants.NODESET);
List<String> values = new ArrayList<>();
if (all != null && all.getLength() > 0) {
for (int i = 0; i < all.getLength(); i++) {
values.add(all.item(i).getTextContent());
}
}
Set<String> expected = new HashSet<>(Arrays.asList("a", "b"));
assertThat("List equality without order",
values, containsInAnyOrder(expected.toArray()));
}

Cross-posting from Compare XML ignoring order of child elements
I had a similar need this evening, and couldn't find something that fit my requirements.
My workaround was to sort the two XML files I wanted to diff, sorting alphabetically by the element name. Once they were both in a consistent order, I could diff the two sorted files using a regular visual diff tool.
If this approach sounds useful to anyone else, I've shared the python script I wrote to do the sorting at http://dalelane.co.uk/blog/?p=3225

Just as an example of how to compare more complex xml elements matching based on equality of attribute name. For instance:
<request>
<param name="foo" style="" type="xs:int"/>
<param name="Cookie" path="cookie" style="header" type="xs:string" />
</request>
vs.
<request>
<param name="Cookie" path="cookie" style="header" type="xs:string" />
<param name="foo" style="query" type="xs:int"/>
</request>
With following custom element qualifier:
final Diff diff = XMLUnit.compareXML(controlXml, testXml);
diff.overrideElementQualifier(new ElementNameAndTextQualifier() {
#Override
public boolean qualifyForComparison(final Element control, final Element test) {
// this condition is copied from super.super class
if (!(control != null && test != null
&& equalsNamespace(control, test)
&& getNonNamespacedNodeName(control).equals(getNonNamespacedNodeName(test)))) {
return false;
}
// matching based on 'name' attribute
if (control.hasAttribute("name") && test.hasAttribute("name")) {
if (control.getAttribute("name").equals(test.getAttribute("name"))) {
return true;
}
}
return false;
}
});
XMLAssert.assertXMLEqual(diff, true);

For me, I also needed to add the method : checkForSimilar() on the DiffBuilder.
Without it, the assert was in error saying that the sequence of the nodes was not the same (the position in the child list was not the same)
My code was :
Diff diff = Diffbuilder.compare(Input.fromFile(control))
.withTest(Input.fromFile(test))
.withNodeMatcher(new DefaultNodeMatcher(ElementSelectors.byNameAndText))
.checkForSimilar()
.build()

I don't know what versions they took for the solutions, but nothing worked (or was simple at least) so here's my solution for who had the same pains.
P.S. I hate people to miss the imports or the FQN class names of static methods
#Test
void given2XMLS_are_different_elements_sequence_with_whitespaces(){
String testXml = "<struct><int>3</int> <boolean>false</boolean> </struct>";
String expected = "<struct><boolean>false</boolean><int>3</int></struct>";
XmlAssert.assertThat(testXml).and(expected)
.ignoreWhitespace()
.normalizeWhitespace()
.withNodeMatcher(new DefaultNodeMatcher(ElementSelectors.byNameAndText))
.areSimilar();
}

OPTION 1
If the XML code is simple, try this:
String testString = ...
assertTrue(testString.matches("(?m)^<test>(\\s*<elem>(a|b)</elem>\\s*){2}</test>$"));
OPTION 2
If the XML is more elaborate, load it with an XML parser and compare the actual nodes found with you reference nodes.

I end up rewriting the xml and comparing it back. Let me know if it helps any of you who stumbled on this similar issue.
import org.apache.commons.lang3.StringUtils;
import org.jdom2.Attribute;
import org.jdom2.Document;
import org.jdom2.Element;
import org.jdom2.input.SAXBuilder;
import org.jdom2.output.Format;
import org.jdom2.output.XMLOutputter;
import java.io.ByteArrayInputStream;
import java.nio.charset.StandardCharsets;
import java.util.List;
import java.util.stream.Collectors;
public class XmlRewriter {
private static String rewriteXml(String xml) throws Exception {
SAXBuilder builder = new SAXBuilder();
Document document = builder.build(new ByteArrayInputStream(xml.getBytes(StandardCharsets.UTF_8)));
Element root = document.getRootElement();
XMLOutputter xmlOutputter = new XMLOutputter(Format.getPrettyFormat());
root.sortChildren((o1, o2) -> {
if(!StringUtils.equals(o1.getName(), o2.getName())){
return o1.getName().compareTo(o2.getName());
}
// get attributes
int attrCompare = transformToStr(o1.getAttributes()).compareTo(transformToStr(o2.getAttributes()));
if(attrCompare!=0){
return attrCompare;
}
if(o1.getValue()!=null && o2.getValue()!=null){
return o1.getValue().compareTo(o2.getValue());
}
return 0;
});
return xmlOutputter.outputString(root);
}
private static String transformToStr(List<Attribute> attributes){
return attributes.stream().map(e-> e.getName()+":"+e.getValue()).sorted().collect(Collectors.joining(","));
}
public static boolean areXmlSimilar(String xml1, String xml2) throws Exception {
Diff diff = DiffBuilder.compare(rewriteXml(xml1)).withTest(rewriteXml(xml2))
.normalizeWhitespace()
.ignoreWhitespace()
.ignoreComments()
.checkForSimilar()
.withNodeMatcher(new DefaultNodeMatcher(ElementSelectors.byNameAndText))
.build();
return !diff.hasDifferences();
}
// move below into another test class..
#Test
public void compareXml() throws Exception {
String xml1 = "<<your first XML str here>>";
String xml2 = "<<another XML str here>>";
assertTrue(XmlUtil.areXmlSimilar(xml1, xml2));
}
}

How to get node contents from JDOM

I'm writing an application in java using import org.jdom.*;
My XML is valid,but sometimes it contains HTML tags. For example, something like this:
<program-title>Anatomy & Physiology</program-title>
<overview>
<content>
For more info click here
<p>Learn more about the human body. Choose from a variety of Physiology (A&P) designed for complementary therapies.&#160; Online studies options are available.</p>
</content>
</overview>
<key-information>
<category>Health & Human Services</category>
So my problem is with the < p > tags inside the overview.content node.
I was hoping that this code would work :
Element overview = sds.getChild("overview");
Element content = overview.getChild("content");
System.out.println(content.getText());
but it returns blank.
How do I return all the text ( nested tags and all ) from the overview.content node ?
Thanks

content.getText() gives immediate text which is only useful fine with the leaf elements with text content.
Trick is to use org.jdom.output.XMLOutputter ( with text mode CompactFormat )
public static void main(String[] args) throws Exception {
SAXBuilder builder = new SAXBuilder();
String xmlFileName = "a.xml";
Document doc = builder.build(xmlFileName);
Element root = doc.getRootElement();
Element overview = root.getChild("overview");
Element content = overview.getChild("content");
XMLOutputter outp = new XMLOutputter();
outp.setFormat(Format.getCompactFormat());
//outp.setFormat(Format.getRawFormat());
//outp.setFormat(Format.getPrettyFormat());
//outp.getFormat().setTextMode(Format.TextMode.PRESERVE);
StringWriter sw = new StringWriter();
outp.output(content.getContent(), sw);
StringBuffer sb = sw.getBuffer();
System.out.println(sb.toString());
}
Output
For more info clickhere<p>Learn more about the human body. Choose from a variety of Physiology (A&P) designed for complementary therapies.&#160; Online studies options are available.</p>
Do explore other formatting options and modify above code to your need.
"Class to encapsulate XMLOutputter format options. Typical users can use the standard format configurations obtained by getRawFormat() (no whitespace changes), getPrettyFormat() (whitespace beautification), and getCompactFormat() (whitespace normalization). "

You could try using method getValue() for the closest approximation, but what this does is concatenate all text within the element and descendants together. This won't give you the <p> tag in any form. If that tag is in your XML like you've shown, it has become part of the XML markup. It'd need to be included as <p> or embedded in a CDATA section to be treated as text.
Alternatively, if you know all elements that either may or may not appear in your XML, you could apply an XSLT transformation that turns stuff which isn't intended as markup into plain text.

Well, maybe that's what you need:
import java.io.StringReader;
import org.custommonkey.xmlunit.XMLTestCase;
import org.custommonkey.xmlunit.XMLUnit;
import org.jdom.input.SAXBuilder;
import org.jdom.output.XMLOutputter;
import org.testng.annotations.Test;
import org.xml.sax.InputSource;
public class HowToGetNodeContentsJDOM extends XMLTestCase
{
private static final String XML = "<root>\n" +
" <program-title>Anatomy & Physiology</program-title>\n" +
" <overview>\n" +
" <content>\n" +
" For more info click here\n" +
" <p>Learn more about the human body. Choose from a variety of Physiology (A&P) designed for complementary therapies.&#160; Online studies options are available.</p>\n" +
" </content>\n" +
" </overview>\n" +
" <key-information>\n" +
" <category>Health & Human Services</category>\n" +
" </key-information>\n" +
"</root>";
private static final String EXPECTED = "For more info click here\n" +
"<p>Learn more about the human body. Choose from a variety of Physiology (A&P) designed for complementary therapies.&#160; Online studies options are available.</p>";
#Test
public void test() throws Exception
{
XMLUnit.setIgnoreWhitespace(true);
Document document = new SAXBuilder().build(new InputSource(new StringReader(XML)));
List<Content> content = document.getRootElement().getChild("overview").getChild("content").getContent();
String out = new XMLOutputter().outputString(content);
assertXMLEqual("<root>" + EXPECTED + "</root>", "<root>" + out + "</root>");
}
}
Output:
PASSED: test on instance null(HowToGetNodeContentsJDOM)
===============================================
Default test
Tests run: 1, Failures: 0, Skips: 0
===============================================
I am using JDom with generics: http://www.junlu.com/list/25/883674.html
Edit: Actually that's not that much different from Prashant Bhate's answer. Maybe you need to tell us what you are missing...

If you're also generating the XML file you should be able to encapsulate your html data in <![CDATA[]]> so that it isn't parsed by the XML parser.

The problem is that the <content> node doesn't have a text child; it has a <p> child that happens to contain text.
Try this:
Element overview = sds.getChild("overview");
Element content = overview.getChild("content");
Element p = content.getChild("p");
System.out.println(p.getText());
If you want all the immediate child nodes, call p.getChildren(). If you want to get ALL the child nodes, you'll have to call it recursively.

Not particularly pretty but works fine (using JDOM API):
public static String getRawText(Element element) {
if (element.getContent().size() == 0) {
return "";
}
StringBuffer text = new StringBuffer();
for (int i = 0; i < element.getContent().size(); i++) {
final Object obj = element.getContent().get(i);
if (obj instanceof Text) {
text.append( ((Text) obj).getText() );
} else if (obj instanceof Element) {
Element e = (Element) obj;
text.append( "<" ).append( e.getName() );
// dump all attributes
for (Attribute attribute : (List<Attribute>)e.getAttributes()) {
text.append(" ").append(attribute.getName()).append("=\"").append(attribute.getValue()).append("\"");
}
text.append(">");
text.append( getRawText( e )).append("</").append(e.getName()).append(">");
}
}
return text.toString();
}
Prashant Bhate's solution is nicer though!

If you want to output the content of some JSOM node just use
System.out.println(new XMLOutputter().outputString(node))