We Keep Coding

XPath and SOAP Body node replace with string [duplicate]

How does XPath deal with XML namespaces?
If I use
/IntuitResponse/QueryResponse/Bill/Id
to parse the XML document below I get 0 nodes back.
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<IntuitResponse xmlns="http://schema.intuit.com/finance/v3"
time="2016-10-14T10:48:39.109-07:00">
<QueryResponse startPosition="1" maxResults="79" totalCount="79">
<Bill domain="QBO" sparse="false">
<Id>=1</Id>
</Bill>
</QueryResponse>
</IntuitResponse>
However, I'm not specifying the namespace in the XPath (i.e. http://schema.intuit.com/finance/v3 is not a prefix of each token of the path). How can XPath know which Id I want if I don't tell it explicitly? I suppose in this case (since there is only one namespace) XPath could get away with ignoring the xmlns entirely. But if there are multiple namespaces, things could get ugly.

XPath 1.0/2.0
Defining namespaces in XPath (recommended)
XPath itself doesn't have a way to bind a namespace prefix with a namespace. Such facilities are provided by the hosting library.
It is recommended that you use those facilities and define namespace prefixes that can then be used to qualify XML element and attribute names as necessary.
Here are some of the various mechanisms which XPath hosts provide for specifying namespace prefix bindings to namespace URIs.
(OP's original XPath, /IntuitResponse/QueryResponse/Bill/Id, has been elided to /IntuitResponse/QueryResponse.)
C#:
XmlNamespaceManager nsmgr = new XmlNamespaceManager(doc.NameTable);
nsmgr.AddNamespace("i", "http://schema.intuit.com/finance/v3");
XmlNodeList nodes = el.SelectNodes(#"/i:IntuitResponse/i:QueryResponse", nsmgr);
Google Docs:
Unfortunately, IMPORTXML() does not provide a namespace prefix binding mechanism. See next section, Defeating namespaces in XPath, for how to use local-name() as a work-around.
Java (SAX):
NamespaceSupport support = new NamespaceSupport();
support.pushContext();
support.declarePrefix("i", "http://schema.intuit.com/finance/v3");
Java (XPath):
xpath.setNamespaceContext(new NamespaceContext() {
public String getNamespaceURI(String prefix) {
switch (prefix) {
case "i": return "http://schema.intuit.com/finance/v3";
// ...
}
});
Remember to call
DocumentBuilderFactory.setNamespaceAware(true).
See also:
Java XPath: Queries with default namespace xmlns
JavaScript:
See Implementing a User Defined Namespace Resolver:
function nsResolver(prefix) {
var ns = {
'i' : 'http://schema.intuit.com/finance/v3'
};
return ns[prefix] || null;
}
document.evaluate( '/i:IntuitResponse/i:QueryResponse',
document, nsResolver, XPathResult.ANY_TYPE,
null );
Note that if the default namespace has an associated namespace prefix defined, using the nsResolver() returned by Document.createNSResolver() can obviate the need for a customer nsResolver().
Perl (LibXML):
my $xc = XML::LibXML::XPathContext->new($doc);
$xc->registerNs('i', 'http://schema.intuit.com/finance/v3');
my #nodes = $xc->findnodes('/i:IntuitResponse/i:QueryResponse');
Python (lxml):
from lxml import etree
f = StringIO('<IntuitResponse>...</IntuitResponse>')
doc = etree.parse(f)
r = doc.xpath('/i:IntuitResponse/i:QueryResponse',
namespaces={'i':'http://schema.intuit.com/finance/v3'})
Python (ElementTree):
namespaces = {'i': 'http://schema.intuit.com/finance/v3'}
root.findall('/i:IntuitResponse/i:QueryResponse', namespaces)
Python (Scrapy):
response.selector.register_namespace('i', 'http://schema.intuit.com/finance/v3')
response.xpath('/i:IntuitResponse/i:QueryResponse').getall()
PhP:
Adapted from #Tomalak's answer using DOMDocument:
$result = new DOMDocument();
$result->loadXML($xml);
$xpath = new DOMXpath($result);
$xpath->registerNamespace("i", "http://schema.intuit.com/finance/v3");
$result = $xpath->query("/i:IntuitResponse/i:QueryResponse");
See also #IMSoP's canonical Q/A on PHP SimpleXML namespaces.
Ruby (Nokogiri):
puts doc.xpath('/i:IntuitResponse/i:QueryResponse',
'i' => "http://schema.intuit.com/finance/v3")
Note that Nokogiri supports removal of namespaces,
doc.remove_namespaces!
but see the below warnings discouraging the defeating of XML namespaces.
VBA:
xmlNS = "xmlns:i='http://schema.intuit.com/finance/v3'"
doc.setProperty "SelectionNamespaces", xmlNS
Set queryResponseElement =doc.SelectSingleNode("/i:IntuitResponse/i:QueryResponse")
VB.NET:
xmlDoc = New XmlDocument()
xmlDoc.Load("file.xml")
nsmgr = New XmlNamespaceManager(New XmlNameTable())
nsmgr.AddNamespace("i", "http://schema.intuit.com/finance/v3");
nodes = xmlDoc.DocumentElement.SelectNodes("/i:IntuitResponse/i:QueryResponse",
nsmgr)
SoapUI (doc):
declare namespace i='http://schema.intuit.com/finance/v3';
/i:IntuitResponse/i:QueryResponse
xmlstarlet:
-N i="http://schema.intuit.com/finance/v3"
XSLT:
<xsl:stylesheet version="1.0"
xmlns:xsl="http://www.w3.org/1999/XSL/Transform"
xmlns:i="http://schema.intuit.com/finance/v3">
...
Once you've declared a namespace prefix, your XPath can be written to use it:
/i:IntuitResponse/i:QueryResponse
Defeating namespaces in XPath (not recommended)
An alternative is to write predicates that test against local-name():
/*[local-name()='IntuitResponse']/*[local-name()='QueryResponse']
Or, in XPath 2.0:
/*:IntuitResponse/*:QueryResponse
Skirting namespaces in this manner works but is not recommended because it
Under-specifies the full element/attribute name.
Fails to differentiate between element/attribute names in different
namespaces (the very purpose of namespaces). Note that this concern could be addressed by adding an additional predicate to check the namespace URI explicitly:
/*[ namespace-uri()='http://schema.intuit.com/finance/v3'
and local-name()='IntuitResponse']
/*[ namespace-uri()='http://schema.intuit.com/finance/v3'
and local-name()='QueryResponse']
Thanks to Daniel Haley for the namespace-uri() note.
Is excessively verbose.
XPath 3.0/3.1
Libraries and tools that support modern XPath 3.0/3.1 allow the specification of a namespace URI directly in an XPath expression:
/Q{http://schema.intuit.com/finance/v3}IntuitResponse/Q{http://schema.intuit.com/finance/v3}QueryResponse
While Q{http://schema.intuit.com/finance/v3} is much more verbose than using an XML namespace prefix, it has the advantage of being independent of the namespace prefix binding mechanism of the hosting library. The Q{} notation is known as Clark Notation after its originator, James Clark. The W3C XPath 3.1 EBNF grammar calls it a BracedURILiteral.
Thanks to Michael Kay for the suggestion to cover XPath 3.0/3.1's BracedURILiteral.

I use /*[name()='...'] in a google sheet to fetch some counts from Wikidata. I have a table like this
thes WD prop links items
NOM P7749 3925 3789
AAT P1014 21157 20224
and the formulas in cols links and items are
=IMPORTXML("https://query.wikidata.org/sparql?query=SELECT(COUNT(*)as?c){?item wdt:"&$B14&"[]}","//*[name()='literal']")
=IMPORTXML("https://query.wikidata.org/sparql?query=SELECT(COUNT(distinct?item)as?c){?item wdt:"&$B14&"[]}","//*[name()='literal']")
respectively. The SPARQL query happens not to have any spaces...
I saw name() used instead of local-name() in Xml Namespace breaking my xpath!, and for some reason //*:literal doesn't work.

What is a good approach to verify XML response from RESTful service in Java?

I am performing simple RESTFUL service API verification in Java.
To handle response in JSON format is very convenient. Using org.json library, it's easy to convert JSON string from RESTFUL response into JSON object, and compare it with that of the expected JSON string.
JSONObject response = new JSONObject(json_response_str);
JSONObject expected = new JSONObject(json_expected_str);
JSONAssert.assertEquals(expected, response, JSONCompareMode.LENIENT);
If it is some element of the JSON response that need to compare, it is also easy because it is easy to extract sub element from JSONObject using APIs like:
JSONObject element_in_response = response.get("..."); or
JSONObject element_in_response = response.getJSONObject("...");
However, to handle response in XML format, things are more difficult. To compare the whole XML response with expected XML is not bad, I can use XMLUnit to do it:
String xml_response_str = ...
String xml_expected_str = ...
assertXMLEquals(xml_response_str, xml_expected_str);
However, there's no such things like xmlOject as there is in JSON.
So what do I do if want to compare some element of the XML response with expected?
I've search forums and JAXB is sometimes mentioned. I checked and it is about parsing XML to Java object. So am I supposed to parse both response XML string and expected XML string, then extract the element as Java object, then compare them? It seems complicated, not to mention I need the XML schema to start with.
What is the effective way to do this, is there anything that is as convenient as in the case of JSON?
Thanks,

You can try to use XPATH.
There is a short example.
Here is XML string:
<?xml version="1.0" encoding="UTF-8"?>
<resp>
<status>good</status>
<msg>hi</msg>
</resp>
The folowing code will get status and message:
String xml = "<resp><status>good</status><msg>hi</msg></resp>";
XPathFactory xpathFactory = XPathFactory.newInstance();
XPath xpath = xpathFactory.newXPath();
InputSource source = new InputSource(new StringReader(xml));
Document doc = (Document) xpath.evaluate("/", source, XPathConstants.NODE);
String status = xpath.evaluate("/resp/status", doc);
String msg = xpath.evaluate("/resp/msg", doc);
System.out.println("status=" + status);
System.out.println("Message=" + msg);
Here is more examples about how to use XPATH:
http://viralpatel.net/blogs/java-xml-xpath-tutorial-parse-xml/

There are a number of ways for testing XML. Converting XML to JSON not being one of them, but can be done.
Testing XML is usually performed using XPath style comparisons which focus on elements, attributes and content and not so much on comparing chunks.
From looking at your code you're already familiar with XML assertions from http://xmlunit.sourceforge.net/api/org/custommonkey/xmlunit/XMLAssert.htm but you might also want to look at http://www.w3schools.com/xsl/xpath_intro.asp.
XML validation is not that easy and does require a lot of effort to begin with. Once you've got your tesing tools in order it gets a whole lot easier.

verify (or extract) XML is independent from protocol, RESTful service, etc. , but it is normally used in SOAP services.
Comparison with JSON is interesting. JSON is more easy to use with php, javascript, ...
If you want to connect two java servers, XML is sufficient, or plain java Objects (not portable solution with other languages).
Better point to use XML: it is more, more powerfull, well standardized, and you have lot of tools to process it.
What you are asking: equivalent of JSONobject exists in XML for a while: it is a DOM document, or a Node.
1 read your XML
String xml="<root>content</root>";
DocumentBuilderFactory builderFactory =DocumentBuilderFactory.newInstance();
DocumentBuilder builder = builderFactory.newDocumentBuilder();
// PARSE
Document document = builder.parse(new InputSource(new StringReader(xml)));
2 Best way to get some particular data: XPath: you give some path to your datas (root/group/class1/other_group/...), you can put wildcards (*), select about parameters, values, etc.
see this:
How to read XML using XPath in Java
XPath xpath = XPathFactory.newInstance().newXPath();
String expression="/root";
3 you can get direct values
expression="/root/text()";
String value = xpath.evaluate(expression, document);
4 or you get all data (if several)
XPathExpression expr = xpath.compile(expression) ;
NodeList nodes = (NodeList) expr.evaluate(document, XPathConstants.NODESET);
for (int i = 0; i < nodes.getLength(); i++)
{
Node nodeSegment = nodes.item(i);
if (nodeSegment.getNodeType() == Node.ELEMENT_NODE)
{
Element eElement = (Element) nodeSegment;
System.out.println("TAG="+eElement.getTagName());
System.out.println("VALUE="+eElement.getNodeValue());

How to do a XSL transform in Java using a not namespace aware parser?

I use tagsoup as (SAX) XMLREader and set the namespace feature to false. This parser is used to feed the Transformer as SAX Source. Complete code:
final TransformerFactory factory = TransformerFactory.newInstance();
final Transformer t = factory.newTransformer(new StreamSource(
getClass().getResourceAsStream("/identity.xsl")));
final XMLReader p = new Parser(); // the tagsoup parser
p.setFeature("http://xml.org/sax/features/namespaces", false);
// getHtml() returns HTML as InputStream
final Source source = new SAXSource(p, new InputSource(getHtml()));
t.transform(source, new StreamResult(System.out));
This results in something like:
< xmlns:html="http://www.w3.org/1999/xhtml">
<>
<>
<>
<>
< height="17" valign="top">
Problem is that the tag names are blank. The XMLReader (tagsoup parser) does report an empty namespaceURI and empty local name in the SAX methods ContentHandler#startElement and ContentHandler#endElement. For a not namespace aware parser this is allowed (see Javadoc).
If i add a XMLFilter which copies the value of the qName to the localName, everything goes fine. However, this is not what i want, i expect this works "out of the box". What am i doing wrong? Any input would be appreciated!

I expect this works "out of the box". What am i doing wrong?
What you are doing wrong is taking a technology (XSLT) that is defined to operate over namespace-well-formed XML and attempting to apply it to data that it is not intended to work with. If you want to use XSLT then you must enable namespaces, declare a prefix for the http://www.w3.org/1999/xhtml namespace in your stylesheet, and use that prefix consistently in your XPath expressions.
If your transformer understands XSLT 2.0 (e.g. Saxon 9) then instead of declaring a prefix and prefixing your element names in XPath expressions, you can put xpath-default-namespace="http://www.w3.org/1999/xhtml" on the xsl:stylesheet element to make it treat unprefixed element names as references to that namespace. But in XSLT 1.0 (the default built-in Java Transformer implementation) your only option is to use a prefix.

Access JDOM elements independent of its namespace

I have a xml like below:
<v2:Root xmlns:v2="www.example.com/xsd/">
<ABC>test data</ABC>
<ABC>test data1</ABC>
<ABC>test data2</ABC>
</v2:Root>
When I'm accessing ABC element using JDOM2, i'm getting the element value in debug like
[Element:ABC[Namespace:"www.example.com/xsd/"]].
That's why i couldn't access the element by just using Xpath expression "//ABC". I'm forced to use expression "/*[local-name()='ABC']".Then it works.
Now, my requirement is to acces the elemnt using expression "//ABC" only. Is there any way?
Thanks in advance for any help.

I think you are mistaken about what your XML actually looks like. I believe you also must have:
xmlns="www.example.com/xsd/"
in there somewhere otherwise your ABC Elements would be in the NO_NAMESPACE namespace (and the ABC toString() method would look like: [Element:ABC] )
So, your XML snippet does not match the ABC Element toString() output.
If you fix your question it will be easier to suggest what your XPath expression should look like.
EDIT, assuming I am right that you have the additional redefinition of the default Namespace, then you can use the following JDOM to get the ABC elements:
XPathFactory xpf = XPathFactory.instance();
Namespace defns = Namespace.getNamespace("defns", "www.example.com/xsd/");
XPathExpression<Element> xpe = xpf.compile("//defns:ABC", Filters.element(), null, defns);
List<Element> abcs = xpe.evaluate(doc);
You should read the following exerpt from the XPath specification carefully:
A QName in the node test is expanded into an expanded-name using the namespace declarations from the expression context. This is the same way expansion is done for element type names in start and end-tags except that the default namespace declared with xmlns is not used: if the QName does not have a prefix, then the namespace URI is null (this is the same way attribute names are expanded). It is an error if the QName has a prefix for which there is no namespace declaration in the expression context.

Java reading xml element without prefix but within the scope of a namespace

Functionally, the two blocks should be the same
<soapenv:Body>
<ns1:login xmlns:ns1="urn:soap.sof.com">
<userInfo>
<username>superuser</username>
<password>qapass</password>
</userInfo>
</ns1:login>
</soapenv:Body>
-----------------------
<soapenv:Body>
<ns1:login xmlns:ns1="urn:soap.sof.com">
<ns1:userInfo>
<ns1:username>superuser</ns1:username>
<ns1:password>qapass</ns1:password>
</ns1:userInfo>
</ns1:login>
</soapenv:Body>
However, how when I read using AXIS2 and I have tested it with java6 as well, I am having a problem.
MessageFactory factory = MessageFactory.newInstance();
SOAPMessage soapMsg = factory.createMessage(new MimeHeaders(), SimpleTest.class.getResourceAsStream("LoginSoap.xml"));
SOAPBody body = soapMsg.getSOAPBody();
NodeList nodeList = body.getElementsByTagNameNS("urn:soap.sof.com", "login");
System.out.println("Try to get login element" + nodeList.getLength()); // I can get the login element
Node item = nodeList.item(0);
NodeList elementsByTagNameNS = ((Element)item).getElementsByTagNameNS("urn:soap.sof.com", "username");
System.out.println("try to get username element " + elementsByTagNameNS.getLength());
So if I replace the 2nd getElementsByTagNameNS with ((Element)item).getElementsByTagName("username");, I am able to get the username element. Doesn't username have ns1 namespace even though it doesn't have the prefix? Am I suppose to keep track of the namespace scope to read an element? Wouldn't it became nasty if my xml elements are many level deep? Is there a workaround where I can read the element in ns1 namespace without knowing whether a prefix is defined?

Short answer is no, those documents are not the same. Namespace is not inherited by elements, and in setting a prefix, your namespace no longer functions as the default namespace for the document.
These two would be the same:
<soapenv:Body>
<login xmlns="urn:soap.sof.com">
<userInfo>
<username>superuser</username>
<password>qapass</password>
</userInfo>
</login>
</soapenv:Body>
-----------------------
<soapenv:Body>
<ns1:login xmlns:ns1="urn:soap.sof.com">
<ns1:userInfo>
<ns1:username>superuser</ns1:username>
<ns1:password>qapass</ns1:password>
</ns1:userInfo>
</ns1:login>
</soapenv:Body>
For a more robust way to read the document, you should probably look into compiling some XPath statements. Namespace issues are only one of the problems with relying on the getElementsByTagName(NS) convenience methods.
-- Edit --
Xpath itself is pretty basic. e.g., //userInfo select all the userInfo elements at any level. //login/userInfo select all the userInfo elements that are children of a login at any level. Like everything else, it gets messier when you have to start adding name spaces.
private NamespaceContext ns = new NamespaceContext() {
public String getNamespaceURI(String prefix) {
if (prefix.equals("urn") return "urn:soap.sof.com";
else return XMLConstants.NULL_NS_URI;
}
public String getPrefix(String namespace) {
throw new UnsupportedOperationException();
}
public Iterator getPrefixes(String namespace) {
throw new UnsupportedOperationException();
}};
XPathFactory xpfactory = XPathFactory.newInstance();
XPath xpath = xpfactory.newXPath();
xpath.setNamespaceContext(ns);
NodeList nodes = (NodeList) xpath.evaluate("//urn:userInfo|//userInfo", myDom, XPathConstants.NODESET);
//find all userInfo at any depth with either namespace.
S'been a long time since I've used JAXP, but I think that's basically correct. Running an xPath isn't slow, but compiling them is. You can compile them to a XPathExpression for performance, but those aren't threadsafe, so you can't just cache them on a servlet. Never is easy =/.
If you are doing a lot of XML, I would recommend using Jaxen instead of JAXP. (On the other hand if you do very little XML and it's just a one of on the front end, maybe getElementsByTagName isn't the worst thing ever :) )