How to remove elements of a page in htmlunit - java

Normally in PHP, I would just parse the old document and write to the new document while ignoring the unwanted elements.

This was the first solution I came up with:
DocumentBuilder builder = DocumentBuilderFactory
.newInstance()
.newDocumentBuilder();
StringReader reader = new StringReader( xml );
Document document = builder.parse( new InputSource(reader) );
XPathExpression expr = XPathFactory
.newInstance()
.newXPath()
.compile( ... );
Object result = expr.evaluate(document, XPathConstants.NODESET);
Element el = document.getDocumentElement();
NodeList nodes = (NodeList) result;
for (int i = 0; i < nodes.getLength(); i++) {
el.removeChild( nodes.item(i) );
}
As you can see it's kinda long. Being a coder who strives for simplicity, I decided to take Ahmed's advice hoping I'll find a better solution and I came up with this:
List<?> elements = page.getByXPath( ... );
DomNode node = null;
for( Object o : elements ) {
node = (DomNode)o;
node.getParentNode().removeChild( node );
}
Please note these are just snippets, I omitted the imports and the XPath expressions but you get the idea.

Have a look at the DOM methods, you can remove nodes.
http://htmlunit.sourceforge.net/apidocs/com/gargoylesoftware/htmlunit/html/DomNode.html

Related

XML - Extract One tag Value

I have to extract tag value from an xml Document that contains a single tag like below:
<error>Permission denied</error>
i have tried:
String xmlRecords = "<error>Permission denied</error>"
DocumentBuilderFactory dbf = DocumentBuilderFactory.newInstance();
DocumentBuilder db = dbf.newDocumentBuilder();
InputSource is = new InputSource();
is.setCharacterStream(new StringReader(xmlRecords));
Document doc = db.parse(is);
Node nodes = doc.getFirstChild();
String = nodes.getNodeValue();
but it dont works.
How can i do it ?
Use doc.getDocumentElement().getTextContent() to get the string Permission denied.
With DOM it´s util to know the structure of the XML document, and which node level are you looking for.
After get Document, you can use document.getElementsByTagName("root") to look for the root or father tags, and get the childs as a list to look for the item. Something like this:
NodeList listresults = document.getElementsByTagName('father/root element string');
NodeList nl = listresults.item(0).getChildNodes();
// Recorremos los nodos
for (int temp = 0; temp < nl.getLength(); temp++) {
Node node = nl.item(temp);
// Check if it is a node
if (node.getNodeType() == Node.ELEMENT_NODE) {
Element element = (Element) node;
if(element.getNodeName().equals("error")){
// check the element
}
}
}
I hope this helps you.
just try following code.
String value = nodes.getTextContent();
You have to construct the string if you are using the above approach. You will get the string values of the tag name and content using the functions.
Tag name = nodes.getTextContent()
tag value = nodes.getLocalName()
I guess this is what you want
Element element = document.getDocumentElement();
NodeList errorTagList = element.getElementsByTagName("error");
if (errorTagList != null && errorTagList.getLength() > 0) {
NodeList errorTagSubList = errorTagList.item(0).getChildNodes();
if (errorTagSubList != null && errorTagSubList.getLength() > 0) {
String value = errorTagSubList.item(0).getNodeValue();
}
}

Java XML - nested elements with same name

How can I reach to elements which have same name and recursive inclusion using Java XML? This has worked in python ElementTree, but for some reason I need to get this running in Java.
I have tried:
String filepath = ("file.xml");
DocumentBuilderFactory docFactory = DocumentBuilderFactory.newInstance();
DocumentBuilder docBuilder = docFactory.newDocumentBuilder();
Document doc = docBuilder.parse(filepath);
NodeList nl = doc.getElementsByTagName("*/*/foo");
Example
<foo>
<foo>
<foo>
</foo>
</foo>
</foo>
You seem to be under the impression that getElementsByTagName takes an XPath expression. It doesn't. As documented:
Returns a NodeList of all the Elements in document order with a given tag name and are contained in the document.
If you need to use XPath, you should look at the javax.xml.xpath package. Sample code:
Object set = xpath.evaluate("*/*/foo", doc, XPathConstants.NODESET);
NodeList list = (NodeList) set;
int count = list.getLength();
for (int i = 0; i < count; i++) {
Node node = list.item(i);
// Handle the node
}

how to get attribute of given node?

I am trying to write DOM XML parsing.
My Xml file
<?xml version="1.0"?>
<BLAH>
<AgentNm type="citi1">
<accName>accName1</accName>
<accType>accType1</accType>
<someThing>someThing1</someThing>
<amt>100000</amt>
</AgentNm>
<AgentNm type="citi2">
<accName>accName2</accName>
<accType>accType2</accType>
<someThing>someThing2</someThing>
<amt>200000</amt>
</AgentNm>
</BLAH>
And i tried following java code
DocumentBuilderFactory docBuilderFactory = DocumentBuilderFactory.newInstance();
DocumentBuilder docBuilder = docBuilderFactory.newDocumentBuilder();
Document doc = docBuilder.parse (new File("c:\\file.xml"));
// normalize text representation
doc.getDocumentElement ().normalize ();
System.out.println ("Root element of the doc is " +doc.getDocumentElement().getNodeName());
NodeList agentNm = doc.getElementsByTagName("AgentNm");
int totalAgentNm = agentNm.getLength();
System.out.println("Total no of Agents : " + totalAgentNm);
for(int s=0; s<agentNm.getLength() ; s++){
Node firstPersonNode = agentNm.item(s);
if(firstPersonNode.getNodeType() == Node.ELEMENT_NODE){
Element firstPersonElement = (Element)firstPersonNode;
PrintNodeElem(firstPersonElement,"type");
}//end of if clause
}//end of for loop with s var
static void PrintNodeElem(Element nodeElem,String elem){
NodeList someThingList = nodeElem.getElementsByTagName(elem);
Element ageElement = (Element)someThingList.item(0);
NodeList textAgeList = ageElement.getChildNodes();
System.out.println(elem+" : " +((Node)textAgeList.item(0)).getNodeValue().trim());
}
But, when i tried to execute above method,
i am getting null pointer exception.
can any one explain me, how to fix this.
if you want an attribute of a given node, I would suggest XPath. It is much easier.
http://onjava.com/onjava/2005/01/12/xpath.html

Parsing xml string containing hyperlink

I am using DOM to parse an XML string as in the following example. This works great except in one instance. The document which I am trying to parse looks like this:
<response requestID=\"1234\">
<expectedValue>Alarm</expectedValue>
<recommendations>For steps on how to resolve visit Website and use the search features for \"Alarm\"<recommendations>
<setting>Active</setting>
<response>
The code I used to parse the XML is as follows:
try {
DocumentBuilder db = DocumentBuilderFactory.newInstance().newDocumentBuilder();
InputSource is = new InputSource();
is.setCharacterStream(new StringReader(xmlResult));
Document doc = db.parse(is);
NodeList nlResponse = doc.getElementsByTagName("response");
String[] String = new String[3]; //result entries
for (int i = 0; i < nlResponse.getLength(); i++) {
Element e = (Element) nlResponse.item(i);
int c1 = 0; //count for string array
NodeList ev = e.getElementsByTagName("expectedValue");
Element line = (Element) ev.item(0);
String[c1] = (getCharacterDataFromElement(line));
c1++;
NodeList rec = e.getElementsByTagName("recommendations");
line = (Element) rec.item(0);
String[c1] = (getCharacterDataFromElement(line));
c1++;
NodeList set = e.getElementsByTagName("settings");
line = (Element) set.item(0);
String[c1] = (getCharacterDataFromElement(line));
c1++;
I am able to parse the code and put the result into a string array (as opposed to the System.out.println()). With the current code, my string array looks as follows:
String[0] = "Alarm"
String[1] = "For steps on how to resolve visit"
String[2] = "Active"
I would like some way of being able to read the rest of the information within "Recommendations" in order to ultimately display the hyperlink (along with other output) in a TextView. How can I do this?
I apologize for my previous answer in assuming your xml was ill-formed.
I think what is happening is that your call to the getCharacterDataFromElement is only looking at the first child node for text, when it will need to look at all the child nodes and getting the href attribute as well as the text for the 2nd child node when looking at the recommendations node.
e.g. after getting the Element for recommendation
String srec = "";
NodeList nl = line.getChildNodes();
srec += nl.item(0).getTextContent();
Node n = nl.item(1);
NamedNodeMap nm = n.getAttributes();
srec += "" + n.getTextContent() + "";
srec += nl.item(2).getTextContent();
String[c1] = srec;

Looping over nodes and extracting specific subnode values using Java's XPath

I understand from Googling that it makes more sense to extract data from XML using XPath than by using DOM looping.
At the moment, I have implemented a solution using DOM, but the code is verbose, and it feels untidy and unmaintainable, so I would like to switch to a cleaner XPath solution.
Let's say I have this structure:
<products>
<product>
<title>Some title 1</title>
<image>Some image 1</image>
</product>
<product>
<title>Some title 2</title>
<image>Some image 2</image>
</product>
...
</products>
I want to be able to run a for loop for each of the <product> elements, and inside this for loop, extract the title and image node values.
My code looks like this:
InputStream is = conn.getInputStream();
DocumentBuilder builder =
DocumentBuilderFactory.newInstance().newDocumentBuilder();
Document doc = builder.parse(is);
XPathFactory factory = XPathFactory.newInstance();
XPath xpath = factory.newXPath();
XPathExpression expr = xpath.compile("/products/product");
Object result = expr.evaluate(doc, XPathConstants.NODESET);
NodeList products = (NodeList) result;
for (int i = 0; i < products.getLength(); i++) {
Node n = products.item(i);
if (n != null && n.getNodeType() == Node.ELEMENT_NODE) {
Element product = (Element) n;
// do some DOM navigation to get the title and image
}
}
Inside my for loop I get each <product> as a Node, which is cast to an Element.
Can I simply use my instance of XPathExpression to compile and run another XPath on the Node or the Element?
Yes, you can always do like this -
XPathFactory factory = XPathFactory.newInstance();
XPath xpath = factory.newXPath();
XPathExpression expr = xpath.compile("/products/product");
Object result = expr.evaluate(doc, XPathConstants.NODESET);
expr = xpath.compile("title"); // The new xpath expression to find 'title' within 'product'.
NodeList products = (NodeList) result;
for (int i = 0; i < products.getLength(); i++) {
Node n = products.item(i);
if (n != null && n.getNodeType() == Node.ELEMENT_NODE) {
Element product = (Element) n;
NodeList nodes = (NodeList) expr.evaluate(product,XPathConstants.NODESET); //Find the 'title' in the 'product'
System.out.println("TITLE: " + nodes.item(0).getTextContent()); // And here is the title
}
}
Here I have given example of extracting the 'title' value. In same way you can do for 'image'
I'm not a big fan of this approach because you have to build a document (which might be expensive) before you can apply XPaths to it.
I've found VTD-XML a lot more efficient when it comes to applying XPaths to documents, because you don't need to load the whole document into memory. Here is some sample code:
final VTDGen vg = new VTDGen();
vg.parseFile("file.xml", false);
final VTDNav vn = vg.getNav();
final AutoPilot ap = new AutoPilot(vn);
ap.selectXPath("/products/product");
while (ap.evalXPath() != -1) {
System.out.println("PRODUCT:");
// you could either apply another xpath or simply get the first child
if (vn.toElement(VTDNav.FIRST_CHILD, "title")) {
int val = vn.getText();
if (val != -1) {
System.out.println("Title: " + vn.toNormalizedString(val));
}
vn.toElement(VTDNav.PARENT);
}
if (vn.toElement(VTDNav.FIRST_CHILD, "image")) {
int val = vn.getText();
if (val != -1) {
System.out.println("Image: " + vn.toNormalizedString(val));
}
vn.toElement(VTDNav.PARENT);
}
}
Also see this post on Faster XPaths with VTD-XML.

Categories