We Keep Coding

Parsing Table with Jsoup for Android App

I am attempting to parse through a table on a website for a give table row in which the first column matches a certain string of characters. Below is the HTML for part of the table (it's very larger)
<table class="table display datatable" id="datatable1">
<thead>
<tr>
<th class="va-m">Miner</th>
<th class="va-m">Shares</th>
<th class="va-m">%</th>
<th class="va-m">Best DL</th>
</tr>
</thead>
<tfoot>
<tr>
<th class="va-m">Miner</th>
<th class="va-m">Shares</th>
<th class="va-m">%</th>
<th class="va-m">Best DL</th>
</tr>
</tfoot>
<tbody>
<tr>
<td>3R8RDBxiux3g1pFCCsQnm2vwD34axsVRTrEWzyX8tngJaRnNWkbnuFEewzuBAKhQrb3LxEQHtuBg1zW4tybt83SS</td>
<td>44279</td>
<td>27.37 %</td>
<td>1154</td>
</tr>
<tr>
<td>5gwVxC9cXguHHjD9wtTpHfsJPaZx4fPcvWD5jGWF1dcuHnAMyXxteaHrEtXviZkvWN3FAnevbVLErABSsP6mS7PR</td>
<td>36369</td>
<td>22.48 %</td>
<td>2725</td>
</tr>
<tr>
<td>2qZXPmop82UiA7LQEQqdoUzjFbcwCSpqf8U1f3656XXSsHnGvGXYTNoP11s2asiVSyVS8LPFqxmpdCeSNxcpFMnF</td>
<td>28596</td>
<td>17.68 %</td>
<td>967</td>
</tr>
<tr>
<td>21mbNSDo7g9BAyjsZGxnNfJUrEtBUVVNQZhR4tkVwdEHPaMNsa2u2JHQPAAe5riGfPA9Khb1Pq3bQGhqmrLEGNqN</td>
<td>6104</td>
<td>3.77 %</td>
<td>4787</td>
</tr>
<tr>
<td>4HAakKK7dSq18Djg7m6cLSyHb5aUU6ngvBQimo8pYyF5F64qX3gE4T8q8kfWHTZ79FvXybSG3JhUfSZDDv2sRwqY</td>
<td>5895</td>
<td>3.64 %</td>
<td>6020</td>
</tr>
<tr>
<td>2r2izPEC5o7ZDnUsdDA97q8wKCeZRRg9n243Rd9vkMQqRCtc6ZRUTruQUyZGduoHy8pTYPuEq9ACXPKfXt8fqKxS</td>
<td>5605</td>
<td>3.46 %</td>
<td>10958</td>
</tr>
</tbody>
</table>
I am trying to step through the table and search for a specific row but I am receiving an IndexOutOfBoundsException.
Would there be a better way to code the statement below?
for (Element table : doc.select("table")){
for(Element row : table.select("tr")){
Elements tds = row.select("td");
if(tds.get(0).text().equals("4HjSN79KUMz7AQC3GBvGkgPa5Qrio9HWTh7hg9JY48fkrYeVZJVmzB9YCB6GZSpuXB7V7DjJVuke3ZaCm5k7sRLE")){
myHistoricShares =tds.get(0).text();
}
}
}

As I said in comments, your table.select("tr") selects rows not only inside <tbody>, but inside the header and the footer too. For those rows row.select("td") returns an empty list, and hence tds.get(0) throws the IndexOutOfBoundsException.
You could simplify your loop by selecting only the rows in <tbody>:
for (Element row: doc.select("table#datatable1>tbody>tr")) {
if (row.children().size() > 0 && "some_long_string".equals(row.child(0).text())) {
doSomething();
}
}
The selector "table#datatable1>tbody>tr" selects the table with id="datatable1", then its exact tbody child and then all its exact tr children. So you only need to iterate through them once.

java find table using jsoup and equivalent xpath

Here is the HTML code:
<table class="textfont" cellspacing="0" cellpadding="0" width="100%" align="center" border="0">
<tbody>
<tr>
<td class="chl" width="20%">Batch ID</td><td class="ctext">d32654464bdb424396f6a91f2af29ecf</td>
</tr>
<tr>
<td class="chl" width="20%">ALM Server</td>
<td class="ctext"></td>
</tr>
<tr>
<td class="chl" width="20%">ALM Domain/Project</td>
<td class="ctext">EBUSINESS/STERLING</td>
</tr>
<tr>
<td class="chl" width="20%">TestSet URL</td>
<td class="ctext">almtestset://localhost</td>
</tr>
<tr>
<td class="chl" width="20%">Tests Executed</td>
<td class="ctext"><b>6</b></td>
</tr>
<tr>
<td class="chl" width="20%">Start Time</td>
<td class="ctext">08/31/2017 12:20:46 PM</td>
</tr>
<tr>
<td class="chl" width="20%">Finish Time</td>
<td class="ctext">08/31/2017 02:31:46 PM</td>
</tr>
<tr>
<td class="chl" width="20%">Total Duration</td>
<td class="ctext"><b>2h 11m </b></td>
</tr>
<tr>
<td class="chl" width="20%">Test Parameters</td>
<td class="ctext"><b>{"browser":"chrome","browser-version":"56","language":"english","country":"US"}</b></td>
</tr>
<tr>
<td class="chl" width="20%">Passed</td>
<td class="ctext" style="color:#269900"><b>0</b></td>
</tr>
<tr>
<td class="chl" width="20%">Failed</td>
<td class="ctext" style="color:#990000"><b>6</b></td>
</tr>
<tr>
<td class="chl" width="20%">Not Completed</td>
<td class="ctext" style="color: ##ff8000;"><b>0</b></td>
</tr>
<tr>
<td class="chl" width="20%">Test Pass %</td>
<td class="ctext" style="color:#990000;font-size:14px"><b>0.0%</b></td>
</tr>
</tbody>
And here is the xpath to get the table:
//td[text() = 'TestSet URL']/ancestor::table[1]
How can I get this table using jSoup? I've tried:
tableElements = doc.select("td:contains('TestSet URL')");
to get the child element, but that doesn't work and returns null. I need to find the table and put all the children into a map. Any help would be greatly appreciated!

The following code will parse your table into a map, this code is subject to a few assumptions:
This xpath //td[text() = 'TestSet URL']/ancestor::table[1] will find any table which contains the text "TestSet URL" anywhere in its body, this seems a little bit brittle but assuming it is sufficient for you the JSoup code in getTable() is functionally equiavalent to that xpath
The code below assumes that every row contains two cells with the first one being the key and the second one being the value, since you want to parse the table content to a map this assumption seems valid
The code below throws exceptions if the above assumptions are not met i.e. if the given HTML does not contain a table definition with "TestSet URL" embedded in its body or if there are more than two cells in any row within that table.
If those assumptions are invalid then the internals of getTable and parseTable will change but the general approach will remain valid.
public void parseTable() {
Document doc = Jsoup.parse(html);
// declare a holder to contain the 'mapped rows', this is a map based on the assumption that every row represents a discreet key:value pair
Map<String, String> asMap = new HashMap<>();
Element table = getTable(doc);
// now walk though the rows creating a map for each one
Elements rows = table.select("tr");
for (int i = 0; i < rows.size(); i++) {
Element row = rows.get(i);
Elements cols = row.select("td");
// expecting this table to consist of key:value pairs where the first cell is the key and the second cell is the value
if (cols.size() == 2) {
asMap.put(cols.get(0).text(), cols.get(1).text());
} else {
throw new RuntimeException(String.format("Cannot parse the table row: %s to a key:value pair because it contains %s cells!", row.text(), cols.size()));
}
}
System.out.println(asMap);
}
private Element getTable(Document doc) {
Elements tables = doc.select("table");
for (int i = 0; i < tables.size(); i++) {
// this xpath //td[text() = 'TestSet URL']/ancestor::table[1] will find the first table which contains the
// text "TestSet URL" anywhere in its body
// this crude evaluation is the JSoup equivalent of that xpath
if (tables.get(i).text().contains("TestSet URL")) {
return tables.get(i);
}
}
throw new RuntimeException("Cannot find a table element which contains 'TestSet URL'!");
}
For the HTML posted in your question, the above code will output:
{Finish Time=08/31/2017 02:31:46 PM, Passed=0, Test Parameters={"browser":"chrome","browser-version":"56","language":"english","country":"US"}, TestSet URL=almtestset://localhost, Failed=6, Test Pass %=0.0%, Not Completed=0, Start Time=08/31/2017 12:20:46 PM, Total Duration=2h 11m, Tests Executed=6, ALM Domain/Project=EBUSINESS/STERLING, Batch ID=d32654464bdb424396f6a91f2af29ecf, ALM Server=}

You have to remove those quotation marks to get the row with the text; just
tableElements = doc.select("td:contains(TestSet URL)");
but note with the above you are only selecting td elements which contain the text "TestSet URL". To select the whole table use
Element table = doc.select("table.textfont").first();
which means select table with class=textfont and to avoid selecting multiple tables which can have the same class value you have to specify which to choose, therefore: first().
To get all the tr elements:
Elements tableRows = doc.select("table.textfont tr");
for(Element e: tableRows)
System.out.println(e);

html table td contents parsing using jsoup in android

I've with me some html table contents.And for my application I want to parse these html contents using JSOUP parsing in android.But I am new to this JSOUP method and I can't parse those html contents properly.
HTML data:
<table id="box-table-a" summary="Tracking Result">
<thead>
<tr>
<th width="20%">AWB / Ref. No.</th>
<th width="30%">Status</th>
<th width="30%">Date Time</th>
<th width="20%">Location</th>
</tr>
</thead>
<tbody>
<tr>
<td width="20%" nowrap="nowrap" class="click">Z45681583</td>
<td width="30%" nowrap="nowrap" class="click">
IN TRANSIT<div id='ntfylink' style='display:block; text-decoration:blink'><a href='#' class='topopup' name='modal' style='text-decoration:none'><font face='Verdana' color='#DF0000'><blink>Notify Me</blink></font></a></div>
</td>
<td width="30%">
Sat, Jan, 31, 2015 07:09 PM
</td>
<td width="20%">DELHI</td>
</tr>
</tbody>
</table>
from this table I need the"td" contents.
Any help would be greatly appreciated.

Everything is described clearly in source code below.
private static String test(String htmlFile) {
File input = null;
Document doc = null;
Elements tdEles = null;
Element table = null;
String tdContents = "";
try {
input = new File(htmlFile);
doc = Jsoup.parse(input, "ASCII", "");
doc.outputSettings().charset("ASCII");
doc.outputSettings().escapeMode(EscapeMode.base);
/** Get table with id = box-table-a **/
table = doc.getElementById("box-table-a");
if (table != null) {
/** Get td tag elements **/
tdEles = table.getElementsByTag("td");
/** Loop each of the td element and get the content by ownText() **/
if (tdEles != null && tdEles.size() > 0) {
for (Element e: tdEles) {
String ownText = e.ownText();
//Delimiter as "||"
if (ownText != null && ownText.length() > 0)
tdContents += ownText + "||";
}
if (tdContents.length() > 0) {
tdContents = tdContents.substring(0, tdContents.length() - 2);
}
}
}
} catch (Exception e) {
e.printStackTrace();
}
return tdContents;
}
You can manipulate the String in your textview. All the TD contents is delimited by ||. Use String.split() to get each content if you want.
String[] data = tdContents.split("\\|\\|");

Java JSP servlet can't see DB in JSP

I begun to learn datebase and servlet, JSP. And I don't know some fails. I can't see my db in JSP.
Java code:
try {
Class.forName("org.postgresql.Driver");
connection = DriverManager.getConnection(url, login, password);
String sql = "select * fiz_phone";
Statement s = connection.createStatement();
s.executeQuery(sql);
rs = s.getResultSet();
while (rs.next()) {
dataList.add(rs.getInt("id"));
dataList.add(rs.getString("name"));
dataList.add(rs.getString("adress"));
dataList.add(rs.getString("phone"));
dataList.add(rs.getString("phone_adress"));
dataList.add(rs.getInt("cost"));
dataList.add(rs.getString("exempt_type"));
dataList.add(rs.getInt("exempt"));
dataList.add(rs.getString("date_claim"));
dataList.add(rs.getInt("number_claim"));
dataList.add(rs.getString("inspektor"));
dataList.add(rs.getString("date_repair"));
dataList.add(rs.getInt("phone_cost"));
dataList.add(rs.getString("call"));
}
rs.close();
s.close();
} catch (Exception e) {
System.out.println("Exception is ;" + e);
}
request.setAttribute("data", dataList);
RequestDispatcher dispatcher = request.getRequestDispatcher(page);
if (dispatcher != null) {
dispatcher.forward(request, response);
}
HTML:
<body>
<table border="1">
<tr>
<td ><b>ID</b></td>
<td ><b>Name</b></td>
<td ><b>Adress</b></td>
<td ><b>Phone</b></td>
<td ><b>Phone adress</b></td>
<td ><b>Cost</b></td>
<td ><b>Exempt type</b></td>
<td ><b>Exempt</b></td>
<td ><b>Date claim</b></td>
<td ><b>Number of claim</b></td>
<td ><b>Inspektor</b></td>
<td ><b>Date repair</b></td>
<td ><b>Phone cost</b></td>
<td ><b>Call</b></td>
</tr>
<%ArrayList<String> f = (ArrayList<String>) request
.getAttribute("data");
Iterator<String> itr = f.iterator();
while (itr.hasNext()) {%>
<tr id="tab">
<td ><%=itr.next()%></td>
<td ><%=itr.next()%></td>
<td ><%=itr.next()%></td>
<td ><%=itr.next()%></td>
<td ><%=itr.next()%></td>
<td ><%=itr.next()%></td>
<td ><%=itr.next()%></td>
<td ><%=itr.next()%></td>
<td ><%=itr.next()%></td>
<td ><%=itr.next()%></td>
<td ><%=itr.next()%></td>
<td ><%=itr.next()%></td>
<td ><%=itr.next()%></td>
<td ><%=itr.next()%></td>
</tr>
<%}%>
</table>
</body>
and servlet
<servlet>
<servlet-name>DataServlet</servlet-name>
<servlet-class>work_project.Data</servlet-class>
</servlet>
<servlet-mapping>
<servlet-name>DataServlet</servlet-name>
<url-pattern>/DataServlet</url-pattern>
</servlet-mapping>
When I start this program, it give me this exception
org.apache.jasper.JasperException: An exception occurred processing JSP page /DataPage.jsp at line 35
32: </tr>
33: <%Iterator itr;%>
34: <% List data= (List)request.getAttribute("data");
35: for (itr=data.iterator(); itr.hasNext(); ) {
36: %>
37: <tr>
38: <td ><%=itr.next()%></td>

I ran an example of your code. It worked. The only difference was that I inserted some code into the data list, instead of accessing a database. Even if the result set was empty, you should not be getting a null pointer exception.
Since you are getting a null pointer exception when you access the data list, it seems to indicate that the original list in the servlet was null. What is the code for instantiating the dataList in the servlet? Do you have a statement somewhere like
List<String> dataList = new ArrayList<String>();
Try debugging your code. Set a breakpoint in the servlet on the setAtribute statement. See if dataList is null.
Also check that the loop is executing. Be sure the result set is not empty.

Building on the answer of Roman C, the scriptlet scope is not the same as the expression language scope. If you use a jstl core template for the loop, then it should see row.
<%# taglib uri="http://java.sun.com/jsp/jstl/core" prefix="c"%>
<c:forEach var="row" items="${data}">
<tr id="tab">
<td >${row.id}</td>
<td >${row.name}</td>
</tr>
</c:forEach>
I do not think this will resolve the original problem. data is probably still null.
Have you tried using the debugger and testing that dataList is not null when it is added to the request with setAttribute?
Responding to comment about debugger:
You need to step one more time, so that rows is initialized. Try the same thing in the servlet for the setAttribute statement, to see if dataList is null.

The code should be a valid SQL query by querying a database with JDBC statement.
String sql = "select * from fiz_phone";
It might be a typo but you ignored it in the code for the catch block
} catch (Exception e) {
//TODO you should handle the exception here
System.out.println("Exception is ;" + e);
}
Also since you are learning, you better return an object for each record from the resultset.
while (rs.next()) {
MyObject obj = new MyObject();
obj.setId(rs.getInt("id"));
obj.setName(rs.getString("name"));
...
dataList.add(obj);
}
Also do it in iterator
<%ArrayList<MyObject> f = (ArrayList<MyOobject>) request.getAttribute("data");
Iterator<MyObject> itr = f.iterator();
while (itr.hasNext()) {
MyObject obj = itr.next();
pageContext.setAttribute("row", obj);%>
<tr id="tab">
<td >${row.id}</td>
<td >$(row.name}</td>
...
</tr>
<%} pageContext.removeAttribute("row");%>
You should make MyObject a java bean (i.e. properties should have getters and setters). Latter you may try to remove scriptlets at all by using JSTL tag library and forEach tag.

Skip table if not found using selenium

I have html code that is very similar to this:
<TH CLASS="ddtitle">MovieOne</TH>
<TABLE CLASS="datadisplaytable" ><CAPTION class="captiontext">Movies</CAPTION>
<TR>
<TH CLASS="ddheader" scope="col" >Genre</TH>
<TH CLASS="ddheader" scope="col" >Time</TH>
<TH CLASS="ddheader" scope="col" >Days</TH>
<TH CLASS="ddheader" scope="col" >Where</TH>
<TH CLASS="ddheader" scope="col" >Date Range</TH>
<TH CLASS="ddheader" scope="col" >Seating</TH>
<TH CLASS="ddheader" scope="col" >Actors</TH>
</TR>
<TR>
<TD CLASS="dddefault">Action</TD>
<TD CLASS="dddefault">10:00 am - 12:00 pm</TD>
<TD CLASS="dddefault">SMTWTHFSA</TD>
<TD CLASS="dddefault">AMC Showplace</TD>
<TD CLASS="dddefault">Aug 20, 2014 - Sept 12, 2014</TD>
<TD CLASS="dddefault">Reservations</TD>
<TD CLASS="dddefault">Will Ferrel (<ABBR title= "Primary">P</ABBR>) target="Will Ferrel" ></TD>
</TR>
</TABLE>
<TH CLASS="ddtitle">MovieTwo</TH>
<TABLE CLASS="datadisplaytable" ><CAPTION class="captiontext">Movies</CAPTION>
<TR>
<TH CLASS="ddheader" scope="col" >Genre</TH>
<TH CLASS="ddheader" scope="col" >Time</TH>
<TH CLASS="ddheader" scope="col" >Days</TH>
<TH CLASS="ddheader" scope="col" >Where</TH>
<TH CLASS="ddheader" scope="col" >Date Range</TH>
<TH CLASS="ddheader" scope="col" >Seating</TH>
<TH CLASS="ddheader" scope="col" >Actors</TH>
</TR>
<TR>
<TD CLASS="dddefault">Action</TD>
<TD CLASS="dddefault">11:00 am - 12:30 pm</TD>
<TD CLASS="dddefault">SMTWTHFSA</TD>
<TD CLASS="dddefault">Showplace Cinemas</TD>
<TD CLASS="dddefault">Aug 20, 2014 - Sept 12, 2014</TD>
<TD CLASS="dddefault">TBA</TD>
<TD CLASS="dddefault">Zach Galifinakis (<ABBR title= "Primary">P</ABBR>) target="Zach Galifinakis" ></TD>
</TR>
</TABLE>
<TH CLASS="ddtitle">MovieThree</TH>
<BR>
<BR>
Coming Soon
<BR>
What I want to be able to do, is take the individual table data that is relevant for the movie title, and if a Movie doesn't have a table I want to say the values are TBA. So far, I am able to get the relevant table information, but I am unable to skip a table. For example I use this code to get the genre of the movie:
int tcounter = 1;
for (Element elements : li) {
WebElement genre = driver.findElement(By.xpath("//table[#class='datadisplaytable']/descendant::table["+tcounter+"]//td[1]"));
WebElement time = driver.findElement(By.xpath("//table[#class='datadisplaytable']/descendant::table["+tcounter+"]//td[2]"));
WebElement days = driver.findElement(By.xpath("//table[#class='datadisplaytable']/descendant::table["+tcounter+"]//td[3]"));
WebElement where = driver.findElement(By.xpath("//table[#class='datadisplaytable']/descendant::table["+tcounter+"]//td[4]"));
WebElement date_range = driver.findElement(By.xpath("//table[#class='datadisplaytable']/descendant::table["+tcounter+"]//td[5]"));
WebElement seating = driver.findElement(By.xpath("//table[#class='datadisplaytable']/descendant::table["+tcounter+"]//td[6]"));
WebElement actors = driver.findElement(By.xpath("//table[#class='datadisplaytable']/descendant::table["+tcounter+"]//td[7]"));
tcounter++;
}
elements refers to a list storing all links on the webpage
(result for [1] would be action, [2] would be 10:00 am - 12:00pm ...).
This is within a for loop that increments the value of the tcounter by 1 in order to receive the data for different tables. Is there a way I can be able to tell the program to see if a table is present under the TH class, and if not give the values TBA and skip it?
This is my second attempt based on siking's answer:
List<WebElement> linstings = driver.findElements(By.className("ddtitle"));
String genre = "";
String time = "";
String days = "";
String where = "";
String dateRange = "";
String seating = "";
String actors = "";
for(WebElement potentialMovie : linstings) {
try {
WebElement actualMovie = potentialMovie.findElement(By.xpath("//table[#class='datadisplaytable']"));
// System.out.println("Actual: " + actualMovie.getText());
// make all your assignments, for example:
type = actualMovie.findElement(By.xpath("/descendant::table//td")).getText();
time = actualMovie.findElement(By.xpath("/descendant::table//td[2]")).getText();
days = actualMovie.findElement(By.xpath("/descendant::table//td[3]")).getText();
location = actualMovie.findElement(By.xpath("/descendant::table//td[4]")).getText();
dates = actualMovie.findElement(By.xpath("/descendant::table//td[5]")).getText();
schedType = actualMovie.findElement(By.xpath("/descendant::table//td[6]")).getText();
instructor = actualMovie.findElement(By.xpath("/descendant::table//td[7]")).getText();
System.out.println(genre+" "+time+" "+days+" "+where+" "+dateRange+" "+actors);
} catch(Exception ex) {
// there is no table, so:
genre = "TBA";
}
}
The problem with this code is that it keeps returning the values for only the first table.

I trimmed down your HTML sample to the following:
<TH CLASS="ddtitle">MovieOne</TH>
<TABLE CLASS="datadisplaytable">
<CAPTION class="captiontext">Movies</CAPTION>
<TR>
<TH CLASS="ddheader" scope="col">Genre</TH>
</TR>
<TR>
<TD CLASS="dddefault">Action</TD>
</TR>
</TABLE>
<TH CLASS="ddtitle">MovieTwo</TH>
<BR/>
<BR/>
Coming Soon
<BR/>
<TH CLASS="ddtitle">MovieThree</TH>
<TABLE CLASS="datadisplaytable">
<CAPTION class="captiontext">Movies</CAPTION>
<TR>
<TH CLASS="ddheader" scope="col">Genre</TH>
</TR>
<TR>
<TD CLASS="dddefault">Action</TD>
</TR>
</TABLE>
Hopefully it is representative of all your cases!
Don't use a counter, but use the actual WebElements to iterate over:
// default all your variables to TBA, like:
String genre = "TBA";
// find all the listings on the page...
List<WebElement> linstings = driver.findElements(By.className("ddtitle"));
// ... and iterate over them
for (WebElement listing : linstings) {
// grab whatever is the _first_ element under the TH ...
WebElement potentialMovie = listing.findElement(By.xpath("following-sibling::*[1]"));
// ... check if it has a child element CAPTION
if (potentialMovie.findElement(By.xpath("caption")) != null) {
// make all your assignments, for example:
genre = potentialMovie.findElement(By.xpath("tr[2]/td[1]")).getText();
}
}
Please note that this code is untested, your mileage may vary!