We Keep Coding

Java selenium - skip span element within div

Here's my html structure and I am trying to skip a span within a div to get div's text only (which is dynamic) for testing.
<div class="items">
<div class="payment void" id="payment-000000899799">
<div class="payment__details">
<div class="method">CASH <span class="label"><span>Payment Voided</span></span></div>
<div class="date">2/12/2021, 3:02:15 PM</div>
</div>
<div class="payment__details">
<div class="amount">$20.00</div>
<div class="ref">Ref ID: REF-ID-01</div>
</div>
</div>
<div class="payment sale" id="payment-000000899806">
<div class="payment__details">
<div class="method">CASH </div>
<div class="date">2/12/2021, 3:02:21 PM</div>
</div>
<div class="payment__details">
<div class="amount">$100.00</div>
<div class="ref">Ref ID: REF-ID-02</div>
</div>
</div>
</div>
In my step definition I've
List<List<String>> data = capturedData.raw();
WebElement paymentDetails = driver.findElement(By.xpath("(//*[#class='payment__details']/div[#class='method'])[" + data.get(1).get(0) + "]"));
String paymentType = paymentDetails.getText();
System.out.println(paymentType); //This prints CASH Payment Voided
But actually I want only 'CASH' which is a text of div and skip the text 'Payment Voided' of span. And data is coming from a feature file.
How do I get text of div only and skip the text span which is inside the same div?

You cannot skip it directly since it is part of element.
1st Approach
String fulltext = driver.findElement(By.xpath("//*[#class='payment__details']/div[#class='method'][1]")).getText();
String spantext = driver.findElement(By.xpath("//*[#class='payment__details']/div[#class='method'][1]/span")).getText();
//Now replace span text with ""
String output = fulltext.replace(span,"");
2nd Approach
WebElement parent = driver.findElement(By.xpath("//*[#class='payment__details']/div[#class='method'][1]"));
JavascriptExecutor js = (JavascriptExecutor) driver;
String output = (String) js.executeScript("return arguments[0].firstChild.textContent",parent);

How to save Element from Jsoup to database

I use Jsoup get all data from website and save element if match some content when i get. I want when we get element. If it match some thing character , I save element from database(MYSQL,Postgress...). I code look like :
Connection conn = Jsoup.connect("https://viblo.asia");
Document doc = conn.userAgent("Mozilla").get();
Elements elements = doc.getElementsByClass("post-feed").get(0).children();
Elements list = new Elements();
Elements strings = new Elements();
for (Element element : elements) {
if (element.hasClass("post-feed-item")) {
list.add(element);
Element e = element.children().get(1).children().get(1).children().get(0);
if (e.text().matches("^.*?(Docker|docker|DOCKER).*$")) {
strings.add(e);
//save to element to DB
}
}
}
for (Element page : elements) {
if (links.add(URL)) {
//Remove the comment from the line below if you want to see it running on your editor
System.out.println(URL);
}
getPageLinks(page.attr("abs:href"));
}
I want if title from element contain : "Docker" it save my element to Database. But in element, It contain div and some thing link url, img , content. How to i save it to database. What if I want to save each element in a field in a database that is feasible? If not I can convert element to html and save it? Please help.
Example html i want save data base:
<div class="post-feed-item">
<img src="https://images.viblo.asia/avatar/1d0e5458-ad41-4d1c-89db-292dc198b4fa.png" srcset="https://images.viblo.asia/avatar/1d0e5458-ad41-4d1c-89db-292dc198b4fa.png 1x, https://images.viblo.asia/avatar-retina/1d0e5458-ad41-4d1c-89db-292dc198b4fa.png 2x" class="avatar avatar--md mr-05">
<div class="post-feed-item__info">
<div class="post-meta--inline">
<div class="user--inline d-inline-flex">
<!---->
Hoàn Kì
<!---->
</div>
<div class="post-meta d-inline-flex align-items-center flex-wrap">
<div class="text-muted mr-05">
<span class="mr-05">about 3 hours ago</span>
<button title="Copy URL" class="icon-btn _13z_mK0hRyRB3dPzawysKe_0"><i aria-hidden="true" class="fa fa-link"></i></button>
</div>
<!---->
<!---->
</div>
</div>
<div class="post-title--inline">
<h3 class="word-break mr-05">Docker: Chưa biết gì đến biết dùng (Phần 3 docker-compose )</h3>
<div class="tags" data-v-cbe11868>
<a href="/tags/docker" class="el-tag _3wKNDsArij9ZFjXe8k4ryR_0 el-tag--info el-tag--mini" data-v-cbe11868>Docker</a>
</div>
</div>
<!---->
<div class="d-flex justify-content-between">
<div class="d-flex">
<div class="stats">
<span title="Views" class="stats-item text-muted"><i aria-hidden="true" class="stats-item__icon fa fa-eye"></i> 62 </span>
<span title="Clips" class="stats-item text-muted"><i aria-hidden="true" class="stats-item__icon fa fa-paperclip"></i> 1 </span>
<span title="Comments" class="stats-item text-muted"><i aria-hidden="true" class="stats-item__icon fa fa-comments"></i> 0 </span>
</div>
<!---->
</div>
<div title="Score" class="points">
<div class="carets">
<i aria-hidden="true" class="fa fa-caret-up"></i>
<i aria-hidden="true" class="fa fa-caret-down"></i>
</div>
<span class="text-muted">4</span>
</div>
</div>
</div>
</div>

First, modify your logic for fetching post-feed-item like this-
Connection conn = Jsoup.connect("https://viblo.asia");
Document doc = conn.userAgent("Mozilla").get();
Elements elements = doc.getElementsByClass("post-feed-item"); //This will get the whole element.
for (Element element : elements) {
String postFeeds = "";
if (element.toString().contains("docker")) {
postFeeds = postFeeds.concat(element.toString());
//save postFeeds to DB
}
}
Extra
/**
* Your parsed element may contain single quote (').
* This will cause error while persisting.
* to avoid this you need to escape single quote (')
* with double single quote ('')
*/
if (element.toString().contains("docker")) {
postFeeds = postFeeds.concat(element.toString().replaceAll("'", "''"));
//save postFeeds to DB
}
Second, What if I want to save each element in a field in a database that is feasible?
You don't need separate columns to store each element at the database. However you can save but the feasibility depends on your use case. If you just want to store the post-feed-items only for writing it back to your web page then it is not feasible.
Third, How can I convert element to html and save?
You don't need to convert the element to html but you need to convert the element to String if you want to save it the database.
All you need is a column type of BLOB data type (you can also save it as VARCHAR but BLOB is safer).
Update
How can I traverse all pages?
By looking at the source code of that page I found this is how you can get the total page number -
Elements pagination = doc.getElementsByAttributeValueMatching("href", "page=\\d");
int totalPageNo = Integer.parseInt(pagination.get(pagination.size() - 2).text());
then loop through each page.
for(int page = 1; page <= totalPageNo; page++) {
Connection conn = Jsoup.connect("https://viblo.asia/?page=" + page);
//rest of your code
}

I properly know what's your mean.Here are some views:First you should clearify what`s your search for and make fields of tables in database. Such as according your ideas, you can make a table_docker table in db and there are field_id,field_content,field_start_time,field_links and so on in it. Second you should code some utils of classes such as JsoupUtils which is get HTML and parse it , HtmlUtils which is used to handle the html remarks and download these pictures,DBUtils which is used to connect db and save data,POIUtils which is used to show your data,DataUtils which is used to handle your data by your ways.

Select and iterate through elements and sub elements with same name (Jsoup)

I need to parse through a page by jsoup. The page has elements with tags div,h3,a etc. I want to parse through the elements and select a (i.e. title) to be displayed in jList.
As an example, the page looks like:
<div class="start">
<div class="g">
<div class="abc">
<a class="picture" href="www.img.com"><img src="img" alt="image1"></a>
<div class="xyz">
<h3 class="_r">
<a class="title" href="www.example.com" onmousedown="return rwt(this,'','','','1','adf','','ahahh','','',event)">THIS IS <em>example</em>1</a>
</h3>
</div>
</div>
</div>
<div class="g">
<div class="abc">
<a class="picture" href="www.img.com"><img src="img" alt="image2"></a>
<div class="xyz">
<h3 class="_r">
<a class="title" href="www.example.com" onmousedown="return rwt(this,'','','','1','adf','','ahahh','','',event)">lead by this<em>example</em></a>
</h3>
</div>
</div>
</div>
<div class="g">
<div class="abc">
<a class="picture" href="www.img.com"><img src="img" alt="image3"></a>
<div class="xyz">
<h3 class="_r">
<a class="title" href="www.example.com" onmousedown="return rwt(this,'','','','1','adf','','ahahh','','',event)">showed<em>example</em>for the people</a>
</h3>
</div>
</div>
</div>
<div class="g">
<div class="abc">
<a class="picture" href="www.img.com"><img src="img" alt="image4"></a>
<div class="xyz">
<h3 class="_r">
<a class="title" href="www.example.com" onmousedown="return rwt(this,'','','','1','adf','','ahahh','','',event)">we set<em>example</em>for people</a>
</h3>
</div>
</div>
</div>
</div>
This is the code:
String url = "http://www.google.com/search?q=example&tbm=nws&source=lnms";
String title = "";
try {
Document doc = Jsoup.connect(url).userAgent("Chrome").timeout(5000).get();
Elements e = doc.select("div.g");
for (Element e1 : e) {
title = e1.getElementsByTag("a").text();
}
DefaultListModel<String> listModel = new DefaultListModel<>();
listModel.addElement(title);
jList.setModel(listModel);
} catch (IOException ex) {
Logger.getLogger(MainUI.class.getName()).log(Level.SEVERE, null, ex);
}
The output that I got was the title of the last element div.g:
we set example for people
I want to select the title from each div.g and display each title separately in jList as item like this:
THIS IS example 1
lead by this example
showed example for the people
we set example for people

Currently you assign the scraped data to title in a loop and then outside the loop you assign title to the jlist. So, the value of title once the loop has completed will always be the last value.
Replace this ...
for (Element e1 : e) {
title = e1.getElementsByTag("a").text();
}
DefaultListModel<String> listModel = new DefaultListModel<>();
listModel.addElement(title);
With this ...
DefaultListModel<String> listModel = new DefaultListModel<>();
for (Element e1 : e) {
listModel.addElement(e1.getElementsByTag("a").text());
}

You actually don't add title each time. The loop replace each time title with the new value found and after the loop you add it in the list. Something like this might work the way you want it :
DefaultListModel<String> listModel = new DefaultListModel<>();
for (Element e1 : e) {
listModel.addElement(e1.getElementsByTag("a").text());
}

Can't figure out how to scrape specific text - Using Jsoup

I just started learning how to use JSoup. I think I've successfully selected this section of the html, and I successfully took "DARK SOULS III Deluxe Edition" out by doing .select("span.title").text but I was trying to get the prices, in this case $84.98 and $55.23. I tried doing .select("div.col search_price responsive_secondrow").text but it comes up as blank. I was wondering if someone could help me figure out how to extract that part, thanks in advance! Here's the html of the section of the page.
The full html is view-source:http://store.steampowered.com/search/?filter=topsellers
<a href="http://store.steampowered.com/sub/94174/?snr=1_7_7_topsellers_150_1" data-ds-packageid="94174" data-ds-appid="374320,442010"onmouseover="GameHover( this, event, 'global_hover', {"type":"sub","id":94174,"public":1,"v6":1} );" onmouseout="HideGameHover( this, event, 'global_hover' )" class="search_result_row ds_collapse_flag" >
<div class="col search_capsule"><img src="http://cdn.edgecast.steamstatic.com/steam/subs/94174/capsule_sm_120.jpg?t=1476893662"></div>
<div class="responsive_search_name_combined">
<div class="col search_name ellipsis">
<span class="title">DARK SOULS III Deluxe Edition</span>
<p>
<span class="platform_img win"></span> </p>
</div>
<div class="col search_released responsive_secondrow">12 Apr, 2016</div>
<div class="col search_reviewscore responsive_secondrow">
<span class="search_review_summary positive" data-store-tooltip="Very Positive<br>86% of the 29,204 user reviews for games in this bundle are positive.">
</span>
</div>
<div class="col search_price_discount_combined responsive_secondrow">
<div class="col search_discount responsive_secondrow">
<span>-35%</span>
</div>
<div class="col search_price discounted responsive_secondrow">
<span style="color: #888888;"><strike>$84.98</strike></span><br>$55.23 </div>
</div>
</div>
<div style="clear: left;"></div>
</a>

Use doc.select("a.search_result_row") instead:
public class JsoupSteamTest {
public static void main(String[] args) throws IOException {
Document doc = Jsoup.connect("http://store.steampowered.com/search/?filter=topsellers").userAgent("Mozilla")
.get();
Elements table = doc.select("a.search_result_row");
Iterator<Element> ite = table.iterator();
while (ite.hasNext()) {
Element element = ite.next();
System.out.println(element.text());
}
}
}
You will get a list like this:
PLAYERUNKNOWN'S BATTLEGROUNDS 23 Mar, 2017 29,99€
Steel Division: Normandy 44 Coming Soon 39,99€
DARK SOULS™ III 11 Apr, 2016 -50% 59,99€ 29,99€
Your particular problem comes from the div that has multiple classes.
To select an element that has multiple classes, use a dot instead of a space in your select:
doc.select("div.col.search_price.discounted.responsive_secondrow");
Take a look at this question: JSOUP get element with multiple classes

How to get input value by using class?

I am using the code below
WebElement inputele = driver.findElement(By.className("class_name"));
String inputeleval = inputele.getAttribute("value");
System.out.println(inputeleval);
but the value is empty. The HTML is below.
<div id="main">
<div id="hiddenresult">
<div class="tech-blog-list">
<label for="Question">1st Question</label>
<input id="txt60" class="form-control" type="text" value="sddf sd sdfsdf sdf sdfsdf sdfsdfsd fsd" />
</div>
</div>
<div class="pagination_main pull-left">
<div id="Pagination">
<div class="pagination">
<a class="previous" onclick="PreviousBtnClickEvent();" href="javascript:void(0)">Previous</a>
<a id="pg59" class="ep" onclick="PaginationBtnClickEvent(this);" href="javascript:void(0)" name="Textbox">1</a>
<a id="pg41" class="ep" onclick="PaginationBtnClickEvent(this);" href="javascript:void(0)" name="Textbox">2</a>
<a id="pg40" class="ep" onclick="PaginationBtnClickEvent(this);" href="javascript:void(0)" name="Textarea">3</a>
<a id="pg60" class="ep current" onclick="PaginationBtnClickEvent(this);" href="javascript:void(0)" name="Textbox">4</a>
</div>
</div>
</div>
</div>

Try using WebDriverWait to wait until element fully loaded on page and visible as below :-
WebDriverWait wait = new WebDriverWait(driver, 10);
WebElement inputele= wait.until(ExpectedConditions.visibilityOfElementLocated(By.className("class_name")));
String inputeleval = inputele.getAttribute("value");
System.out.println(inputeleval);
Note :-By.className("class_name") will give that element which class attribute equal to class_name. Make sure which element you want to locate is unique element with class attribute equal to class_name otherwise wise it will give first element with condition true.
Hope it will work..:)

Looks like your code is pretty close but you have the wrong class name? In your code above, you had "class_name" instead of "form-control". I'm assuming that was some sample code and not the actual code you are using? There is only one INPUT in the HTML and the code below should work. It also has an ID so that should be more specific in case there are more than one INPUTs on the page.
WebElement inputele= driver.findElement(By.className("form-control"));
String inputeleval = inputele.getAttribute("value");
System.out.println(inputeleval);