I'm new to selenium and following a small tutorial on scraping that scrapes jobs from indeed.com, but I am having issues as it seems some of the elements have been renamed since the tutorial was written. I'm stuck on this part:
List<WebElement> pagination = driver.findElements(By.xpath("//ul[#class='pagination-
list']/li"));
int pgSize = pagination.size();
for (int j = 1; j < pgSize; j++) {
Thread.sleep(1000);
WebElement pagei = driver.findElement(By.xpath("(//ul[#class='pagination-list']/li)[" + j + "]"));
pagei.click();
The xpath needs to be updated as the element can no longer be found. I have changed the xpath to this, but the list does not get populated:
//nav[#aria-label='pagination']
When I do this, it prints zero which means no elements have been added to the list:
List<WebElement> pagination = driver.findElements(By.xpath("//nav[#aria-label='pagination']"));
int size = pagination.size();
System.out.println(size);
Is this the correct xpath? I'm not certain as to what is supposed to get populated into the list of Webelements? Should they be page numbers?
You could try this path //nav[#role='navigation']/div This would give you 6 elements. 5 for the pages and 1 for the next button
If this is not the correct page, please share the URL and the element, I will try to provide a solution.
The website indeed.com have changed since the article Web Scraping using selenium and Java was written. The pagination elements have changed. Currently the pagination is implemented through:
<nav role="navigation" aria-label="pagination" class="css-jbuxu0 ecydgvn0">
<div class="css-tvvxwd ecydgvn1">
<button data-testid="pagination-page-current" class="css-1cpyzlr e8ju0x51">1</button>
</div>
<div class="css-tvvxwd ecydgvn1">
<a data-testid="pagination-page-2" rel="nofollow" aria-label="2" href="/jobs?q=Api+Testing&l=Pune&start=10" class="css-e9oyys e8ju0x50">2</a>
</div>
<div class="css-tvvxwd ecydgvn1">
<a data-testid="pagination-page-3" rel="nofollow" aria-label="3" href="/jobs?q=Api+Testing&l=Pune&start=20" class="css-e9oyys e8ju0x50">3</a>
</div>
<div class="css-tvvxwd ecydgvn1">
<a data-testid="pagination-page-4" rel="nofollow" aria-label="4" href="/jobs?q=Api+Testing&l=Pune&start=30" class="css-e9oyys e8ju0x50">4</a>
</div>
<div class="css-tvvxwd ecydgvn1">
<a data-testid="pagination-page-5" rel="nofollow" aria-label="5" href="/jobs?q=Api+Testing&l=Pune&start=40" class="css-e9oyys e8ju0x50">5</a>
</div>
<div class="css-tvvxwd ecydgvn1">
<a data-testid="pagination-page-next" rel="nofollow" aria-label="Next Page" href="/jobs?q=Api+Testing&l=Pune&start=10" class="css-cy0uue e8ju0x50">
<svg xmlns="http://www.w3.org/2000/svg" focusable="false" role="img" fill="currentColor" viewBox="0 0 24 24" aria-hidden="true" class="css-1xqhio eac13zx0">
<path d="M9.888 5.998a.506.506 0 00-.716-.008l-.707.707a.506.506 0 00.01.716L13.06 12l-4.587 4.587c-.2.2-.204.521-.009.716l.707.707a.507.507 0 00.717-.009l5.647-5.648c.1-.1.148-.233.144-.366a.492.492 0 00-.144-.34v-.001a.611.611 0 00-.009-.009L9.888 5.998z">
</path>
</svg>
</a>
</div>
</nav>
Solution
Instead of creating a list of the pagination elements, as the pagination keeps on increasing as you move forward you can use the following solution:
System.setProperty("webdriver.chrome.driver", "C:\\BrowserDrivers\\chromedriver.exe");
ChromeOptions options = new ChromeOptions();
options.addArguments("--start-maximized");
ChromeDriver driver = new ChromeDriver(options);
driver.get("https://in.indeed.com/");
WebDriverWait wait = new WebDriverWait(driver, Duration.ofSeconds(5));
// Create a Scanner object to prompt for user input
Scanner myObj = new Scanner(System.in);
System.out.println("What jobs are you looking for ? ");
String job = myObj.nextLine();
System.out.println("Which city are you looking for ? ");
String city = myObj.nextLine();
wait.until(ExpectedConditions.elementToBeClickable(By.cssSelector("input#text-input-what"))).sendKeys(job);
driver.findElement(By.cssSelector("input#text-input-where")).sendKeys(city + Keys.RETURN);
for (int j=1; j<6; j++)
{
driver.findElement(By.xpath("//button[#data-testid='pagination-page-current']//following::div[1]/a[#data-testid]")).click();
Thread.sleep(15000);
}
Note:
Here 6 is the number of (pages-1) you want to scrape.
Thread.sleep(15000) is to provide the window to pulate Email Address after the first click on pagination item.
Related
Say there is a website that loads data dynamically using Ajax calls. The data runs to 200 pages with about 50 rows per page. Pagination is generated at the bottom which is like
<div class="pagination">
<a class="paginate" id="previous">Previous</a>
<span>
<a class="paginate" id="p1">1</a>
<a class="paginate" id="p2">2</a>
<a class="paginate" id="p3">3</a>
<span class="ellipsis">…</span>
<a class="paginate" id="p200">200</a>
</span>
<a class="paginate" id="next">Next</a>
</div>
Using Selenium with Java, is it possible to dynamically generate a link such as
<a class="paginate" id="p100">100</a>
and click on it, so as to directly go to page 100?
See if this works:-
Random rand = new Random();
int lowerBound = 1;
int upperBound = 11;
int pageNum = rand.nextInt((upperBound - lowerBound)+1)+lowerBound;
System.out.println("Page num generated "+pageNum);
String xpathStr = ".//a[#id='p"+pageNum+"']";
System.out.println(xpathStr);
driver.findElement(By.xpath(xpathStr)).click();
Output:
Page num generated 8
.//a[#id='p8']
I am sending defined data from excel file. I tried some code but they are not selecting all the data from excel file at some point of time code is giving me exception for WebElement not found.
Here is the HTML code:
<div class="ui-dialog-titlebar ui-widget-header ui-corner-all ui-helper-clearfix">
<div id="addDialog" class="hidden ui-dialog-content ui-widget-content" style="display: block; width: auto; min-height: 30px; height: auto; max-height: 351.05px; overflow-y: auto;">
<div class="field-container">
<fieldset class="field-container">
<legend>Contracts:</legend>
<a class="select-all" href="#">Select All</a>
<a class="deselect-all" href="#">Deselect All</a>
<select id="addContract" class="searchable" multiple="multiple" style="position: absolute; left: -9999px;">
<option value="93370956">93370956</option>
<option value="93796167">93796167</option>
<option value="94203239">94203239</option>
</select>
<div id="ms-addContract" class="ms-container">
<div class="ms-selectable">
<input class="search-input" type="text" placeholder="filter" autocomplete="off"/>
<ul class="ms-list" tabindex="-1">
<li id="86355560-selectable" class="ms-elem-selectable">
<span>93370956</span>
</li>
<li id="202890296-selectable" class="ms-elem-selectable">
<span>93796167</span>
</li>
<li id="938848030-selectable" class="ms-elem-selectable">
<span>94203239</span>
</li>
</ul>
</div>
</div>
Need to select values from list.
Efforts done:
This effort for code worked but it selected only one value and then gave exception
WebDriverWait Wait=new WebDriverWait(driver, 10);
Wait.until(ExpectedConditions.presenceOfElementLocated(By.xpath("//ul[#class='ms-list']/li/span")));
//now you can find element
List<WebElement>options=driver.findElements(By.xpath("//ul[#class='ms-list']/li/span[contains(text(),'"+testData+"')]"));
for (WebElement option: options) {
if(testData.equals(option.getText())) option.click();
}
Tried above code but it only selects one value !!
WebDriverWait Wait = new WebDriverWait(driver, 10);
Wait.until(ExpectedConditions.presenceOfElementLocated(By.xpath("//*[#id='ms-addContract']//descendant::div[#class='ms-selectable']/ul[#class='ms-list']]//span")));
List<WebElement> options = driver.findElements(By.xpath("//*[#id='ms-addContract']//descendant::div[#class='ms-selectable']/ul[#class='ms-list']]//span[contains(text(), '"+testData+"')]"));
for (WebElement option : options) {
if(testData.equals(option.getText()))
option.click();--tried this xpath-no success
WebDriverWait wait = new WebDriverWait(driver, 10);
wait.until(ExpectedConditions.visibilityOfElementLocated(By.xpath(".//*[#id='ms-addContract']/div[1]/input"))).click();
driver.findElement(By.xpath(".//*[#id='ms-addContract']/div[1]/input")).sendKeys(testData);
WebDriverWait wait1 = new WebDriverWait(driver, 10);
wait1.until(ExpectedConditions.visibilityOfElementLocated(By.xpath(".//*[#id='ms-addContract']/div[1]/input"))).sendKeys(Keys.ARROW_DOWN,Keys.ARROW_DOWN,Keys.SPACE);
//Clear the input text value.
driver.findElement(By.xpath(".//*[#id='ms-addContract']/div[1]/input")).clear();---worked but not satisfactory
Please need help to locate the element. I am using keyword driven framework reading values from excel file.
It sounds like you are trying to get all span elements that are contained within a ul element (although correct me if I am wrong). This is the code you are using:
List<WebElement>options = driver.findElements(By.xpath("//ul[#class='ms-list']/li/span[contains(text(),'"+testData+"')]"));
In your case, this XPath will only return 1 value. This is because you are trying to match the contents of each span with an input value "testData". If you want to get all span elements inside of the ul then use:
List<WebElement>options = driver.findElements(By.xpath("//ul[#class='ms-list']/li/span"));
This XPath will select accomplish selecting all span elements in the list. If you need to select only span elements containing text within your testData, then you can iterate over that list selecting the appropriate elements:
ArrayList<WebElement> optionsInTestData = new ArrayList<WebElement>();
for(WebElement element: options){
for(String data: testData){
if(element.getText() == data){
optionsInTestData.add(element);
}
}
}
I have a web page contains 2 list boxes left & Right with transfer arrows.
I need to select an item from left and click on arrow so that it will move to rigtside list box.
I did my code like below, but it is not working.
List<WebElement> li = driver.findElements(By.xpath(".//*[#id='availableClients']/div/ul"));
for (WebElement lit : li)
{
System.out.println(lit.getText());
if (lit.getText().equalsIgnoreCase("CHKD"))
{
lit.click();
break;
}
}
HTML SOURCE
<div id="availableClients" class="left"> <label for="Available_Clients">Available Clients</label> <div class="list-swap-left"> <span style="width:0"/> <ul class="list-swap-list ui-sortable" title="Select the client to have access to this help link"> <li id="1" class="">abc</li> <li id="22" class="">CHKD</li> <li id="83" class="">Peg</li> <li id="95" class="">Sale</li>
Finally i myself got the solution:
// To Select the Item from List Box
WebElement xps = driver.findElement(By.xpath(xpath));
List<WebElement> sli = xps.findElements(By.tagName("li"));
for (int i = 0; i < sli.size(); i++)
{
if (sli.get(i).getText().equalsIgnoreCase(Itemtext))
{
sli.get(i).click();
break;
}
else{System.out.println("Item not fount in the list");}
}
//Click on Arrow Icon outside the List box to move the Item
driver.findElement(By.cssSelector("i.icn.cir-fwd")).click();
I'm having trouble with clicking at an element, which I find using text which is a variable. This is the code of the page:
<div class="recommendedProfileList fl">
<h3>
<ul class="ctrlResearchProfiles">
<li>
<li>
<li>
<li>
<li>
<li>
<span class="profileBtn ctrlSelectDefProfile ctrlClickSubmit" data-value="143" data-form="formChooseProfile" data-profileid="143">Sales manager</span>
<span class="profileTooltip" style="display: none;">
<span class="arrow"/>
<span class="profileTooltipContent">
</span>
and the name of the variable is profile. This is how I've tried to do this, but did not work:
WebDriverWait wait = new WebDriverWait(driver, 5);
wait.until(ExpectedConditions.elementToBeClickable(By.xpath("//*[text()=' + profile + ']")));
second:
driver.findElement(By.xpath("//*[text()=' + profile + ']"));
also:
driver.findElement(By.linkText("" +profile)).click();
Do you know how to click such element?
You are almost there buddy...
wait = new WebDriverWait(driver, 5);
wait.until(ExpectedConditions.elementToBeClickable(By.xpath("//*[text()='" + profile + "']")));
second:
driver.findElement(By.xpath("//*[text()='" + profile + "']"));
The thing that you missed was double quotes to insert ur variable values in xpath.
I am trying to automate the drop down in the website Naukri.com. That drop down consists of multi select check-boxes. How can we automate it using Selenium Web driver?
The structure of the drop list is:
<div class="DDwrap">
<ul class="DDsearch">
<li class="tagit" data-id="tg_indCja_a8_A">
<span class="tagTxt">Accounting , Finance</span>
<span class="dCross"></span>
</li>
<li class="frst" style="float: left;">
<input id="cjaInd" class="srchTxt" type="text" placeholder="" name="" autocomplete="off" style="width: 30px;">
<input id="hid_indCja" type="hidden" name="indType" value="["8"]">
</li>
</ul>
</div>
Can anyone help me regarding this?
Check out the code below, It navigates to the concerned form, opens the dropdown of "Industry" and selects two checkboxes: 'Accounting , Finance' and 'Government , Defence':
WebDriver driver = new FirefoxDriver(); //Opening firefox instance
driver.manage().window().maximize(); //maximizing window
driver.manage().timeouts().implicitlyWait(20, TimeUnit.SECONDS); //Giving implicit timeout of 20 seconds
driver.get("http://www.naukri.com/");
//Since there are two windows popping up, hence switching and closing the unnecessary window.
Set<String> windows = driver.getWindowHandles();
Iterator iter = windows.iterator();
String parentWindow = iter.next().toString();
String childWindow = iter.next().toString();
driver.switchTo().window(childWindow);
driver.close();
driver.switchTo().window(parentWindow);
//Hovering over "Jobs"
Actions act = new Actions(driver);
WebElement jobs = driver.findElement(By.xpath("//ul[#class='midSec menu']//div[.='Jobs']"));
act.moveToElement(jobs).build().perform();
//Clicking on "Advance Search"
WebElement Adv_search = driver.findElement(By.xpath("//ul[#class='midSec menu']/li[1]//a[.='Advanced Search']"));
act.moveToElement(Adv_search).click().perform();
//Clicking on the industry dropdown
driver.findElement(By.xpath("//div[#class='DDinputWrap']/input[contains(#placeholder,'Select the industry')]")).click();
//Selecting the checkbox containing text as "Accounting"
driver.findElement(By.xpath("//ul[#class='ChkboxEnb']//a[contains(text(),'Accounting')]")).click();
//Selecting the checkbox containing text as 'Government'
driver.findElement(By.xpath("//ul[#class='ChkboxEnb']//a[contains(text(),'Government')]")).click();