selenium fire StaleElementReferenceException - java

i try to make a web crawler with selenium.
My program fire a StaleElementReferenceException.
I thought that were because i crawl a page recursive and when a page have no more links the function navigate to next page and not previously to the parent page.
Therefore i have introduced a tree data structure to navigate back to the parent when the current url not equal the parent url. But this was not the solution for my problem.
Can anybody help me?
Code:
public class crawler {
private static FirefoxDriver driver;
private static String main_url = "https://robhammond.co/tools/seo-crawler";
private static List<String> uniqueLinks = new ArrayList<String>();
public static void main(String[] args) {
driver = new FirefoxDriver();
Node<String> root = new Node<>(main_url);
scrape(root, main_url);
}
public static void scrape(Node<String> node, String url) {
if(node.getParent() != null && (!driver.getCurrentUrl().equals(node.getParent().getData()))) {
driver.navigate().to(node.getParent().getData());
}
driver.navigate().to(url);
List<WebElement> allLinks = driver.findElements(By.tagName("a"));
for(WebElement link : allLinks) {
if(link.getAttribute("href").contains(main_url) && !uniqueLinks.contains(link.getAttribute("href")) && link.isDisplayed()) {
uniqueLinks.add(link.getAttribute("href"));
System.out.println(link.getAttribute("href"));
scrape(new Node<>(link.getAttribute("href")), link.getAttribute("href"));
}
}
}
}
And this is the output from the console:
D:\Programme\openjdk-12.0.1_windows-x64_bin\jdk-12.0.1\bin\java.exe "-javaagent:D:\Programme\JetBrains\IntelliJ IDEA 2019.1.2\lib\idea_rt.jar=60461:D:\Programme\JetBrains\IntelliJ IDEA 2019.1.2\bin" -Dfile.encoding=UTF-8 -classpath C:\Users\admin\Desktop\SeleniumWebScraper\out\production\SeleniumWebScraper;D:\Downloads\selenium-server-standalone-3.141.59.jar de.company.crawler.crawler
1557924446770 mozrunner::runner INFO Running command: "C:\\Program Files\\Mozilla Firefox\\firefox.exe" "-marionette" "-foreground" "-no-remote" "-profile" "C:\\Users\\admin\\AppData\\Local\\Temp\\rust_mozprofile.YqmEqE8y1pjv"
1557924447037 addons.webextension.screenshots#mozilla.org WARN Loading extension 'screenshots#mozilla.org': Reading manifest: Invalid extension permission: mozillaAddons
1557924447037 addons.webextension.screenshots#mozilla.org WARN Loading extension 'screenshots#mozilla.org': Reading manifest: Invalid extension permission: resource://pdf.js/
1557924447037 addons.webextension.screenshots#mozilla.org WARN Loading extension 'screenshots#mozilla.org': Reading manifest: Invalid extension permission: about:reader*
1557924448047 Marionette INFO Listening on port 60468
1557924448383 Marionette WARN TLS certificate errors will be ignored for this session
Mai 15, 2019 2:47:28 NACHM. org.openqa.selenium.remote.ProtocolHandshake createSession
INFO: Detected dialect: W3C
JavaScript warning: https://robhammond.co/js/jquery.min.js, line 4: Using //# to indicate sourceMappingURL pragmas is deprecated. Use //# instead
https://robhammond.co/tools/seo-crawler#content
https://twitter.com/intent/tweet?text=SEO%20Crawler&url=https://robhammond.co/tools/seo-crawler&via=robhammond
Exception in thread "main" org.openqa.selenium.StaleElementReferenceException: The element reference of <a href="/tools/"> is stale; either the element is no longer attached to the DOM, it is not in the current frame context, or the document has been refreshed
For documentation on this error, please visit: https://www.seleniumhq.org/exceptions/stale_element_reference.html
Build info: version: '3.141.59', revision: 'e82be7d358', time: '2018-11-14T08:25:53'
System info: host: 'DESKTOP-admin', ip: '192.168.233.1', os.name: 'Windows 10', os.arch: 'amd64', os.version: '10.0', java.version: '12.0.1'
Driver info: org.openqa.selenium.firefox.FirefoxDriver
Capabilities {acceptInsecureCerts: true, browserName: firefox, browserVersion: 66.0.5, javascriptEnabled: true, moz:accessibilityChecks: false, moz:geckodriverVersion: 0.24.0, moz:headless: false, moz:processID: 19124, moz:profile: C:\Users\admin\AppData\Loca..., moz:shutdownTimeout: 60000, moz:useNonSpecCompliantPointerOrigin: false, moz:webdriverClick: true, pageLoadStrategy: normal, platform: WINDOWS, platformName: WINDOWS, platformVersion: 10.0, rotatable: false, setWindowRect: true, strictFileInteractability: false, timeouts: {implicit: 0, pageLoad: 300000, script: 30000}, unhandledPromptBehavior: dismiss and notify}
Session ID: b3b87675-57c8-4b48-9a20-8df5e4d37503
at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:500)
at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:481)
at org.openqa.selenium.remote.http.W3CHttpResponseCodec.createException(W3CHttpResponseCodec.java:187)
at org.openqa.selenium.remote.http.W3CHttpResponseCodec.decode(W3CHttpResponseCodec.java:122)
at org.openqa.selenium.remote.http.W3CHttpResponseCodec.decode(W3CHttpResponseCodec.java:49)
at org.openqa.selenium.remote.HttpCommandExecutor.execute(HttpCommandExecutor.java:158)
at org.openqa.selenium.remote.service.DriverCommandExecutor.execute(DriverCommandExecutor.java:83)
at org.openqa.selenium.remote.RemoteWebDriver.execute(RemoteWebDriver.java:552)
at org.openqa.selenium.remote.RemoteWebElement.execute(RemoteWebElement.java:285)
at org.openqa.selenium.remote.RemoteWebElement.getAttribute(RemoteWebElement.java:134)
at de.company.crawler.crawler.scrape(crawler.java:33)
at de.company.crawler.crawler.scrape(crawler.java:38)
at de.company.crawler.crawler.main(crawler.java:20)
Process finished with exit code 1

When you navigate away from the first page all WebElements in the allLinks list get lost.
I would recommend converting it from the list of WebElement to the list of normal Strings like:
List<String> allLinksHrefs = allLinks.stream().map(link -> link.getAttribute("href")).collect(Collectors.toList());
and iterate through this new allLinksHrefs list instead.
You can use a hash-based collection for holding the uniqueLinks like HashSet - this way duplicates will be automatically eliminated
The current approach can take days to complete, consider using Selenium Grid and running your scraper in Parallel

Related

Java - Selenium ChromeDriver hangs out when new page is loaded

I am implementing a script that logs into a web page and perform certain actions, that are irrelevant in this case. This is the code that executes the login:
WebDriver driver;
JavascriptExecutor exec;
String dni = "...";
String passwd = "...";
driver = new ChromeDriver();
exec = (JavascriptExecutor) driver;
// Search web page
driver.get("http://example.com");
// Accept cookies
driver.findElement(By.id("aceptarCookies")).click();
// Login
driver.findElement(By.id("areaPersonal")).click();
driver.findElement(By.id("tab_login_user")).sendKeys(dni);
driver.findElement(By.id("tab_login_password")).sendKeys(passwd);
driver.findElement(By.id("loginFormHeader")).findElement(By.id("buttonHeader")).click();
The bot logs in successfully and, once logged in, if I want to execute any instruction I get very strange behaviors. If I try to search for any item within the home page it doesn't find it because it hasn't been loaded yet, for example adding this instruction just below:
// Search body item
driver.findElement(By.id("my_panel_page"));
However, if I add an implicit wait:
driver.manage().timeouts().implicitlyWait(20, TimeUnit.SECONDS);
// Search body item
driver.findElement(By.id("my_panel_page"));
Or an explicit wait:
WebDriverWait wait = new WebDriverWait(driver, 20);
wait.until(ExpectedConditions.presenceOfElementLocated(By.id("id_of_new_item")));
driver.findElement(By.id("my_panel_page"));
The script hangs and does not execute any more instructions. For example, if I add the following print, it never gets executed:
WebDriverWait wait = new WebDriverWait(driver, 20);
wait.until(ExpectedConditions.presenceOfElementLocated(By.id("id_of_new_item")));
System.out.println("Page loaded!!!");
driver.findElement(By.id("my_panel_page"));
The behavior is very strange, it is as if selenium does not work once the page has been loaded, it only works when the new page is still loading and the HTML objects I want to use are not yet available. I'm using the following versions:
ChromeDriver: 109.0.5414.74
Chrome: 109.0.5414.120
Selenium: 3.141.59
JDK: 11.0.16.1
I can't indicate specific data of the web page I'm using because it is detected as spam, but I have previously made sure that the items I'm searching on the script exists in the HTML code, in fact, the script worked correctly until a few months ago.
EDIT:
I add some examples of output to better understand the problem. If I execute the instruction driver.findElement(By.id("my_panel_page")); without using any type of wait the output is as follows:
... (Initialize ChromeDriver)
no such element: Unable to locate element: {"method":"css selector","selector":"#my_panel_page"}
(Session info: chrome=109.0.5414.120)
For documentation on this error, please visit: https://www.seleniumhq.org/exceptions/no_such_element.html
Build info: version: '3.141.59', revision: 'e82be7d358', time: '2018-11-14T08:17:03'
System info: host: 'xxx', ip: 'xxx', os.name: 'Windows 11', os.arch: 'amd64', os.version: '10.0', java.version: '11.0.16.1'
Driver info: org.openqa.selenium.chrome.ChromeDriver
Capabilities {acceptInsecureCerts: false, browserName: chrome, browserVersion: 109.0.5414.120, chrome: {chromedriverVersion: 109.0.5414.74 (e7c5703604da..., userDataDir: C:\Users\xxx\AppData\Loca...}, goog:chromeOptions: {debuggerAddress: localhost:52373}, javascriptEnabled: true, networkConnectionEnabled: false, pageLoadStrategy: normal, platform: WINDOWS, platformName: WINDOWS, proxy: Proxy(), setWindowRect: true, strictFileInteractability: false, timeouts: {implicit: 0, pageLoad: 300000, script: 30000}, unhandledPromptBehavior: dismiss and notify, webauthn:extension:credBlob: true, webauthn:extension:largeBlob: true, webauthn:virtualAuthenticators: true}
Session ID: 533a8a11a33254b8ab788522dada77ab
*** Element info: {Using=id, value=my_panel_page}
If I use some kind of wait I didn't get an output, the ChromeDriver initializes and logs into the web page successfully, but its execution hangs when it reaches the wait statement. It only prints the ChromeDriver initialization info:
Starting ChromeDriver 109.0.5414.74 (e7c5703604daa9cc128ccf5a5d3e993513758913-refs/branch-heads/5414#{#1172}) on port 15136
Only local connections are allowed.
Please see https://chromedriver.chromium.org/security-considerations for suggestions on keeping ChromeDriver safe.
ChromeDriver was started successfully.
feb. 01, 2023 5:28:28 P. M. org.openqa.selenium.remote.ProtocolHandshake createSession
INFORMACIËN: Detected dialect: W3C
presenceOfElementLocated()
presenceOfElementLocated() is the expectation for checking that an element is present on the DOM of a page. This does not necessarily mean that the element is visible.
Solution
So instead of presenceOfElementLocated(), you have to induce WebDriverWait for the visibilityOfElementLocated() and you can use the following solution:
So instead of presenceOfElementLocated(), you have to induce WebDriverWait for the visibilityOfElementLocated() and you can use the following solution:
try
{
new WebDriverWait(driver, 20).until(ExpectedConditions.visibilityOfElementLocated(By.id("id_of_new_item")));
System.out.println("new_item is visible");
System.out.println("Page loaded!!!");
driver.findElement(By.id("my_panel_page"));
// lines of the program
}
catch(TimeoutException e) {
System.out.println("new_item is not visible");
}
// remaining lines of the program

javascript error: Cannot read properties of undefined (reading 'element')

I am trying to run the following code:
WebElement body = driver.findElement(By.xpath("/html/body"));
js.executeScript("var injector = window.angular.element(argument[0]).injector(); var $http = injector.get('$http'); return ($http.pendingRequests.length === 0);",body);
I however get the following error:
org.openqa.selenium.JavascriptException: javascript error: Cannot read properties of undefined (reading 'element')
(Session info: chrome=103.0.5060.134)
Build info: version: '4.2.2', revision: '683ccb65d6'
System info: host: 'DESKTOP-UPVJF2U', ip: '192.168.0.132', os.name: 'Windows 10', os.arch: 'amd64', os.version: '10.0', java.version: '18.0.1.1'
Driver info: org.openqa.selenium.chrome.ChromeDriver
Command: [c6b4ce36aa115d9746065cead6c3271b, executeScript {script=var injector = window.angular.element(argument[0]).injector(); var $http = injector.get('$http'); return ($http.pendingRequests.length === 0);, args=[{ELEMENT=a0c33e06-9db9-49ed-984f-535b6bc72f7a, element-6066-11e4-a52e-4f735466cecf=a0c33e06-9db9-49ed-984f-535b6bc72f7a}]}]
Capabilities {acceptInsecureCerts: false, browserName: chrome, browserVersion: 103.0.5060.134, chrome: {chromedriverVersion: 103.0.5060.134 (8ec6fce403b..., userDataDir: C:\Users\win10\AppData\Loca...}, goog:chromeOptions: {debuggerAddress: localhost:53279}, networkConnectionEnabled: false, pageLoadStrategy: normal, platformName: WINDOWS, proxy: Proxy(), se:cdp: ws://localhost:53279/devtoo..., se:cdpVersion: 103.0.5060.134, setWindowRect: true, strictFileInteractability: false, timeouts: {implicit: 0, pageLoad: 300000, script: 30000}, unhandledPromptBehavior: dismiss and notify, webauthn:extension:credBlob: true, webauthn:extension:largeBlob: true, webauthn:virtualAuthenticators: true}
Session ID: c6b4ce36aa115d9746065cead6c3271b
at org.openqa.selenium.remote.codec.w3c.W3CHttpResponseCodec.createException(W3CHttpResponseCodec.java:200)
at org.openqa.selenium.remote.codec.w3c.W3CHttpResponseCodec.decode(W3CHttpResponseCodec.java:133)
at org.openqa.selenium.remote.codec.w3c.W3CHttpResponseCodec.decode(W3CHttpResponseCodec.java:53)
at org.openqa.selenium.remote.HttpCommandExecutor.execute(HttpCommandExecutor.java:184)
at org.openqa.selenium.remote.service.DriverCommandExecutor.invokeExecute(DriverCommandExecutor.java:167)
at org.openqa.selenium.remote.service.DriverCommandExecutor.execute(DriverCommandExecutor.java:142)
at org.openqa.selenium.remote.RemoteWebDriver.execute(RemoteWebDriver.java:588)
at org.openqa.selenium.remote.RemoteWebDriver.executeScript(RemoteWebDriver.java:522)
at com.vilcart.app.CustomerCreation.testCreation(CustomerCreation.java:118)
at java.base/java.util.ArrayList.forEach(ArrayList.java:1511)
... Removed 28 stack frames
Can you please let me know how i could resolve this issue?
Since you want to get arguments list element by index try to replace
argument[0]
with
arguments[0]
#suhail shaik You can try the following, assuming you want to check if all the pending $http requests are served by Angular in the background.
String angularReady = "return angular.element(document).injector().get('$http').pendingRequests.length === 0";
boolean isAngularReady = (boolean)js.executeScript(angularReady);
I have gone through this link which helped me find a solution. This has a way to put wait for angular version 5 which is working for my current angular version 11.
https://www.swtestacademy.com/selenium-wait-javascript-angular-ajax/

org.openqa.selenium.JavascriptException: javascript error: $ is not defined error taking coordinate screenshot with Ashot using ChromeDriver Selenium

I have encountered a weird problem while working with AShot.
Ashot works fine for whole screenShots but while selective screenShot it is BAD.
It throws error in CoordsProvider Class while getting co-ordinates of element.
am I using a faulty build or product?
public abstract class CoordsProvider implements Serializable {
public abstract Coords ofElement(WebDriver driver, WebElement element);
public Set<Coords> ofElements(WebDriver driver, Iterable<WebElement> elements) {
Set<Coords> elementsCoords = new HashSet<>();
for (WebElement element : elements) {
***Coords elementCoords = ofElement(driver, element); //fails here***
if (!elementCoords.isEmpty()) {
elementsCoords.add(elementCoords);
}
}
return Collections.unmodifiableSet(elementsCoords);
}
#SuppressWarnings("UnusedDeclaration")
public Set<Coords> ofElements(WebDriver driver, WebElement... elements) {
return ofElements(driver, Arrays.asList(elements));
}
#SuppressWarnings("UnusedDeclaration")
public Set<Coords> locatedBy(WebDriver driver, By locator) {
return ofElements(driver, driver.findElements(locator));
}
}
Stack trace
org.openqa.selenium.JavascriptException: javascript error: $ is not defined
(Session info: chrome=79.0.3945.130)
Build info: version: '3.14.0', revision: 'aacccce0', time: '2018-08-02T20:19:58.91Z'
System info: host: 'OPTIMIZEQ-LTP03', ip: '192.168.99.1', os.name: 'Windows 10', os.arch: 'amd64', os.version: '10.0', java.version: '11.0.2'
Driver info: org.openqa.selenium.remote.RemoteWebDriver
Capabilities {acceptInsecureCerts: false, browserName: chrome, browserVersion: 79.0.3945.130, chrome: {chromedriverVersion: 78.0.3904.105 (60e2d8774a81..., userDataDir: C:\Users\SHAILE~1.SIN\AppDa...}, goog:chromeOptions: {debuggerAddress: localhost:61006}, javascriptEnabled: true, networkConnectionEnabled: false, pageLoadStrategy: normal, platform: XP, platformName: XP, proxy: Proxy(manual, http=localhos..., setWindowRect: true, strictFileInteractability: false, timeouts: {implicit: 0, pageLoad: 300000, script: 30000}, unhandledPromptBehavior: dismiss and notify, webdriver.remote.sessionid: f5a47b27537f019dacb73462732...}
Session ID: f5a47b27537f019dacb734627324a790
at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[na:na]
at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[na:na]
at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[na:na]
at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:490) ~[na:na]
at org.openqa.selenium.remote.http.W3CHttpResponseCodec.createException(W3CHttpResponseCodec.java:187) ~[selenium-remote-driver-3.14.0.jar:na]
at org.openqa.selenium.remote.http.W3CHttpResponseCodec.decode(W3CHttpResponseCodec.java:122) ~[selenium-remote-driver-3.14.0.jar:na]
at org.openqa.selenium.remote.http.W3CHttpResponseCodec.decode(W3CHttpResponseCodec.java:49) ~[selenium-remote-driver-3.14.0.jar:na]
at org.openqa.selenium.remote.HttpCommandExecutor.execute(HttpCommandExecutor.java:158) ~[selenium-remote-driver-3.14.0.jar:na]
at org.openqa.selenium.remote.RemoteWebDriver.execute(RemoteWebDriver.java:548) ~[selenium-remote-driver-3.14.0.jar:na]
at org.openqa.selenium.remote.RemoteWebDriver.executeScript(RemoteWebDriver.java:485) ~[selenium-remote-driver-3.14.0.jar:na]
at ru.yandex.qatools.ashot.util.JsCoords.findCoordsWithJquery(JsCoords.java:30) ~[ashot-1.5.2.jar:na]
at ru.yandex.qatools.ashot.coordinates.JqueryCoordsProvider.ofElement(JqueryCoordsProvider.java:13) ~[ashot-1.5.2.jar:na]
at ru.yandex.qatools.ashot.coordinates.CoordsProvider.ofElements(CoordsProvider.java:21) ~[ashot-1.5.2.jar:na]
at ru.yandex.qatools.ashot.AShot.takeScreenshot(AShot.java:115) ~[ashot-1.5.2.jar:na]
at ru.yandex.qatools.ashot.AShot.takeScreenshot(AShot.java:132) ~[ashot-1.5.2.jar:na]
at com.optq.main.util.SeleniumDriverUtility.captureAShotElement(SeleniumDriverUtility.java:563) ~[classes/:na]
Any help or Alternatives For Ashot are welcome,Please help me into this.
This error message...
org.openqa.selenium.JavascriptException: javascript error: $ is not defined
(Session info: chrome=79.0.3945.130)
Build info: version: '3.14.0', revision: 'aacccce0', time: '2018-08-02T20:19:58.91Z'
System info: host: 'OPTIMIZEQ-LTP03', ip: '192.168.99.1', os.name: 'Windows 10', os.arch: 'amd64', os.version: '10.0', java.version: '11.0.2'
Driver info: org.openqa.selenium.remote.RemoteWebDriver
Capabilities {acceptInsecureCerts: false, browserName: chrome, browserVersion: 79.0.3945.130, chrome: {chromedriverVersion: 78.0.3904.105 (60e2d8774a81..., userDataDir: C:\Users\SHAILE~1.SIN\AppDa...}, goog:chromeOptions: {debuggerAddress: localhost:61006}, javascriptEnabled: true, networkConnectionEnabled: false, pageLoadStrategy: normal, platform: XP, platformName: XP, proxy: Proxy(manual, http=localhos..., setWindowRect: true, strictFileInteractability: false, timeouts: {implicit: 0, pageLoad: 300000, script: 30000}, unhandledPromptBehavior: dismiss and notify, webdriver.remote.sessionid: f5a47b27537f019dacb73462732...}
...implies that the ChromeDriver was unable to interact with the Browsing Context i.e. Chrome Browser session.
Deep dive
As per the documentation in ReferenceError: "x" is not defined this error means there is a non-existent variable referenced somewhere within the DOM Tree. This variable needs to be declared, or you need to make sure it is available in your current script or scope.
Hint: When loading a library (such as jQuery), make sure it is loaded before you access library variables, such as "$". Put the tag that loads the library before your code that uses it.
As per the discussion JavaScript/jQuery - “$ is not defined- $function()” error #Ketan mentions that this error occurs when you have not made jQuery available to your script, i.e. possibly the JavaScript / jQuery / AJAX haven't completed rendering the HTML DOM.
Solution
In these cases there are 3(three) different approaches available to solve the issue as follows:
The jQuery library is a single JavaScript file, and you reference it with the HTML <script> tag within the <head> section as follows:
<head>
<script src="jquery-3.4.1.min.js"></script>
</head>
This goes out and gets the jQuery code from the source.
Note: You do not have to include type="text/javascript" inside the <script> tag as this is not required in HTML5. JavaScript is the default scripting language in HTML5 and in all modern browsers.
You can also download the jQuery library and reference it locally on the server.
You can induce WebDriverWait inconjunction with ExpectedConditions for the element's desired state either to be present / visible / interactable.
You can find a detailed discussion in Selenium: How selenium identifies elements visible or not? Is is possible that it is loaded in DOM but not rendered on UI?
This usecase
As you mentioned, the following line fails:
Coords elementCoords = ofElement(driver, element);
This method takes one of the arguments (last) as element, but while defining you seem to be treating it as a list of elements, as in:
#SuppressWarnings("UnusedDeclaration")
public Set<Coords> ofElements(WebDriver driver, WebElement... elements) {
return ofElements(driver, Arrays.asList(elements));
}
Seems some mismatch in argument types here.
Additional considerations
You need to take care of a couple of things more:
You are using chromedriver=78.0.3904.105
Release Notes of chromedriver=78.0 clearly mentions the following :
Supports Chrome version 78
You are using chrome=79.0
Release Notes of ChromeDriver v79.0 clearly mentions the following :
Supports Chrome version 79
Your Selenium Client version is 3.14.0 of 2018-08-02T20:19:58.91Z which is almost 1.5 years older.
Your JDK version is 11.0.2.
So there is a clear mismatch between JDK v8u111 , Selenium Client v3.3.1 , ChromeDriver v2.41 and the Chrome Browser v79.0
Solution
Ensure that:
JDK is upgraded to current levels JDK 8u241.
Selenium is upgraded to current levels Version 3.141.59.
ChromeDriver is updated to current ChromeDriver v80.0 level.
Chrome is updated to current Chrome Version 80.0 level. (as per ChromeDriver v80.0 release notes)
I found a solution here, https://medium.com/virtualmind-io/jquery-injection-for-selenium-automation-tests-f6121ea57993. Basically it injects jquery into the page.
So I followed the steps and created this method that i used before calling AShot methods and could solve this issue, leaving the code here:
JavascriptExecutor js = (JavascriptExecutor) driver;
if(!(Boolean) js.executeScript("return (typeof jQuery != \"undefined\")")) {
js.executeScript(
"var headID = document.getElementsByTagName('head')[0];" +
"var newScript = document.createElement('script');" +
"newScript.type = 'text/javascript';" +
"newScript.src = 'https://ajax.googleapis.com/ajax/libs/jquery/3.3.1/jquery.min.js';" +
"headID.appendChild(newScript);");
WebDriverWait waitJQ = new WebDriverWait(driver, 30);
Function<WebDriver, Boolean> jQueryAvailable = WebDriver -> (
(Boolean) js.executeScript("return (typeof jQuery != \"undefined\")")
);
waitJQ.until(jQueryAvailable);
}

org.openqa.selenium.NoSuchElementException: Unable to locate element error while locating an element to extract the text using Selenium and Java

I would like to write some text using selenium but instead of output I see a lot of errors. However, the code worked before and I feel that. I did not overwrite any part of code.
I do not know what is wrong.
I don't know much about the output. Do you understand the faulty output?
import org.openqa.selenium.By;
import org.openqa.selenium.Keys;
import org.openqa.selenium.WebDriver;
import org.openqa.selenium.WebElement;
import org.openqa.selenium.firefox.FirefoxDriver;
import org.openqa.selenium.support.ui.WebDriverWait;
import static org.openqa.selenium.support.ui.ExpectedConditions.presenceOfElementLocated;
import java.time.Duration;
public class project_selenium {
public static void main(String[] args) {
WebDriver driver = new FirefoxDriver();
System.setProperty("webdriver.gecko.driver", "C:\\Users\\david\\Desktop\\dotazy\\geckodriver-v0.26.0 win64\\geckodriver.exe");
driver.get("https://edition.cnn.com/");
WebElement word1 = driver.findElement(By.xpath("/html[1]/body[1]/div[5]/div[1]/div[1]/header[1]/div[1]/div[1]/div[1]/div[2]/nav[1]/ul[1]/li[3]/a[1]"));
System.out.println(word1.getText());
//should return word "Business"
WebElement word2 = driver.findElement(By.className("politics"));
System.out.println(word2.getText());
//should return word "US Politics"
String currentURL = driver.getCurrentUrl();
System.out.println(currentURL);
}
}
There is my valid output:
1580136956525 mozrunner::runner INFO Running command: "C:\\Program Files\\Mozilla Firefox\\firefox.exe" "-marionette" "-foreground" "-no-remote" "-profile" "C:\\Users\\david\\AppData\\Local\\Temp\\rust_mozprofileduNDR7"
1580136956833 addons.webextension.doh-rollout#mozilla.org WARN Loading extension 'doh-rollout#mozilla.org': Reading manifest: Invalid extension permission: networkStatus
1580136956926 addons.webextension.screenshots#mozilla.org WARN Loading extension 'screenshots#mozilla.org': Reading manifest: Invalid extension permission: mozillaAddons
1580136956927 addons.webextension.screenshots#mozilla.org WARN Loading extension 'screenshots#mozilla.org': Reading manifest: Invalid extension permission: telemetry
1580136956927 addons.webextension.screenshots#mozilla.org WARN Loading extension 'screenshots#mozilla.org': Reading manifest: Invalid extension permission: resource://pdf.js/
1580136956927 addons.webextension.screenshots#mozilla.org WARN Loading extension 'screenshots#mozilla.org': Reading manifest: Invalid extension permission: about:reader*
JavaScript error: resource://gre/modules/XULStore.jsm, line 66: Error: Can't find profile directory.
1580136959269 Marionette INFO Listening on port 61686
1580136959745 Marionette WARN TLS certificate errors will be ignored for this session
led 27, 2020 3:55:59 ODP. org.openqa.selenium.remote.ProtocolHandshake createSession
INFO: Detected dialect: W3C
JavaScript warning: https://confiant-integrations.global.ssl.fastly.net/gpt/202001101133/wrap.js, line 8: unreachable code after return statement
JavaScript warning: https://confiant-integrations.global.ssl.fastly.net/gpt/202001101133/wrap.js, line 8: unreachable code after return statement
JavaScript warning: https://confiant-integrations.global.ssl.fastly.net/gpt/202001101133/wrap.js, line 8: unreachable code after return statement
JavaScript warning: https://confiant-integrations.global.ssl.fastly.net/gpt/202001101133/wrap.js, line 8: unreachable code after return statement
JavaScript warning: https://confiant-integrations.global.ssl.fastly.net/gpt/202001101133/wrap.js, line 8: unreachable code after return statement
JavaScript warning: https://confiant-integrations.global.ssl.fastly.net/gpt/202001101133/wrap.js, line 8: unreachable code after return statement
JavaScript warning: https://confiant-integrations.global.ssl.fastly.net/gpt/202001101133/wrap.js, line 8: unreachable code after return statement
JavaScript warning: https://confiant-integrations.global.ssl.fastly.net/gpt/202001101133/wrap.js, line 8: unreachable code after return statement
Exception in thread "main" org.openqa.selenium.NoSuchElementException: Unable to locate element: /html[1]/body[1]/div[5]/div[1]/div[1]/header[1]/div[1]/div[1]/div[1]/div[2]/nav[1]/ul[1]/li[3]/a[1]
For documentation on this error, please visit: https://www.seleniumhq.org/exceptions/no_such_element.html
Build info: version: '3.141.59', revision: 'e82be7d358', time: '2018-11-14T08:25:48'
System info: host: 'DESKTOP-7SSQHFU', ip: '192.168.31.1', os.name: 'Windows 10', os.arch: 'amd64', os.version: '10.0', java.version: '13'
Driver info: org.openqa.selenium.firefox.FirefoxDriver
Capabilities {acceptInsecureCerts: true, browserName: firefox, browserVersion: 72.0.2, javascriptEnabled: true, moz:accessibilityChecks: false, moz:buildID: 20200117190643, moz:geckodriverVersion: 0.26.0, moz:headless: false, moz:processID: 19352, moz:profile: C:\Users\david\AppData\Loca..., moz:shutdownTimeout: 60000, moz:useNonSpecCompliantPointerOrigin: false, moz:webdriverClick: true, pageLoadStrategy: normal, platform: WINDOWS, platformName: WINDOWS, platformVersion: 10.0, rotatable: false, setWindowRect: true, strictFileInteractability: false, timeouts: {implicit: 0, pageLoad: 300000, script: 30000}, unhandledPromptBehavior: dismiss and notify}
Session ID: c96eed46-c306-4563-9346-c7016b1ee855
*** Element info: {Using=xpath, value=/html[1]/body[1]/div[5]/div[1]/div[1]/header[1]/div[1]/div[1]/div[1]/div[2]/nav[1]/ul[1]/li[3]/a[1]}
at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:500)
at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:481)
at org.openqa.selenium.remote.http.W3CHttpResponseCodec.createException(W3CHttpResponseCodec.java:187)
at org.openqa.selenium.remote.http.W3CHttpResponseCodec.decode(W3CHttpResponseCodec.java:122)
at org.openqa.selenium.remote.http.W3CHttpResponseCodec.decode(W3CHttpResponseCodec.java:49)
at org.openqa.selenium.remote.HttpCommandExecutor.execute(HttpCommandExecutor.java:158)
at org.openqa.selenium.remote.service.DriverCommandExecutor.execute(DriverCommandExecutor.java:83)
at org.openqa.selenium.remote.RemoteWebDriver.execute(RemoteWebDriver.java:552)
at org.openqa.selenium.remote.RemoteWebDriver.findElement(RemoteWebDriver.java:323)
at org.openqa.selenium.remote.RemoteWebDriver.findElementByXPath(RemoteWebDriver.java:428)
at org.openqa.selenium.By$ByXPath.findElement(By.java:353)
at org.openqa.selenium.remote.RemoteWebDriver.findElement(RemoteWebDriver.java:315)
at project_selenium.main(project_selenium.java:20)
Process finished with exit code 1
You should be using dynamic selectors. http://pragmatictestlabs.com/2018/05/16/mastering-xpath-for-selenium-test-automation/
This is certainly a tough xpath if you're just starting but if you want to click the "Business" button in the header it would be:
driver.findElement(By.xpath("//header[#id='header-nav-container'] //div[contains(#class,'NavGrid')] [not(#style='false:unset')]//a[text()='Business']"));
As for all the error messages, I'm not sure why it's throwing them. What's missing actually is the Exception that should be thrown when the findElement method failed. I believe it should be a NoSuchElement Exception.
Can you surround your code with a try/catch block and throw an Exception like so:
try {
//code
} catch (Exception e) {
System.out.println(e)
}
EDIT: Actually, it is throwing the NoSuchElementException. I guess Geckodriver throws it in the midst of other log entries.
Exception in thread "main" org.openqa.selenium.NoSuchElementException: Unable to locate element: /html[1]/body[1]/div[5]/div[1]/div[1]/header[1]/div[1]/div[1]/div[1]/div[2]/nav[1]/ul[1]/li[3]/a[1]
To extract the text Business you need to induce WebDriverWait for the visibilityOfElementLocated() and you can use either of the following Locator Strategies:
cssSelector:
System.out.println(new WebDriverWait(driver, 20).until(ExpectedConditions.visibilityOfElementLocated(By.cssSelector("li[data-section='business']>a[name='business'][data-analytics='header_top-nav']"))).getText());
xpath:
System.out.println(new WebDriverWait(driver, 20).until(ExpectedConditions.visibilityOfElementLocated(By.xpath("//li[#data-section='business']/a[#name='business' and #data-analytics='header_top-nav']"))).getText());

Problem to select a button using selenium webdriver

I am trying of different ways to select a specific button using seleninum webdriver with java but unfortunately, nothing is working.
When I tested using the Selenium IDE is working. I copied the same xpath, for example, but when I try to test in my java application nothing is working. I tried using different ways, By.cssSelector and By.path.
This is my html:
<section class="fd-section"><fd-action-bar><div class="fd-action-bar"><fd-action-bar-header class="fd-action-bar__header"><fd-action-bar-title><h1 class="fd-action-bar__title"> Applications </h1></fd-action-bar-title></fd-action-bar-header><fd-action-bar-actions class="fd-action-bar__actions"><y-list-search _nghost-c4="" hidden=""><!----><!----><div _ngcontent-c4="" clickoutsideevents="click,mousedown" excludebeforeclick="true" class="ng-star-inserted"><!----><button _ngcontent-c4="" fd-button="" class="fd-button xyz-icon--search fd-button--light ng-star-inserted"></button><!----></div></y-list-search><y-list-filter _nghost-c5="" hidden=""><!----></y-list-filter><!----><button class="open-create-namespace-modal fd-button xyz-icon--add ng-star-inserted" fd-button=""> Create Application </button></fd-action-bar-actions></div></fd-action-bar></section>
I need to select the button with the text " Create Application ".
When I created a test using Selenium IDE the xpath for this button is:
//button[contains(.,' Create Application')]
Basically, my java code is:
public WebElement wElement;
wElement = driver.findElement(By.xpath("//button[contains(.,' Create Application')]"));
wElement.click();
This is the exception message:
org.openqa.selenium.NoSuchElementException: no such element: Unable to locate element: {"method":"xpath","selector":"//button[contains(.,' Create Application')]"}
(Session info: chrome=76.0.3809.100)
(Driver info: chromedriver=72.0.3626.69 (3c16f8a135abc0d4da2dff33804db79b849a7c38),platform=Mac OS X 10.14.6 x86_64) (WARNING: The server did not provide any stacktrace information)
Command duration or timeout: 0 milliseconds
For documentation on this error, please visit: http://seleniumhq.org/exceptions/no_such_element.html
Build info: version: '3.14.0', revision: 'aacccce0', time: '2018-08-02T20:19:58.91Z'
System info: host: 'C02WW0BZHTD8', ip: 'fe80:0:0:0:8f6:17e1:1a28:1e23%en0', os.name: 'Mac OS X', os.arch: 'x86_64', os.version: '10.14.6', java.version: '1.8.0_171'
Driver info: org.openqa.selenium.chrome.ChromeDriver
Capabilities {acceptInsecureCerts: false, acceptSslCerts: false, applicationCacheEnabled: false, browserConnectionEnabled: false, browserName: chrome, chrome: {chromedriverVersion: 72.0.3626.69 (3c16f8a135abc..., userDataDir: /var/folders/2r/99nyn7t16cz...}, cssSelectorsEnabled: true, databaseEnabled: false, goog:chromeOptions: {debuggerAddress: localhost:60374}, handlesAlerts: true, hasTouchScreen: false, javascriptEnabled: true, locationContextEnabled: true, mobileEmulationEnabled: false, nativeEvents: true, networkConnectionEnabled: false, pageLoadStrategy: normal, platform: MAC, platformName: MAC, proxy: Proxy(), rotatable: false, setWindowRect: true, takesHeapSnapshot: true, takesScreenshot: true, timeouts: {implicit: 0, pageLoad: 300000, script: 30000}, unexpectedAlertBehaviour: ignore, unhandledPromptBehavior: ignore, version: 76.0.3809.100, webStorageEnabled: true}
Session ID: b2341899cd9b62b0169b02371aaa3018
*** Element info: {Using=xpath, value=//button[contains(.,' Create Application')]}
When this piece of code is being executed, is the button loaded into the page already?
{implicit: 0, pageLoad: 300000, script: 30000}, suggests that the driver will not implicitly wait for finding any elements. i.e. if the element is not available, it will throw the exception immediately.
Try driver.manage().timeouts().implicitlyWait(10,TimeUnit.SECONDS); before you attempt to find the button.
Also try using below locators -
//button[contains(.,'Create Application')]
//button[contains(text(),'Create Application')]
If none of the above works, please can you provide a URL (if it is public)
Also check whether the button is inside a frame.
You can use JavascriptExecutor interface in this case and try to click the button.
WebElement element = driver.findElement(By.xpath("//button[contains(text(),'Create Application')]"));
JavascriptExecutor executor = (JavascriptExecutor)driver;
executor.executeScript("arguments[0].click();", element);
Also give a try with WebDriverWait
new WebDriverWait(driver, 20).until(ExpectedConditions.elementToBeClickable(By.xpath("//button[contains(text(),'Create Application')]"))).click();
The desired element is an Angular element so to locate the element you have to induce WebDriverWait for the elementToBeClickable() and you can use either of the following Locator Strategies:
cssSelector:
new WebDriverWait(driver, 20).until(ExpectedConditions.elementToBeClickable(By.cssSelector("fd-action-bar-actions.fd-action-bar__actions button.open-create-namespace-modal.fd-button.xyz-icon--add.ng-star-inserted"))).click();
xpath:
new WebDriverWait(driver, 20).until(ExpectedConditions.elementToBeClickable(By.xpath("//button[#class='open-create-namespace-modal fd-button xyz-icon--add ng-star-inserted' and contains(., 'Create Application')]"))).click();
Additional Considerations
Ensure that:
JDK is upgraded to current levels JDK 8u212.
Selenium is upgraded to current levels Version 3.141.59.
ChromeDriver is updated to current ChromeDriver v76.0 level.
Chrome is updated to current Chrome Version 76.0 level. (as per ChromeDriver v76.0 release notes)
Here you can find a detailed discussion on NoSuchElementException, Selenium unable to locate element

Categories