We Keep Coding

Java - Selenium ChromeDriver hangs out when new page is loaded

I am implementing a script that logs into a web page and perform certain actions, that are irrelevant in this case. This is the code that executes the login:
WebDriver driver;
JavascriptExecutor exec;
String dni = "...";
String passwd = "...";
driver = new ChromeDriver();
exec = (JavascriptExecutor) driver;
// Search web page
driver.get("http://example.com");
// Accept cookies
driver.findElement(By.id("aceptarCookies")).click();
// Login
driver.findElement(By.id("areaPersonal")).click();
driver.findElement(By.id("tab_login_user")).sendKeys(dni);
driver.findElement(By.id("tab_login_password")).sendKeys(passwd);
driver.findElement(By.id("loginFormHeader")).findElement(By.id("buttonHeader")).click();
The bot logs in successfully and, once logged in, if I want to execute any instruction I get very strange behaviors. If I try to search for any item within the home page it doesn't find it because it hasn't been loaded yet, for example adding this instruction just below:
// Search body item
driver.findElement(By.id("my_panel_page"));
However, if I add an implicit wait:
driver.manage().timeouts().implicitlyWait(20, TimeUnit.SECONDS);
// Search body item
driver.findElement(By.id("my_panel_page"));
Or an explicit wait:
WebDriverWait wait = new WebDriverWait(driver, 20);
wait.until(ExpectedConditions.presenceOfElementLocated(By.id("id_of_new_item")));
driver.findElement(By.id("my_panel_page"));
The script hangs and does not execute any more instructions. For example, if I add the following print, it never gets executed:
WebDriverWait wait = new WebDriverWait(driver, 20);
wait.until(ExpectedConditions.presenceOfElementLocated(By.id("id_of_new_item")));
System.out.println("Page loaded!!!");
driver.findElement(By.id("my_panel_page"));
The behavior is very strange, it is as if selenium does not work once the page has been loaded, it only works when the new page is still loading and the HTML objects I want to use are not yet available. I'm using the following versions:
ChromeDriver: 109.0.5414.74
Chrome: 109.0.5414.120
Selenium: 3.141.59
JDK: 11.0.16.1
I can't indicate specific data of the web page I'm using because it is detected as spam, but I have previously made sure that the items I'm searching on the script exists in the HTML code, in fact, the script worked correctly until a few months ago.
EDIT:
I add some examples of output to better understand the problem. If I execute the instruction driver.findElement(By.id("my_panel_page")); without using any type of wait the output is as follows:
... (Initialize ChromeDriver)
no such element: Unable to locate element: {"method":"css selector","selector":"#my_panel_page"}
(Session info: chrome=109.0.5414.120)
For documentation on this error, please visit: https://www.seleniumhq.org/exceptions/no_such_element.html
Build info: version: '3.141.59', revision: 'e82be7d358', time: '2018-11-14T08:17:03'
System info: host: 'xxx', ip: 'xxx', os.name: 'Windows 11', os.arch: 'amd64', os.version: '10.0', java.version: '11.0.16.1'
Driver info: org.openqa.selenium.chrome.ChromeDriver
Capabilities {acceptInsecureCerts: false, browserName: chrome, browserVersion: 109.0.5414.120, chrome: {chromedriverVersion: 109.0.5414.74 (e7c5703604da..., userDataDir: C:\Users\xxx\AppData\Loca...}, goog:chromeOptions: {debuggerAddress: localhost:52373}, javascriptEnabled: true, networkConnectionEnabled: false, pageLoadStrategy: normal, platform: WINDOWS, platformName: WINDOWS, proxy: Proxy(), setWindowRect: true, strictFileInteractability: false, timeouts: {implicit: 0, pageLoad: 300000, script: 30000}, unhandledPromptBehavior: dismiss and notify, webauthn:extension:credBlob: true, webauthn:extension:largeBlob: true, webauthn:virtualAuthenticators: true}
Session ID: 533a8a11a33254b8ab788522dada77ab
*** Element info: {Using=id, value=my_panel_page}
If I use some kind of wait I didn't get an output, the ChromeDriver initializes and logs into the web page successfully, but its execution hangs when it reaches the wait statement. It only prints the ChromeDriver initialization info:
Starting ChromeDriver 109.0.5414.74 (e7c5703604daa9cc128ccf5a5d3e993513758913-refs/branch-heads/5414#{#1172}) on port 15136
Only local connections are allowed.
Please see https://chromedriver.chromium.org/security-considerations for suggestions on keeping ChromeDriver safe.
ChromeDriver was started successfully.
feb. 01, 2023 5:28:28 P. M. org.openqa.selenium.remote.ProtocolHandshake createSession
INFORMACIËN: Detected dialect: W3C

presenceOfElementLocated()
presenceOfElementLocated() is the expectation for checking that an element is present on the DOM of a page. This does not necessarily mean that the element is visible.
Solution
So instead of presenceOfElementLocated(), you have to induce WebDriverWait for the visibilityOfElementLocated() and you can use the following solution:
So instead of presenceOfElementLocated(), you have to induce WebDriverWait for the visibilityOfElementLocated() and you can use the following solution:
try
{
new WebDriverWait(driver, 20).until(ExpectedConditions.visibilityOfElementLocated(By.id("id_of_new_item")));
System.out.println("new_item is visible");
System.out.println("Page loaded!!!");
driver.findElement(By.id("my_panel_page"));
// lines of the program
}
catch(TimeoutException e) {
System.out.println("new_item is not visible");
}
// remaining lines of the program

Selenium java code runs test successfully but fails when run as headless

My java selenium java test case with Chrome version 77 runs fine when options.setHeadless(false); but fails when I change this line options.setHeadless(true);
The java code was exported by the katalon selenium recorder chrome extension.
Output when it runs fine with options.setHeadless(false);:
Title of the page is 9 -> Scan History
Title of the page is 10 -> Scan History
Title of the page is 10.1 -> Scan History
Title of the page is 10.2 -> Scan History
Title of the page is 10.3 -> Scan History
Title of the page is 10.4 -> Scan History
Title of the page is 10.5 -> Scan History
When I run the same code with options.setHeadless(true); I get a below warning as well as an error message and the execution terminates before completion.
Title of the page is 10 -> Scan History
[0530/125542.102:INFO:CONSOLE(0)] "Error parsing header X-XSS-Protection: 1; mod
e=block, 1;mode=block: expected semicolon at character position 13. The default
protections will be applied.", source: https://qualysguard.myshop.com/fo/scan/sca
nList.php (0)
Title of the page is 10.1 -> Scan History
Title of the page is 10.2 -> Scan History
Title of the page is 10.3 -> Scan History
Exception in thread "main" org.openqa.selenium.NoSuchElementException: no such e
lement: Unable to locate element: {"method":"css selector","selector":"#ext\-gen
117"}
(Session info: headless chrome=77.0.3865.75)
For documentation on this error, please visit: https://www.seleniumhq.org/except
ions/no_such_element.html
Build info: version: '3.141.59', revision: 'e82be7d358', time: '2018-11-14T08:17
:03'
System info: host: 'myhostserver', ip: '10.9.111.32', os.name: 'Windows Serve
r 2012 R2', os.arch: 'amd64', os.version: '6.3', java.version: '1.8.0_45'
Driver info: org.openqa.selenium.chrome.ChromeDriver
Capabilities {acceptInsecureCerts: false, browserName: chrome, browserVersion: 7
7.0.3865.75, chrome: {chromedriverVersion: 77.0.3865.40 (f484704e052e0..., userD
ataDir: C:\Users\1886\AppData\Loc...}, goog:chromeOptions: {debuggerAddress: l
ocalhost:56128}, javascriptEnabled: true, networkConnectionEnabled: false, pageL
oadStrategy: normal, platform: XP, platformName: XP, proxy: Proxy(), setWindowRe
ct: true, strictFileInteractability: false, timeouts: {implicit: 0, pageLoad: 30
0000, script: 30000}, unhandledPromptBehavior: dismiss and notify}
Session ID: 09a309a747f7bcb42735bb8b6c39ad1f
*** Element info: {Using=id, value=ext-gen117}
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(Unknown Source)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown Sou
rce)
at java.lang.reflect.Constructor.newInstance(Unknown Source)
at org.openqa.selenium.remote.http.W3CHttpResponseCodec.createException(
W3CHttpResponseCodec.java:187)
at org.openqa.selenium.remote.http.W3CHttpResponseCodec.decode(W3CHttpRe
sponseCodec.java:122)
at org.openqa.selenium.remote.http.W3CHttpResponseCodec.decode(W3CHttpRe
sponseCodec.java:49)
at org.openqa.selenium.remote.HttpCommandExecutor.execute(HttpCommandExe
cutor.java:158)
at org.openqa.selenium.remote.service.DriverCommandExecutor.execute(Driv
erCommandExecutor.java:83)
at org.openqa.selenium.remote.RemoteWebDriver.execute(RemoteWebDriver.ja
va:552)
at org.openqa.selenium.remote.RemoteWebDriver.findElement(RemoteWebDrive
r.java:323)
at org.openqa.selenium.remote.RemoteWebDriver.findElementById(RemoteWebD
river.java:372)
at org.openqa.selenium.By$ById.findElement(By.java:188)
at org.openqa.selenium.remote.RemoteWebDriver.findElement(RemoteWebDrive
r.java:315)
at pack.QualysScan.testQualysScan(QualysScan.java:182)
at pack.QualysScan.main(QualysScan.java:284)
Here is the part of the code concerning the issue:
System.out.println("Title of the page is 10 -> " + driver.getTitle());
driver.findElement(By.id("body-header")).click();
driver.get("https://qualysguard.myshop.com/fo/scan/scanList.php");
System.out.println("Title of the page is 10.1 -> " + driver.getTitle());
driver.findElement(By.xpath("(.//*[normalize-space(text()) and normalize-space(.)='Add to my Calendar'])[1]/following::em[2]")).click();
System.out.println("Title of the page is 10.2 -> " + driver.getTitle());
System.out.println("Title of the page is 10.3 -> " + driver.getTitle());
driver.findElement(By.id("ext-gen117")).click();
System.out.println("Title of the page is 10.4 -> " + driver.getTitle());
System.out.println("Title of the page is 10.5 -> " + driver.getTitle());
String your_title = "Launch Compliance Scan";
And here is how the driver is setup:
public static void setUp() throws Exception {
System.setProperty("webdriver.chrome.driver", "H:\\Downloads\\Qualys\\vdi\\chromedriver.exe");
ChromeOptions options = new ChromeOptions();
options.setHeadless(true);
DesiredCapabilities capabilities = DesiredCapabilities.chrome();
capabilities.setCapability(CapabilityType.ForSeleniumServer.ENSURING_CLEAN_SESSION, true);
capabilities.setCapability("chrome.switches", Arrays.asList("--incognito"));
options.merge(capabilities);
driver = new ChromeDriver(options);
driver.manage().timeouts().implicitlyWait(60, TimeUnit.SECONDS);
}
I'm sharing snapshot of element ext\-gen117 as visible on the portal.
Video of the issue:
https://www.youtube.com/watch?v=abSwsPtOOG4
Can you please suggest how can I overcome the issue?

This is likely caused by a race condition between the browser and selenium, where selenium attempts to click on something before it exists in the DOM. Using an explicit wait should resolve the issue:
WebDriverWait wait = new WebDriverWait (driver, 30);
WebElement element = wait.until(ExpectedConditions.elementToBeClickable(By.id("ext-gen117");
element.click();

org.openqa.selenium.JavascriptException: javascript error: $ is not defined error taking coordinate screenshot with Ashot using ChromeDriver Selenium

I have encountered a weird problem while working with AShot.
Ashot works fine for whole screenShots but while selective screenShot it is BAD.
It throws error in CoordsProvider Class while getting co-ordinates of element.
am I using a faulty build or product?
public abstract class CoordsProvider implements Serializable {
public abstract Coords ofElement(WebDriver driver, WebElement element);
public Set<Coords> ofElements(WebDriver driver, Iterable<WebElement> elements) {
Set<Coords> elementsCoords = new HashSet<>();
for (WebElement element : elements) {
***Coords elementCoords = ofElement(driver, element); //fails here***
if (!elementCoords.isEmpty()) {
elementsCoords.add(elementCoords);
}
}
return Collections.unmodifiableSet(elementsCoords);
}
#SuppressWarnings("UnusedDeclaration")
public Set<Coords> ofElements(WebDriver driver, WebElement... elements) {
return ofElements(driver, Arrays.asList(elements));
}
#SuppressWarnings("UnusedDeclaration")
public Set<Coords> locatedBy(WebDriver driver, By locator) {
return ofElements(driver, driver.findElements(locator));
}
}
Stack trace
org.openqa.selenium.JavascriptException: javascript error: $ is not defined
(Session info: chrome=79.0.3945.130)
Build info: version: '3.14.0', revision: 'aacccce0', time: '2018-08-02T20:19:58.91Z'
System info: host: 'OPTIMIZEQ-LTP03', ip: '192.168.99.1', os.name: 'Windows 10', os.arch: 'amd64', os.version: '10.0', java.version: '11.0.2'
Driver info: org.openqa.selenium.remote.RemoteWebDriver
Capabilities {acceptInsecureCerts: false, browserName: chrome, browserVersion: 79.0.3945.130, chrome: {chromedriverVersion: 78.0.3904.105 (60e2d8774a81..., userDataDir: C:\Users\SHAILE~1.SIN\AppDa...}, goog:chromeOptions: {debuggerAddress: localhost:61006}, javascriptEnabled: true, networkConnectionEnabled: false, pageLoadStrategy: normal, platform: XP, platformName: XP, proxy: Proxy(manual, http=localhos..., setWindowRect: true, strictFileInteractability: false, timeouts: {implicit: 0, pageLoad: 300000, script: 30000}, unhandledPromptBehavior: dismiss and notify, webdriver.remote.sessionid: f5a47b27537f019dacb73462732...}
Session ID: f5a47b27537f019dacb734627324a790
at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[na:na]
at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[na:na]
at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[na:na]
at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:490) ~[na:na]
at org.openqa.selenium.remote.http.W3CHttpResponseCodec.createException(W3CHttpResponseCodec.java:187) ~[selenium-remote-driver-3.14.0.jar:na]
at org.openqa.selenium.remote.http.W3CHttpResponseCodec.decode(W3CHttpResponseCodec.java:122) ~[selenium-remote-driver-3.14.0.jar:na]
at org.openqa.selenium.remote.http.W3CHttpResponseCodec.decode(W3CHttpResponseCodec.java:49) ~[selenium-remote-driver-3.14.0.jar:na]
at org.openqa.selenium.remote.HttpCommandExecutor.execute(HttpCommandExecutor.java:158) ~[selenium-remote-driver-3.14.0.jar:na]
at org.openqa.selenium.remote.RemoteWebDriver.execute(RemoteWebDriver.java:548) ~[selenium-remote-driver-3.14.0.jar:na]
at org.openqa.selenium.remote.RemoteWebDriver.executeScript(RemoteWebDriver.java:485) ~[selenium-remote-driver-3.14.0.jar:na]
at ru.yandex.qatools.ashot.util.JsCoords.findCoordsWithJquery(JsCoords.java:30) ~[ashot-1.5.2.jar:na]
at ru.yandex.qatools.ashot.coordinates.JqueryCoordsProvider.ofElement(JqueryCoordsProvider.java:13) ~[ashot-1.5.2.jar:na]
at ru.yandex.qatools.ashot.coordinates.CoordsProvider.ofElements(CoordsProvider.java:21) ~[ashot-1.5.2.jar:na]
at ru.yandex.qatools.ashot.AShot.takeScreenshot(AShot.java:115) ~[ashot-1.5.2.jar:na]
at ru.yandex.qatools.ashot.AShot.takeScreenshot(AShot.java:132) ~[ashot-1.5.2.jar:na]
at com.optq.main.util.SeleniumDriverUtility.captureAShotElement(SeleniumDriverUtility.java:563) ~[classes/:na]
Any help or Alternatives For Ashot are welcome,Please help me into this.

This error message...
org.openqa.selenium.JavascriptException: javascript error: $ is not defined
(Session info: chrome=79.0.3945.130)
Build info: version: '3.14.0', revision: 'aacccce0', time: '2018-08-02T20:19:58.91Z'
System info: host: 'OPTIMIZEQ-LTP03', ip: '192.168.99.1', os.name: 'Windows 10', os.arch: 'amd64', os.version: '10.0', java.version: '11.0.2'
Driver info: org.openqa.selenium.remote.RemoteWebDriver
Capabilities {acceptInsecureCerts: false, browserName: chrome, browserVersion: 79.0.3945.130, chrome: {chromedriverVersion: 78.0.3904.105 (60e2d8774a81..., userDataDir: C:\Users\SHAILE~1.SIN\AppDa...}, goog:chromeOptions: {debuggerAddress: localhost:61006}, javascriptEnabled: true, networkConnectionEnabled: false, pageLoadStrategy: normal, platform: XP, platformName: XP, proxy: Proxy(manual, http=localhos..., setWindowRect: true, strictFileInteractability: false, timeouts: {implicit: 0, pageLoad: 300000, script: 30000}, unhandledPromptBehavior: dismiss and notify, webdriver.remote.sessionid: f5a47b27537f019dacb73462732...}
...implies that the ChromeDriver was unable to interact with the Browsing Context i.e. Chrome Browser session.
Deep dive
As per the documentation in ReferenceError: "x" is not defined this error means there is a non-existent variable referenced somewhere within the DOM Tree. This variable needs to be declared, or you need to make sure it is available in your current script or scope.
Hint: When loading a library (such as jQuery), make sure it is loaded before you access library variables, such as "$". Put the tag that loads the library before your code that uses it.
As per the discussion JavaScript/jQuery - “$ is not defined- $function()” error #Ketan mentions that this error occurs when you have not made jQuery available to your script, i.e. possibly the JavaScript / jQuery / AJAX haven't completed rendering the HTML DOM.
Solution
In these cases there are 3(three) different approaches available to solve the issue as follows:
The jQuery library is a single JavaScript file, and you reference it with the HTML <script> tag within the <head> section as follows:
<head>
<script src="jquery-3.4.1.min.js"></script>
</head>
This goes out and gets the jQuery code from the source.
Note: You do not have to include type="text/javascript" inside the <script> tag as this is not required in HTML5. JavaScript is the default scripting language in HTML5 and in all modern browsers.
You can also download the jQuery library and reference it locally on the server.
You can induce WebDriverWait inconjunction with ExpectedConditions for the element's desired state either to be present / visible / interactable.
You can find a detailed discussion in Selenium: How selenium identifies elements visible or not? Is is possible that it is loaded in DOM but not rendered on UI?
This usecase
As you mentioned, the following line fails:
Coords elementCoords = ofElement(driver, element);
This method takes one of the arguments (last) as element, but while defining you seem to be treating it as a list of elements, as in:
#SuppressWarnings("UnusedDeclaration")
public Set<Coords> ofElements(WebDriver driver, WebElement... elements) {
return ofElements(driver, Arrays.asList(elements));
}
Seems some mismatch in argument types here.
Additional considerations
You need to take care of a couple of things more:
You are using chromedriver=78.0.3904.105
Release Notes of chromedriver=78.0 clearly mentions the following :
Supports Chrome version 78
You are using chrome=79.0
Release Notes of ChromeDriver v79.0 clearly mentions the following :
Supports Chrome version 79
Your Selenium Client version is 3.14.0 of 2018-08-02T20:19:58.91Z which is almost 1.5 years older.
Your JDK version is 11.0.2.
So there is a clear mismatch between JDK v8u111 , Selenium Client v3.3.1 , ChromeDriver v2.41 and the Chrome Browser v79.0
Solution
Ensure that:
JDK is upgraded to current levels JDK 8u241.
Selenium is upgraded to current levels Version 3.141.59.
ChromeDriver is updated to current ChromeDriver v80.0 level.
Chrome is updated to current Chrome Version 80.0 level. (as per ChromeDriver v80.0 release notes)

I found a solution here, https://medium.com/virtualmind-io/jquery-injection-for-selenium-automation-tests-f6121ea57993. Basically it injects jquery into the page.
So I followed the steps and created this method that i used before calling AShot methods and could solve this issue, leaving the code here:
JavascriptExecutor js = (JavascriptExecutor) driver;
if(!(Boolean) js.executeScript("return (typeof jQuery != \"undefined\")")) {
js.executeScript(
"var headID = document.getElementsByTagName('head')[0];" +
"var newScript = document.createElement('script');" +
"newScript.type = 'text/javascript';" +
"newScript.src = 'https://ajax.googleapis.com/ajax/libs/jquery/3.3.1/jquery.min.js';" +
"headID.appendChild(newScript);");
WebDriverWait waitJQ = new WebDriverWait(driver, 30);
Function<WebDriver, Boolean> jQueryAvailable = WebDriver -> (
(Boolean) js.executeScript("return (typeof jQuery != \"undefined\")")
);
waitJQ.until(jQueryAvailable);
}

selenium fire StaleElementReferenceException

i try to make a web crawler with selenium.
My program fire a StaleElementReferenceException.
I thought that were because i crawl a page recursive and when a page have no more links the function navigate to next page and not previously to the parent page.
Therefore i have introduced a tree data structure to navigate back to the parent when the current url not equal the parent url. But this was not the solution for my problem.
Can anybody help me?
Code:
public class crawler {
private static FirefoxDriver driver;
private static String main_url = "https://robhammond.co/tools/seo-crawler";
private static List<String> uniqueLinks = new ArrayList<String>();
public static void main(String[] args) {
driver = new FirefoxDriver();
Node<String> root = new Node<>(main_url);
scrape(root, main_url);
}
public static void scrape(Node<String> node, String url) {
if(node.getParent() != null && (!driver.getCurrentUrl().equals(node.getParent().getData()))) {
driver.navigate().to(node.getParent().getData());
}
driver.navigate().to(url);
List<WebElement> allLinks = driver.findElements(By.tagName("a"));
for(WebElement link : allLinks) {
if(link.getAttribute("href").contains(main_url) && !uniqueLinks.contains(link.getAttribute("href")) && link.isDisplayed()) {
uniqueLinks.add(link.getAttribute("href"));
System.out.println(link.getAttribute("href"));
scrape(new Node<>(link.getAttribute("href")), link.getAttribute("href"));
}
}
}
}
And this is the output from the console:
D:\Programme\openjdk-12.0.1_windows-x64_bin\jdk-12.0.1\bin\java.exe "-javaagent:D:\Programme\JetBrains\IntelliJ IDEA 2019.1.2\lib\idea_rt.jar=60461:D:\Programme\JetBrains\IntelliJ IDEA 2019.1.2\bin" -Dfile.encoding=UTF-8 -classpath C:\Users\admin\Desktop\SeleniumWebScraper\out\production\SeleniumWebScraper;D:\Downloads\selenium-server-standalone-3.141.59.jar de.company.crawler.crawler
1557924446770 mozrunner::runner INFO Running command: "C:\\Program Files\\Mozilla Firefox\\firefox.exe" "-marionette" "-foreground" "-no-remote" "-profile" "C:\\Users\\admin\\AppData\\Local\\Temp\\rust_mozprofile.YqmEqE8y1pjv"
1557924447037 addons.webextension.screenshots#mozilla.org WARN Loading extension 'screenshots#mozilla.org': Reading manifest: Invalid extension permission: mozillaAddons
1557924447037 addons.webextension.screenshots#mozilla.org WARN Loading extension 'screenshots#mozilla.org': Reading manifest: Invalid extension permission: resource://pdf.js/
1557924447037 addons.webextension.screenshots#mozilla.org WARN Loading extension 'screenshots#mozilla.org': Reading manifest: Invalid extension permission: about:reader*
1557924448047 Marionette INFO Listening on port 60468
1557924448383 Marionette WARN TLS certificate errors will be ignored for this session
Mai 15, 2019 2:47:28 NACHM. org.openqa.selenium.remote.ProtocolHandshake createSession
INFO: Detected dialect: W3C
JavaScript warning: https://robhammond.co/js/jquery.min.js, line 4: Using //# to indicate sourceMappingURL pragmas is deprecated. Use //# instead
https://robhammond.co/tools/seo-crawler#content
https://twitter.com/intent/tweet?text=SEO%20Crawler&url=https://robhammond.co/tools/seo-crawler&via=robhammond
Exception in thread "main" org.openqa.selenium.StaleElementReferenceException: The element reference of <a href="/tools/"> is stale; either the element is no longer attached to the DOM, it is not in the current frame context, or the document has been refreshed
For documentation on this error, please visit: https://www.seleniumhq.org/exceptions/stale_element_reference.html
Build info: version: '3.141.59', revision: 'e82be7d358', time: '2018-11-14T08:25:53'
System info: host: 'DESKTOP-admin', ip: '192.168.233.1', os.name: 'Windows 10', os.arch: 'amd64', os.version: '10.0', java.version: '12.0.1'
Driver info: org.openqa.selenium.firefox.FirefoxDriver
Capabilities {acceptInsecureCerts: true, browserName: firefox, browserVersion: 66.0.5, javascriptEnabled: true, moz:accessibilityChecks: false, moz:geckodriverVersion: 0.24.0, moz:headless: false, moz:processID: 19124, moz:profile: C:\Users\admin\AppData\Loca..., moz:shutdownTimeout: 60000, moz:useNonSpecCompliantPointerOrigin: false, moz:webdriverClick: true, pageLoadStrategy: normal, platform: WINDOWS, platformName: WINDOWS, platformVersion: 10.0, rotatable: false, setWindowRect: true, strictFileInteractability: false, timeouts: {implicit: 0, pageLoad: 300000, script: 30000}, unhandledPromptBehavior: dismiss and notify}
Session ID: b3b87675-57c8-4b48-9a20-8df5e4d37503
at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:500)
at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:481)
at org.openqa.selenium.remote.http.W3CHttpResponseCodec.createException(W3CHttpResponseCodec.java:187)
at org.openqa.selenium.remote.http.W3CHttpResponseCodec.decode(W3CHttpResponseCodec.java:122)
at org.openqa.selenium.remote.http.W3CHttpResponseCodec.decode(W3CHttpResponseCodec.java:49)
at org.openqa.selenium.remote.HttpCommandExecutor.execute(HttpCommandExecutor.java:158)
at org.openqa.selenium.remote.service.DriverCommandExecutor.execute(DriverCommandExecutor.java:83)
at org.openqa.selenium.remote.RemoteWebDriver.execute(RemoteWebDriver.java:552)
at org.openqa.selenium.remote.RemoteWebElement.execute(RemoteWebElement.java:285)
at org.openqa.selenium.remote.RemoteWebElement.getAttribute(RemoteWebElement.java:134)
at de.company.crawler.crawler.scrape(crawler.java:33)
at de.company.crawler.crawler.scrape(crawler.java:38)
at de.company.crawler.crawler.main(crawler.java:20)
Process finished with exit code 1

When you navigate away from the first page all WebElements in the allLinks list get lost.
I would recommend converting it from the list of WebElement to the list of normal Strings like:
List<String> allLinksHrefs = allLinks.stream().map(link -> link.getAttribute("href")).collect(Collectors.toList());
and iterate through this new allLinksHrefs list instead.
You can use a hash-based collection for holding the uniqueLinks like HashSet - this way duplicates will be automatically eliminated
The current approach can take days to complete, consider using Selenium Grid and running your scraper in Parallel

pageLoadTimeout in Selenium not working

I am using below code using selenium in Java. I am adding pageLoadTimeout of 4 seconds, however, the driver continues to wait till the complete page is loaded. Any help?
System.setProperty("webdriver.gecko.driver", System.getProperty("user.home") + "\\Desktop\\geckodriver.exe");
FirefoxBinary b = new FirefoxBinary(new File(System.getProperty("user.home") + "\\desktop\\Mozilla Firefox\\firefox.exe"));
FirefoxOptions options = new FirefoxOptions().setBinary(b);
driver = new FirefoxDriver(options);
//driver.manage().timeouts().implicitlyWait(1, TimeUnit.SECONDS);
driver.manage().timeouts().pageLoadTimeout(4, TimeUnit.SECONDS);
driver.get("https://www.booking.com/hotel/in/the-taj-mahal-palace-tower.html?label=gen173nr-1FCAEoggJCAlhYSDNiBW5vcmVmaGyIAQGYATG4AQbIAQzYAQHoAQH4AQKSAgF5qAID;sid=338ad58d8e83c71e6aa78c67a2996616;dest_id=-2092174;dest_type=city;dist=0;group_adults=2;hip_dst=1;hpos=1;room1=A%2CA;sb_price_type=total;srfid=ccd41231d2f37b82d695970f081412152a59586aX1;srpvid=c71751e539ea01ce;type=total;ucfs=1&#hotelTmpl");
List<WebElement> facilitySectionList = driver.findElements(By.className("facilitiesChecklistSection"));

The solution to your pageLoadTimeout issue would be to bump up your Selenium version to v3.5.0. Here is effective code block and the resulted org.openqa.selenium.TimeoutException: Timeout loading page after 2000ms of your own code in minimal lines:
Code block:
public class Q45591282_pageloadtimeout
{
public static void main(String[] args)
{
System.setProperty("webdriver.gecko.driver", "C:\\Utility\\BrowserDrivers\\geckodriver.exe");
WebDriver driver=new FirefoxDriver();
driver.manage().timeouts().pageLoadTimeout(2, TimeUnit.SECONDS);
driver.get("https://www.booking.com/hotel/in/the-taj-mahal-palace-tower.html?label=gen173nr-1FCAEoggJCAlhYSDNiBW5vcmVmaGyIAQGYATG4AQbIAQzYAQHoAQH4AQKSAgF5qAID;sid=338ad58d8e83c71e6aa78c67a2996616;dest_id=-2092174;dest_type=city;dist=0;group_adults=2;hip_dst=1;hpos=1;room1=A%2CA;sb_price_type=total;srfid=ccd41231d2f37b82d695970f081412152a59586aX1;srpvid=c71751e539ea01ce;type=total;ucfs=1&#hotelTmpl");
}
}
Console Output:
1502530864350 geckodriver INFO geckodriver 0.18.0
1502530864365 geckodriver INFO Listening on 127.0.0.1:29688
1502530865042 geckodriver::marionette INFO Starting browser C:\Program Files\Mozilla Firefox\firefox.exe with args ["-marionette"]
1502530903170 Marionette INFO Listening on port 1900
Aug 12, 2017 3:11:44 PM org.openqa.selenium.remote.ProtocolHandshake createSession
INFO: Detected dialect: W3C
Exception in thread "main" org.openqa.selenium.TimeoutException: Timeout loading page after 2000ms
Build info: version: '3.5.0', revision: '8def36e068', time: '2017-08-10T23:00:22.093Z'
System info: host: 'ATECHM-03', ip: '192.168.1.48', os.name: 'Windows 8', os.arch: 'amd64', os.version: '6.2', java.version: '1.8.0_77'
Driver info: org.openqa.selenium.firefox.FirefoxDriver
Capabilities [{moz:profile=C:\Users\ATECHM~1\AppData\Local\Temp\rust_mozprofile.LSsvaNqlDbxE, rotatable=false, timeouts={implicit=0.0, pageLoad=300000.0, script=30000.0}, pageLoadStrategy=normal, platform=ANY, specificationLevel=0.0, moz:accessibilityChecks=false, acceptInsecureCerts=false, browserVersion=53.0, platformVersion=6.2, moz:processID=3652.0, browserName=firefox, javascriptEnabled=true, platformName=windows_nt}]
Session ID: 8b841376-00fd-4359-8cae-a68912b23706
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at org.openqa.selenium.remote.http.W3CHttpResponseCodec.createException(W3CHttpResponseCodec.java:185)
at org.openqa.selenium.remote.http.W3CHttpResponseCodec.decode(W3CHttpResponseCodec.java:120)
at org.openqa.selenium.remote.http.W3CHttpResponseCodec.decode(W3CHttpResponseCodec.java:49)
at org.openqa.selenium.remote.HttpCommandExecutor.execute(HttpCommandExecutor.java:164)
at org.openqa.selenium.remote.service.DriverCommandExecutor.execute(DriverCommandExecutor.java:82)
at org.openqa.selenium.remote.RemoteWebDriver.execute(RemoteWebDriver.java:641)
at org.openqa.selenium.remote.RemoteWebDriver.get(RemoteWebDriver.java:368)
at demo.Q45591282_pageloadtimeout.main(Q45591282_pageloadtimeout.java:20)
Catching the WebDriverException
Code Block:
public class pageLoadTimeout
{
public static void main(String[] args)
{
System.setProperty("webdriver.chrome.driver", "C:\\Utility\\BrowserDrivers\\chromedriver.exe");
WebDriver driver = new ChromeDriver();
driver.manage().timeouts().pageLoadTimeout(2, TimeUnit.SECONDS);
try{
driver.get("https://www.booking.com/hotel/in/the-taj-mahal-palace-tower.html?label=gen173nr-1FCAEoggJCAlhYSDNiBW5vcmVmaGyIAQGYATG4AQbIAQzYAQHoAQH4AQKSAgF5qAID;sid=338ad58d8e83c71e6aa78c67a2996616;dest_id=-2092174;dest_type=city;dist=0;group_adults=2;hip_dst=1;hpos=1;room1=A%2CA;sb_price_type=total;srfid=ccd41231d2f37b82d695970f081412152a59586aX1;srpvid=c71751e539ea01ce;type=total;ucfs=1&#hotelTmpl");
}catch(WebDriverException e){
System.out.println("WebDriverException occured");
}
driver.quit();
}
}
Console Output:
Only local connections are allowed.
Please protect ports used by ChromeDriver and related test frameworks to prevent access by malicious code.
Jul 17, 2019 8:53:26 PM org.openqa.selenium.remote.ProtocolHandshake createSession
INFO: Detected dialect: W3C
[1563377008.449][SEVERE]: Timed out receiving message from renderer: 1.999
[1563377008.450][SEVERE]: Timed out receiving message from renderer: -0.001
[1563377008.461][SEVERE]: Timed out receiving message from renderer: -0.012
[1563377010.466][SEVERE]: Timed out receiving message from renderer: 1.998
[1563377010.467][SEVERE]: Timed out receiving message from renderer: -0.001
[1563377010.476][SEVERE]: Timed out receiving message from renderer: -0.010
WebDriverException occured

if you specify the time, and selenium is not able to load the site is specified time, it will throw TimedoutException. You should handle that in your code.
driver.manage().timeouts().pageLoadTimeout(4, TimeUnit.SECONDS);
long startTime = System.currentTimeMillis();
try{
driver.get("https://yahoo.com");
}catch (TimeoutException te){
long estimatedTime = System.currentTimeMillis() - startTime;
System.out.println("it took "+estimatedTime+" Time");
}
if it's not throwing TimeoutException, either website has done loading within specified time or something is wrong with Selenium API.