We Keep Coding

Selenium in production Spring boot

I'm writing a web page in Spring Boot and using Selenium to get data from another website based on user's action, which means Selenium will get data on the fly. The website will require login to get the data. So the website will first need to initiate the webDriver and log in (I'm using user-data attribute but sometimes the login will expire so need to check every time). I'm trying to find the best approach for using Selenium to get data on the fly. From what I have experienced, if I use something like this to initialize Selenium driver:
try {
webDriver.get(url);
WebDriverWait wait = new WebDriverWait(webDriver, Duration.ofSeconds(TIME_OUT));
wait.until(ExpectedConditions.presenceOfElementLocated(By.cssSelector("button")));
logger.info("Already logged in");
} catch (TimeoutException e) {
logger.error("Not logged in yet");
login(webDriver);
}
The "button" here will only be present if it is logged in. I added a log above and after this line and found out it will take about 8 seconds to finish this part, which is too long. I initially added this when user triggers the action, but this will obviously be too long.
My thought to overcome this is: for each connection, once user hits the website main page (does not need Selenium at main page) then start creating a new instance of webDriver and the login process and hold the webDriver for each of the connections. But how should I approach in such a way? Should I use Spring boot HttpSession and do something like:
session.setAttribute("driver", webDriver);
And then later if user actually makes the request that needs Selenium, then it can retrieve the webDriver using
session.getAttribute("driver");
Is this something that is reliable to use?
Or could anyone please enlighten me on how should I optimize this?
Thanks a lot!

Avoid the execution stop while browsing in Selenium WebDriver

I need help for this thing that's driving me crazy.
I want to check the browser url in and endless loop, waiting a little (Thread.Sleep) between a loop and another, to not overload the CPU. Then, if the browser url is what I need, I want to add/change/remove an element through Javascript before the page is fully loaded, otherwise the person who uses this could see the change. (I don't need help for the javascript part)
But there's a problem: it seems that in Selenium Webdriver when I navigate to a page (with .get(), .navigate().to() or also directly from the client) the execution is forced to stop until the page is loaded.
I tried to set a "fake" timeout, but (at least in Chrome) when it catches the TimeoutException, the page stops loading. I know that in Firefox there's an option for unstable loading, but I don't want to use it because my program isn't only for Firefox.
public static void main(String[] args) throws InterruptedException {
System.setProperty("webdriver.chrome.driver", "chromedriver.exe");
WebDriver driver = new ChromeDriver();
driver.manage().timeouts().pageLoadTimeout(0, TimeUnit.MILLISECONDS); // Fake timeout
while (true) {
try {
// If the url (driver.getCurrentUrl()) is what I want, then execute javascript without needing that page is fully loaded
// ...
// ...
}
catch (TimeoutException e) {
// It ignores the Exception, but unfortunately the page stops loading.
}
Thread.sleep(500); // Then wait some time to not overload the cpu
}
}
I need to do this in Chrome, and if possible with Firefox and Internet Explorer. I'm programming in Java. Thanks in advance.

Selenium is designed to stop once the webpage is loaded into the browser so that it can proceed with execution.
In your case there are two options:
1) If the browser url will change automatically (ajax) at an arbitrary time, then just keep getting browser url until your condition satisfies.
while(currentURL.equals("Your Condition")){
currentURL = driver.getCurrentUrl();
Thread.sleep(2000);
}
2) If the browser needs to be refreshed use the refresh method in a loop until you get your desired url
while(currentURL.equals("Your Condition")){
driver.navigate().refresh();
currentURL =
Thread.sleep(2000);
}

As know, if user tried with driver.get("url");, selenium waits until page is loaded (may not be very long). so if you want to do some thing on navigate to URL without waiting total load time use below code instead of get or navigate
JavascriptExecutor js=(JavascriptExecutor)driver;
js.executeScript("window.open('http://seleniumtrainer.com/components/buttons/','_self');");
After this used
driver.findElement(By.id("button1")).click();
to click on button but i am getting no such element exception, so i am expecting its not waiting for page load. so times page loading very quick and click working fine.
i hope this will help you to figure it out your issue at start up. for loop i hope solution already provided.
Thanks

PhantomJS slower than ChromeDriver, using Selenium

I'm trying to use PhantomJS 2.0/GhostDriver instead the ChromeDriver, since I have read I could speed up my UI tests.
This is the test code I'm running, as part of a Junit test:
#Override
public void runTestCase() throws Exception {
long startTime = System.currentTimeMillis();
// log in as admin
Login.loginAs("admin", "password");
System.out.println(System.currentTimeMillis() - startTime);
}
The loginAs function fills in the text fields for the username and password, then click on the submit button and finally moves in the home section of the new returned page.
Now, I'm running once a time this simple test using both Phantomjs and ChromeDriver as driver for Selenium in Java (v2.45).
They are initialized as follow:
ChromeDriver
System.setProperty("webdriver.chrome.logfile", workingDirectory + "\\chromedriver.log");
service = new ChromeDriverService.Builder().usingDriverExecutable(new File(workingDirectory + "\\chromedriver.exe")).build();
capabilities = DesiredCapabilities.chrome();
options = new ChromeOptions();
options.addArguments("--allow-file-access-from-files");
options.addArguments("--verbose");
capabilities.setVersion("");
capabilities.setCapability(ChromeOptions.CAPABILITY, options);
driver = new ChromeDriver(service, capabilities);
PhantomJS
System.setProperty("phantomjs.binary.path", workingDirectory + "\\phantomjs.exe");
cliArgsCap = new ArrayList<String>();
capabilities = DesiredCapabilities.phantomjs();
cliArgsCap.add("--web-security=false");
cliArgsCap.add("--ssl-protocol=any");
cliArgsCap.add("--ignore-ssl-errors=true");
cliArgsCap.add("--webdriver-loglevel=INFO");
cliArgsCap.add("--load-images=false");
capabilities.setCapability(CapabilityType.SUPPORTS_FINDING_BY_CSS, true);
capabilities.setCapability(CapabilityType.TAKES_SCREENSHOT, true);
capabilities.setCapability(PhantomJSDriverService.PHANTOMJS_CLI_ARGS, cliArgsCap);
driver = new PhantomJSDriver(capabilities);
I'm running my test on a 64bit Windows 7 machine. So, having a look the time took by the test, I always note that ChromeDriver is faster than PhantomJS. Always. For instance, if the test with ChromeDriver takes about 3-4 seconds, the same with PhantomJS takes about 5-6 seconds.
Has anyone experienced with this issue? Or could anyone give me any reason for that? Am I setting something wrong?
Furthermore, if you need more details, let me know.

I have found that this setting uses a lot of memory that seems to keep growing:
cliArgsCap.add("--load-images=false");
But when I use this setting the memory usage is stable:
cliArgsCap.add("--load-images=true");

"PhantomJS is a headless WebKit scriptable with a JavaScript API" as it's explained on the main page of the project.
Google split from WebKit to create Blink to use it in Chrome.
What are the main differences between them - unfortunately I'm not the expert here.
I run one of my really long scenarios both on Chrome and PhantomJS and to my surprise the difference was pretty significant:
PhantomJS - 583.251 s
Chrome - 448.384 s
Using PhantomJS doesn't bring performance benefits in my case but running tests headless does. I can use machine without graphical desktop and save computing power for some additional threads.

The slowest aspect of a web page is the downloading of html, JavaScript, css, images, etc and making AJAX request.
To anybody who says Headless is faster, how can headless address any of these?

Appium's implicitlyWait does not work

I am using Appium to automate an iOS app but met a problem, is there anyone meet the same problem before?
Appium's implicitlyWait API seems to not work. I am using Java and JUnit to run the test, here is the line of code:
driver.manage().timeouts().implicitlyWait(50, TimeUnit.SECONDS);
I have checked the debug info in the appium console, it looks correctly:
info: [debug] Set iOS implicit wait to 50000ms
My Environment:
Latest Appium 1.2.1, Java client library 1.6.1, Selenium Java language binding 2.42.2 and sample app 'UICatalog'provided by Sauce Lab.
Thanks in advance for the reply.

The code you have posted manages the timeout to wait for a maximum of 50 seconds. It doesn't make the driver wait 50 seconds. You can use the wait like:
driver.wait(); //this will wait a max of 50 seconds cuz you said so
If you ask me the proper way you would want to use waiting on Webdriver is:
WebDriverWait wait;
wait = new WebDriverWait(driver, 60);
wait.until(ExpectedConditions.elementToBeClickable(By.id("blabla"));
The code above checks if blabla is clickable until that condition is proved or 60 seconds(stated above) passes the driver waits.

In Appium it is possible to set implicit way in this way:
Java code:
AppiumFieldDecorator decorator = new AppiumFieldDecorator(driver);
decorator.resetImplicitlyWaitTimeOut(50, TimeUnit.SECONDS);
PageFactory.initElements(decorator, this /* refers to current page object class*/);
Such timeout will work for the whole time.
It is not possible (at least I don't know) to change it.
As when web drivers are used you can do this with:
driver.manage().timeouts().implicitlyWait(0, TimeUnit.SECONDS);
// some actions for which you don't want to wait implicitly
driver.manage().timeouts().implicitlyWait(50, TimeUnit.SECONDS);

Try this:
public static void WaitForElementPresent1(String locator, int timeout)
{
WebDriverWait wait = new WebDriverWait(driver, timeout);
try{
wait.until(ExpectedConditions.visibilityOfElementLocated(By.xpath(locator)));
} catch (Exception e){
e.printStackTrace();
}
}

If you are using PageFactory model, you can specify the implicit wait with initElements() method as given below -
PageFactory.initElements(new AppiumFieldDecorator(driver, 10, TimeUnit.SECONDS), this);
I have tried this with Appium 1.6 and it works fine.

The new way of setting the implicit time out is using the code
AppiumFieldDecorator decorator = new AppiumFieldDecorator(mobDriver);
decorator.DEFAULT_IMPLICITLY_WAIT_TIMEOUT = longValue;
decorator.DEFAULT_TIMEUNIT = TimeUnit.TimeUnit ;
Hope this helps

How do I set the selenium webdriver get timeout?

When I am using a proxy in webdriver like FirefoxDriver, if the proxy is bad then the get method will block forever. I set some timeout parameters, but this did not work out.
This is my code:
FirefoxProfile profile = new FirefoxProfile();
profile.setPreference("general.useragent.override", ua);
Proxy p = new Proxy();
p.setHttpProxy(proxy);
profile.setProxyPreferences(p);
profile.setEnableNativeEvents(true);
// create a driver
WebDriver driver = new FirefoxDriver(profile);
driver.manage().timeouts().pageLoadTimeout(30, TimeUnit.SECONDS);
driver.manage().timeouts().setScriptTimeout(30, TimeUnit.SECONDS);
driver.get("www.sina.com.cn")
The call to driver.get will block for ever, but I want it to wait for 30 seconds and if the page is not loaded then throw an exception.

Try this:
driver.manage().timeouts().implicitlyWait(30, TimeUnit.SECONDS);

The timeouts() methods are not implemented in some drivers and are very unreliable in general.
I use a separate thread for the timeouts (passing the url to access as the thread name):
Thread t = new Thread(new Runnable() {
public void run() {
driver.get(Thread.currentThread().getName());
}
}, url);
t.start();
try {
t.join(YOUR_TIMEOUT_HERE_IN_MS);
} catch (InterruptedException e) { // ignore
}
if (t.isAlive()) { // Thread still alive, we need to abort
logger.warning("Timeout on loading page " + url);
t.interrupt();
}
This seems to work most of the time, however it might happen that the driver is really stuck and any subsequent call to driver will be blocked (I experience that with Chrome driver on Windows). Even something as innocuous as a driver.findElements() call could end up being blocked. Unfortunately I have no solutions for blocked drivers.

try
driver.executeScript("window.location.href='http://www.sina.com.cn'")
If you have set pageLoadStrategy to none, this statement will return immediately.
And after that , you can add a WebDriverWait with timeout to check if the page title or any element is ok.
Hope this will help you.

I had the same problem and thanks to this forum and some other found the answer.
Initially I also thought of separate thread but it complicates the code a bit. So I tried to find an answer that aligns with my principle "elegance and simplicity".
Please have a look at such forum:
https://sqa.stackexchange.com/questions/2606/what-is-seleniums-default-timeout-for-page-loading
#
SOLUTION:
In the code, before the line with 'get' method you can use for example:
driver.manage().timeouts().pageLoadTimeout(10, TimeUnit.SECONDS);
#
One thing is that it throws timeoutException so you have to encapsulate it in the try catch block or wrap in some method.
I haven't found the getter for the pageLoadTimeout so I don't know what is the default value, but probably very high since my script was frozen for many hours and nothing moved forward.
#
NOTICE:
'pageLoadTimeout' is NOT implemented for Chrome driver and thus causes exception. I saw by users comments that there are plans to make it.

You can set the timeout on the HTTP Client like this
int connectionTimeout=5000;
int socketTimeout=15000;
ApacheHttpClient.Factory clientFactory = new ApacheHttpClient.Factory(new HttpClientFactory(connectionTimeout, socketTimeout));
HttpCommandExecutor executor =
new HttpCommandExecutor(new HashMap<String, CommandInfo>(), new URL(seleniumServerUrl), clientFactory);
RemoteWebDriver driver = new RemoteWebDriver(executor, capabilities);

The solution of driver.manage().timeouts().pageLoadTimeout(10, TimeUnit.SECONDS) will work on the pages with synch loading. This does not solve, however, the problem on pages loading stuff in async, then the tests will fail all the time if we set the pageLoadTimeOut.

I find that the timeout calls are not reliable enough in real life, particularly for internet explorer , however the following solutions may be of help:
You can timeout the complete test by using #Test(timeout=10000) in the junit test that you are running the selenium process from. This will free up the main thread for executing the other tests, instead of blocking up the whole show. However even this does not work at times.
Anyway by timing out you do not intend to salvage the test case, because timing out even a single operation might leave the entire test sequence in inconsistent state. You might just want to proceed with the other testcases without blocking (or perhaps retry the same test again). In such a case a fool-proof method would be to write a poller that polls processes of Webdriver (eg. IEDriverServer.exe, Phantomjs.exe) running for more than say 10 min and kill them. An example could be found at Automatically identify (and kill) processes with long processing time

Used below code in similar situation
driver.manage().timeouts().pageLoadTimeout(60, TimeUnit.SECONDS);
and embedded driver.get code in a try catch, which solved the issue of loading pages which were taking more than 1 minute.