Selenium in production Spring boot - java

I'm writing a web page in Spring Boot and using Selenium to get data from another website based on user's action, which means Selenium will get data on the fly. The website will require login to get the data. So the website will first need to initiate the webDriver and log in (I'm using user-data attribute but sometimes the login will expire so need to check every time). I'm trying to find the best approach for using Selenium to get data on the fly. From what I have experienced, if I use something like this to initialize Selenium driver:
try {
webDriver.get(url);
WebDriverWait wait = new WebDriverWait(webDriver, Duration.ofSeconds(TIME_OUT));
wait.until(ExpectedConditions.presenceOfElementLocated(By.cssSelector("button")));
logger.info("Already logged in");
} catch (TimeoutException e) {
logger.error("Not logged in yet");
login(webDriver);
}
The "button" here will only be present if it is logged in. I added a log above and after this line and found out it will take about 8 seconds to finish this part, which is too long. I initially added this when user triggers the action, but this will obviously be too long.
My thought to overcome this is: for each connection, once user hits the website main page (does not need Selenium at main page) then start creating a new instance of webDriver and the login process and hold the webDriver for each of the connections. But how should I approach in such a way? Should I use Spring boot HttpSession and do something like:
session.setAttribute("driver", webDriver);
And then later if user actually makes the request that needs Selenium, then it can retrieve the webDriver using
session.getAttribute("driver");
Is this something that is reliable to use?
Or could anyone please enlighten me on how should I optimize this?
Thanks a lot!

Related

Wicket: double login required due to extended browser info

I have a Wicket 8.6 application. Currently, when logging in to the application, mostly (does not always happen) the user has to login twice. After the first login (after entering the credentials and clicking the submit button) a white page appears saying "If you see this, it means that both javascript and meta-refresh are not support by your browser configuration. Please click this link to continue to the original destination." This is the BrowserInfoPage. After a few seconds the user is redirected to the login page again where he/she has to enter his/her credentials again and press the login button. This time, the user logs in successfully. My question is, how do I prevent that the user hast to enter his/her credentials twice.
From my research I know that it has something to do with the collection of extended browser info. In the init method of my WicketApplication class, I had the following code:
getRequestCycleSettings().setGatherExtendedBrowserInfo(true);
However, I already commented out this code several month ago. For some reason, the described effect occurs for every new deploy now. Maybe a newly added package in the application is the reason for it. I don't know. Is there a possibility to prevent this second login maybe by creating a customized bowser info page which forwards the login? Please point me in the right direction. Thanks.
After some research, I came up with a work around. It is probably not very efficient but it works for me so far. In my custom Session class which inherits from AuthenticatedWebSession, I added the following code.
#Override
protected WebPage newBrowserInfoPage() {
final Request request = RequestCycle.get().getRequest();
if(request.getUrl().toString().contains("LoginPage")) {
if(!isSignedIn()) {
signIn(username, password);
}
PageParametersEncoder encoder = new PageParametersEncoder();
PageParameters parameters = encoder.decodePageParameters(request.getUrl());
String url = parameters != null && parameters.get("originUrl") !=null && !parameters.get("originUrl").isNull() && !parameters.get("originUrl").isEmpty()?
parameters.get("originUrl").toString("pages/home"):"pages/home";
String finalUrl=url.startsWith("pages/")?url.substring("pages/".length()):url;
throw new RedirectToUrlException(finalUrl);
}
return super.newBrowserInfoPage();
}
Some explanation to the code. As mentioned in the question, I want to prevent the user from logging in multiple times. Thus, I check if the request comes from the LoginPage and perform my work around only in that case.
During my implementation, I realized, that the method newBrowserInfoPage is called in the process when I call session.signIn(username,password); on my LoginPage. In this signIn process the authenticate method of my custom Session is called but the signedIn flag in the AuthenticatedWebSession is not changed (keeps false on successfull authentication). Is this a bug? Thus, I have to login again to set the flag to true.
Finally, I read the URL of the LoginPage where I have stored the target URL and forward the user to the target URL.
I am aware this is probably not the best approach but it is the only solution I came up with. If someone has a better idea, I am happy to hear it.

How can we use pre-populate cookies in Selenium Web driver to run the tests faster

How do we pre-populate cookies in Selenium Web driver to make the selenium tests run faster ?
If we need to write test for multiple pages how can we have a single reusable method to pre-populate the cookies and how can we use inside tests ?
public void addCookie(String name, String value) {
Cookie pagecookieSize = new Cookie.Builder(name, value)
.domain("somedomain.com.au")
.expiresOn(new Date(2020,12,31))
.isSecure(true)
.path("/")
.build();
driver.manage().addCookie(pagecookieSize);
}
pre-populate cookies means? cookie before starting the selenium or execution. As we know selenium/webdriver starts always new session and cant use old cookies and also we cant save cookies which we see in execution once execution is done.
preferably, use profile concept, create browser profile.. do required navigation's.. so there is cache/cookie/temp what ever may be and call that profile.
here is one of my old post to start chrome with profile
How do you start selenium using Chrome driver and all existing browser cookies?

Java program for detecting server login prompt?

I'm writing a program in Java to automate a web test of server management console URL's to find which ones present users with a box for login credentials.
I have begun looking into doing this with Selenium WebDriver to automate the actual test function of opening and closing windows, but what I can't figure out is how I should go about detecting whether or not the login box is presented to a user.
Is there a specific response code that the server presents when this happens? I know when a user is unauthenticated/forbidden there will be an HTTP response of 401 or 403, respectively, but I am not sure if this 401 unauthenticated response Code will happen regardless of whether or not a box for login credentials is presented to the user.
The goal is to find which URL's allowed a user the opportunity to type in credentials and then publish those to a .txt file for further evaluation. I already have the java I/O elements working properly to do this, so all I really need is to figure out what condition I'm looking for. Any help would be appreciated!
Given you have an instantiated WebDriver instance (driver)
and a collection of URLs (urls) you want to check,
and given your loginbox contains some element located by ID "username"
then one possible Selenium solution in Java could look like this:
for (String url : urls) {
driver.get(url);
//adjust to your needs: long timeOutInSeconds, long sleepInMillis
WebDriverWait wait = new WebDriverWait(driver, 120L, 1000L);
try {
wait.until(ExpectedConditions.presenceOfElementLocated(By.id("username")));
//TODO: Element found within timeOutInSeconds
} catch (WebDriverException e) {
//TODO: Element not present after timeOutInSeconds, write url to txt file
}
}
If you cannot locate the element by id, use other locating mechanisms as described here.

Selenium RemoteWebDriver - Do something if element could not be found

I am trying to develop a Suite of classes for testing my websites functionality every night and I do this in Chrome, Firefox, Edge and IE. Because sometimes Selenium doesn't find an element I need something that e.g. takes a screenshot of the browser before giving out an error. I don't need a function for taking a screenshot I need something that triggers when Selenium can't continue.
Best regards,
MK
If I understand correct, you need to set up the trigger for your another system, which can react on Selenium test error.
In your test code you can use :
try {
// find element and test code
} catch (NoSuchElementException e) {
// set up the trigger code
}
To notify another system you can choose any system, which can provide notification mechanism.
In your case, you could use for example Redis with pub/sub.
So your reaction system will be a subscriber and test - provider of the event.

Difference between webdriver.get() and webdriver.navigate()

What is the difference between get() and navigate() methods?
Does any of this or maybe another method waits for page content to load?
What do I really need is something like Selenium 1.0's WaitForPageToLoad but for using via webdriver.
Any suggestions?
Navigating
The first thing you’ll want to do with WebDriver is navigate to a page. The normal way to do this is by calling get:
driver.get("http://www.google.com");
WebDriver will wait until the page has fully loaded (that is, the onload event has fired) before returning control to your test or script. It’s worth noting that if your page uses a lot of AJAX on load then WebDriver may not know when it has completely loaded. If you need to ensure such pages are fully loaded then you can use waits.
Navigation: History and Location
Earlier, we covered navigating to a page using the get command (driver.get("http://www.example.com")) As you’ve seen, WebDriver has a number of smaller, task-focused interfaces, and navigation is a useful task. Because loading a page is such a fundamental requirement, the method to do this lives on the main WebDriver interface, but it’s simply a synonym to:
driver.navigate().to("http://www.example.com");
To reiterate: navigate().to() and get() do exactly the same thing. One's just a lot easier to type than the other!
The navigate interface also exposes the ability to move backwards and forwards in your browser’s history:
driver.navigate().forward();
driver.navigate().back();
(Emphasis added)
They both seems to navigate to the given webpage and quoting #matt answer:
navigate().to() and get() do exactly the same thing.
Single-Page Applications are an exception to this.
The difference between these two methods comes not from their behavior, but from the behavior in the way the application works and how browser deal with it.
navigate().to() navigates to the page by changing the URL like doing forward/backward navigation.
Whereas, get() refreshes the page to changing the URL.
So, in cases where application domain changes, both the method behaves similarly. That is, page is refreshed in both the cases. But, in single-page applications, while navigate().to() do not refreshes the page, get() do.
Moreover, this is the reason browser history is getting lost when get() is used due to application being refreshed.
Originally answered: https://stackoverflow.com/a/33868976/3619412
driver.get() : It's used to go to the particular website , But it doesn't maintain the browser History and cookies so , we can't use forward and backward button , if we click on that , page will not get schedule
driver.navigate() : it's used to go to the particular website , but it maintains the browser history and cookies, so we can use forward and backward button to navigate between the pages during the coding of Testcase
Not sure it applies here also but in the case of protractor when using navigate().to(...) the history is being kept but when using get() it is lost.
One of my test was failing because I was using get() 2 times in a row and then doing a navigate().back(). Because the history was lost, when going back it went to the about page and an error was thrown:
Error: Error while waiting for Protractor to sync with the page: {}
driver.get() is used to navigate particular URL(website) and wait till page load.
driver.navigate() is used to navigate to particular URL and does not wait to page load. It maintains browser history or cookies to navigate back or forward.
As per the javadoc for get(), it is the synonym for Navigate.to()
View javadoc screenshot below:
Javadoc for get() says it all -
Load a new web page in the current browser window. This is done using
an HTTP GET operation, and the method will block until the load is
complete. This will follow redirects issued either by the server or as
a meta-redirect from within the returned HTML. Should a meta-redirect
"rest" for any duration of time, it is best to wait until this timeout
is over, since should the underlying page change whilst your test is
executing the results of future calls against this interface will be
against the freshly loaded page. Synonym for
org.openqa.selenium.WebDriver.Navigation.to(String).
navigate().to() and get() will work same when you use for the first time. When you use it more than once then using navigate().to() you can come to the previous page at any time whereas you can do the same using get().
Conclusion: navigate().to() holds the entire history of the current window and get() just reload the page and hold any history.
For what it's worth, from my IE9 testing, it looks like there's a difference for URLs that contain a hashbang (a single page app, in my case):
http://www.example.com#page
The driver.get("http://www.example.com#anotherpage") method is handled by the browser as a fragment identifier and JavaScript variables are retained from the previous URL.
While, the navigate().to("http://www.example.com#anotherpage") method is handled by the browser as a address/location/URL bar input and JavaScript variables are not retained from the previous URL.
There are some differences between webdriver.get() and webdriver.navigate() method.
get()
As per the API Docs get() method in the WebDriver interface extends the SearchContext and is defined as:
/**
* Load a new web page in the current browser window. This is done using an HTTP POST operation,
* and the method will block until the load is complete.
* This will follow redirects issued either by the server or as a meta-redirect from within the
* returned HTML.
* Synonym for {#link org.openqa.selenium.WebDriver.Navigation#to(String)}.
*/
void get(String url);
Usage:
driver.get("https://www.google.com/");
navigate()
On the other hand, navigate() is the abstraction which allows the WebDriver instance i.e. the driver to access the browser's history as well as to navigate to a given URL. The methods along with the usage are as follows:
to(java.lang.String url): Load a new web page in the current browser window.
driver.navigate().to("https://www.google.com/");
to(java.net.URL url): Overloaded version of to(String) that makes it easy to pass in a URL.
refresh(): Refresh the current page.
driver.navigate().refresh();
back(): Move back a single "item" in the browser's history.
driver.navigate().back();
forward(): Move a single "item" forward in the browser's history.
driver.navigate().forward();
driver.get("url") and driver.navigate( ).to("url") both are same/synonymous.
to("url") internally calling get("url") method. Please find the below image for reference.
Either of them does not store history - this is the wrong information that is available on most of the blogs/websites.
Below, statements 1, 2, and 3, 4 will do the same things i.e land in the given URL.
statemnt 1: driver.get("http://www.google.com");
statemnt 2: driver.navigate( ).to("http://www.amazon.in");
statemnt 3: driver.get("http://www.google.com");
statemnt 4: driver.get("http://www.amazon.in");
Only navigate() can do different things i.e. moving back, forward, etc. But not the to("url") method.
Otherwise you prob want the get method:
Load a new web page in the current browser window. This is done using an
HTTP GET operation, and the method will block until the load is complete.
Navigate allows you to work with browser history as far as i understand it.
Both perform the same function but driver.get(); seems more popular.
driver.navigate().to(); is best used when you are already in the middle of a script and you want to redirect from current URL to a new one. For the sake of differentiating your codes, you can use driver.get();to launch the first URL after opening a browser instance, albeit both will work either way.
CASE-1
In the below code I navigated to 3 different URLs and when the execution comes to navigate command, it navigated back to facebook home page.
public class FirefoxInvoke {
#Test
public static void browserInvoke()
{
System.setProperty("webdriver.gecko.driver", "gecko-driver-path");
WebDriver driver=new FirefoxDriver();
System.out.println("Before"+driver.getTitle());
driver.get("http://www.google.com");
driver.get("http://www.facebook.com");
driver.get("http://www.india.com");
driver.navigate().back();
driver.quit();
}
public static void main(String[] args) {
// TODO Auto-generated method stub
browserInvoke();
}
}
CASE-2:
In below code, I have used navigate() instead of get(), but both the snippets(Case-1 and Case-2) are working exactly the same, just the case-2 execution time is less than of case-1
public class FirefoxInvoke {
#Test
public static void browserInvoke()
{
System.setProperty("webdriver.gecko.driver", "gecko-driver-path");
WebDriver driver=new FirefoxDriver();
System.out.println("Before"+driver.getTitle());
driver.navigate().to("http://www.google.com");
driver.navigate().to("http://www.facebook.com");
driver.navigate().to("http://www.india.com");
driver.navigate().back();
driver.quit();
}
public static void main(String[] args) {
// TODO Auto-generated method stub
browserInvoke();
}
}
So the main difference between get() and navigate() is, both are
performing the same task but with the use of navigate() you can move
back() or forward() in your session's history.
navigate() is faster than get() because navigate() does not wait for
the page to load fully or completely.
driver.get(url) and navigate.to(url) both are used to go to particular web page. The key difference is that
driver.get(url): It does not maintain the browser history and cookies and wait till page fully loaded.
driver.navigate.to(url):It is also used to go to particular web page.it maintain browser history and cookies and does not wait till page fully loaded and have navigation between the pages back, forward and refresh.
To get a better understanding on it, one must see the architecture of Selenium WebDriver.
Just visit https://github.com/SeleniumHQ/selenium/wiki/JsonWireProtocol
and search for "Navigate to a new URL." text. You will see both methods GET and POST.
Hence the conclusion given below:
driver.get() method internally sends Get request to Selenium Server Standalone. Whereas driver.navigate() method sends Post request to Selenium Server Standalone.

Categories