The question:
Is is possible to tell a browser that is controlled by selenium webdriver to not load any content from external sources, or alternatively, not load resources from a given list of domains?
Backround:
I have a webpage against which I write a java based test script with selenium webdriver - I can't change the page, I just have to write the tests. There are issues with some external content that the site loads from another domain. The external stuff is some javascript code that is actually not needed for my tests, but that the page in question includes. Now the problem. Sometimes the external sources are super slow, preventing the the webdriver to load the page within the given page load timeout (20 sec). My tests actually would run fine, because the page is in fact loaded - all html is there, all internal scripts are loaded and would work.
Random thoughts about this:
There are extensions for different browsers that would do what I ask, but I need to run my tests with several browsers, namely chrome, firefox and phantomjs. And there is no such thing like phantomjs extensions. I need a solution that is purely based on the webdriver technology if possible. I am willing to program a separate solution for each browser, though.
I appreciate any idea about how to address this.
Solution is to use proxy. Webdriver integrates very well with browsermob proxy: http://bmp.lightbody.net/
private WebDriver initializeDriver() throws Exception {
// Start the server and get the selenium proxy object
ProxyServer server = new ProxyServer(proxy_port); // package net.lightbody.bmp.proxy
server.start();
server.setCaptureHeaders(true);
// Blacklist google analytics
server.blacklistRequests("https?://.*\\.google-analytics\\.com/.*", 410);
// Or whitelist what you need
server.whitelistRequests("https?://*.*.yoursite.com/.*. https://*.*.someOtherYourSite.*".split(","), 200);
Proxy proxy = server.seleniumProxy(); // Proxy is package org.openqa.selenium.Proxy
// configure it as a desired capability
DesiredCapabilities capabilities = new DesiredCapabilities();
capabilities.setCapability(CapabilityType.PROXY, proxy);
// start the driver ;
Webdriver driver = new FirefoxDriver(capabilities);
return driver;
}
EDIT:
people are often asking for http status codes, you can easily retrive them using the proxy. Code can be something like this:
// create a new har with given label
public void setHar(String label) {
server.newHar(label);
}
public void getHar() throws IOException {
// FIXME : What should be done with the this data?
Har har = server.getHar();
if (har == null) return;
File harFile = new File("C:\\localdev\\bla.har");
har.writeTo(harFile);
for (HarEntry entry : har.getLog().getEntries()) {
// Check for any 4XX and 5XX HTTP status codes
if ((String.valueOf(entry.getResponse().getStatus()).startsWith("4"))
|| (String.valueOf(entry.getResponse().getStatus()).startsWith("5"))) {
log.warn(String.format("%s %d %s", entry.getRequest().getUrl(), entry.getResponse().getStatus(),
entry.getResponse().getStatusText()));
//throw new UnsupportedOperationException("Not implemented");
}
}
}
You can chain the proxy, there isn't much documentation out there about doing so:
http://www.nerdnuts.com/2014/10/browsermob-behind-a-corporate-proxy/
We were able to use browsermob behind a corporate proxy using the following code:
// start the proxy
server = new ProxyServer(9090);
server.start();
server.setCaptureContent(true);
server.setCaptureHeaders(true);
server.addHeader(“accept-encoding”, “”);//turn off gzip
// Configure proxy server to use our network proxy
server.setLocalHost(InetAddress.getByName(“127.0.0.1″));
/**
* THIS IS THE MAJICK!
**/
HashMap<String, String> options = new HashMap<String, String>();
options.put(“httpProxy”, “172.20.4.115:8080″);
server.setOptions(options);
server.autoBasicAuthorization(“172.20.4.115″, “username”, “password”);
// get the Selenium proxy object
Proxy proxy = server.seleniumProxy();
DesiredCapabilities capabilities = DesiredCapabilities.phantomjs();
capabilities.setCapability(CapabilityType.PROXY, proxy);
Related
I'm writing a simple script and I want selenium to fail the script or at least log the errors that come in http response.
Stable Selenium versions (below 4) are not supporting such functionality.
It has no ability to read HTTP responses.
Tools like Rest Assured are used for such things.
You can use browsermob-proxy tool to achieve that. You only should understand that when you call a page is actually causes a lot of requests to (different) servers and each responds with its own code.
For example the code could look like:
#BeforeTest
public void setUp(){
/*
Create and start proxy instance here
Configure driver to use that created proxy here
*/
}
#Test
public void testHeaders(){
// Create a map that would be storing url-to-status association
// Use synchronized version of a hash map because
// proxy handles connections in parallel
Map<String, Integer> urlToStatuses = new ConcurrentHashMap<>();
// Configure handler that would be listening for responses
// and fill the map
browserMobProxy.addResponseFilter((response, contents, messageInfo) -> {
urlToStatuses.put(messageInfo.getOriginalUrl(), response.status().code());
});
// Call a page
driver.get("https://google.com");
// Test the code of required URL
Assert.assertEquals(urlToStatuses.get("https://google.com").intValue(), 200);
// Do no forget to clear the map before the next assertion
urlToStatuses.clear();
}
I am quite new to ZAP and I have to use ZAP java API to perform security tests on a web application, using Selenium to navigate on a browser, creating traffic for ZAP. I am using Eclipse IDE for Enterprise Java Developers version 2019-03 (4.11.0) with Java jdk-1.8.0_212 and Maven 3.5.3, and 3 imported libraries: harlib version 1.1.1 (edu.umass.cs.benchlab harlib), zap api version 1.7.0 (org.owasp zaproxy-api) and proxy version 2.4.2 snapshot (net.continuumsecurity zap-java-api)
I followed this tutorial: https://dzone.com/articles/automate-zap-security-tests-with-selenium-webdrive-1, which worked. Just to recap, it is using 3 classes: one to instantiate the web browser (BrowserDriverFactory.java), one to actually store all the navigation functions and parameters (WebSiteNavigation.java) and one to create the ZAP proxy, configure it and perform the security tests as #Test functions (ZapSecurityTest.java).
I am having a bit of a problem proxying my local application: : it seems that my local application does not use the ZAP proxy even though I the chromedriver is well aware of the proxy (I see the non local traffic in ZAP UI). The weird thing is that I even explicitly added my local URL to the context and set this to in scope:
clientapi.context.includeInContext(contextName,
java.util.regex.Pattern.quote(WebSiteNavigation.BASE_URL));
clientapi.context.setContextInScope(contextName, "true");
After doing that and navigating on my local app, I see the related non local traffic in ZAP UI but not the local one. And when I check what clientapi.context.urls(contextName) returns, it is empty.
I create my proxy with:
private static Proxy createZapProxyConfiguration() {
Proxy proxy = new Proxy();
proxy.setHttpProxy(ZAP_PROXYHOST + ":" + ZAP_PROXYPORT);
proxy.setSslProxy(ZAP_PROXYHOST + ":" + ZAP_PROXYPORT);
return proxy;
}
Which is called by:
#Before
public void setUp() {
// Configure ZAP Scanner
zapScanner = new ZAProxyScanner(ZAP_PROXYHOST, ZAP_PROXYPORT,
ZAP_APIKEY);
clientapi = new ClientApi(ZAP_PROXYHOST, ZAP_PROXYPORT);
// Start new session
zapScanner.clear();
log.info("Started a new session: Scanner");
// Create ZAP API client
zapSpider = (Spider) zapScanner;
// Create driver object
driver = BrowserDriverFactory.createChromeDriver
(createZapProxyConfiguration(), BROWSER_DRIVER_PATH);
siteNavigation = new WebSiteNavigation(driver);
}
Which also calls :
public static WebDriver createChromeDriver(Proxy proxy, String path) {
// Set proxy in the chrome browser
DesiredCapabilities capabilities = DesiredCapabilities.chrome();
capabilities.setCapability("proxy", proxy);
// Set system property for chrome driver with the path
System.setProperty("webdriver.chrome.driver", path);
capabilities.setCapability(CapabilityType.ACCEPT_SSL_CERTS, true);
ChromeOptions options = new ChromeOptions();
options.merge(capabilities);
return new ChromeDriver(options);
}
Weirdly enough, a few days ago when I first set this up, it was working like a charm… I then tried to automate the authentication process, and since then, it does not work…
Am I missing something here? I am at least looking at the right place?
Chrome by default does not proxy requests sent to localhost by default, so if this is your case you need to pass an additional flag to the chrome driver. This is how it is configured using JSON:
capabilities: {
browserName: 'chrome',
proxy: proxy
chromeOptions: {
args: ['--proxy-bypass-list=<-loopback>']
}
},
The same can be done using Java API.
In selenium test step (like a button click) i want to prevent the selenium waiting for page finish loading. I cant throw the load Exception because then i cant work with the page anymore.
Its possible to do a simmilar thing like this:
DesiredCapabilities dr = DesiredCapabilities.chrome();
dr.setCapability("pageLoadStrategy", "none");
WebDriver driver = new RemoteWebDriver(new URL("...."), dr);
What I want is like "dr.setCapability("pageLoadStrategy", "none");" but just for one specifique step.
Does anyone know a way to do this?
Capabilities are no longer editable once the browser is launched.
One way to temporary disable the waiting is to implement your own get with a script injection.
Something like this:
//
// loads the page and stops the loading without exception after 2 sec if
// the page is still loading.
//
load(driver, "https://httpbin.org/delay/10", 2000);
public static void load(WebDriver driver, String url, int timeout) {
((JavascriptExecutor)driver).executeScript(
"var url = arguments[0], timeout = arguments[1];"
"window.setTimeout(function(){window.location.href = url}, 1);" +
"var timer = window.setTimeout(window.stop, timeout);" +
"window.onload = function(){window.clearTimeout(timer)}; "
, url, timeout);
}
As of the current implementation of Selenium once we configure the WebDriver instance with our intended configuration through DesiredCapabilities class and initialize the WebDriver session to open a Browser, we cannot change the capabilities runtime.
It is worth to mention, somehow if you are able to retrieve the runtime capabilities still you won't be able to change them back.
So, in-order to make a change in the pageLoadStrategy you have to initiate a new WebDriver session.
Here is #JimEvans clear and concise answer (as of Oct 24 '13 at 13:02) related to proxy settings capability:
When you set a proxy for any given driver, it is set only at the time WebDriver session is created; it cannot be changed at runtime. Even if you get the capabilities of the created session, you won't be able to change it. So the answer is, no, you must start a new session if you want to use different proxy settings.
I have a very simple selenium-webdriver script. I would like to do HTTP authentication using webdriver.
Script:
WebDriver driver = new FirefoxDriver();
driver.get("http://www.httpwatch.com/httpgallery/authentication/");
driver.findElement(By.id("displayImage")).click();
Thread.sleep(2000);
driver.switchTo().alert().sendKeys("httpwatch");
Issue:
driver.switchTo().alert().sendKeys("httpwatch");
throws
org.openqa.selenium.NoAlertPresentException: No alert is present
Question:
Does Webdriver find only an alert dialog as alert?
What are my options to automate this without using AutoIt OR http:// username:password #somesite
EDIT
Alert has below method and does not seem to have been implemented yet.
driver.switchTo().alert().authenticateUsing(new UsernameAndPassword("username","password"))
The problem is that this is not a javascript popup hence you cannot manipulate it via selenium's alert().
If both AutoIt and submitting credentials in the URL (the easiest option - just open up the url and click "Display Image") are not options for you, another approach could be to use AutoAuth firefox addon to automatically submit the previously saved credentials:
AutoAuth automatically submits HTTP authentication dialogs when you’ve
chosen to have the browser save your login information. (If you’ve
already told the browser what your username and password are, and
you’ve told it to remember that username and password, why not just
have it automatically submit it instead of asking you each time?)
Following the answer suggested in HTTP Basic Auth via URL in Firefox does not work? thread:
Install AutoAuth Firefox plugin;
Visit the site where the authentication is needed. Enter your username and password and make sure to choose to save the credentials;
Save AutoAuth installation file at your hard drive: at the plugin page, right click at “Add to Firefox” and “Save link as”;
Instantiate Firefox webdriver as following:
FirefoxProfile firefoxProfile = new ProfilesIni().getProfile("default");
File pluginAutoAuth = new File("src/test/resources/autoauth-2.1-fx+fn.xpi");
firefoxProfile.addExtension(pluginAutoAuth);
driver = new FirefoxDriver(firefoxProfile);
Also, in a way similar to AutoIt option - you can use sikuli screen recognition and automation tool to submit the credentials in the popup.
Also see other ideas and options:
Support BASIC and Digest HTTP authentication
Handling browser level authentication using Selenium
The Basic/NTLM authentication pop-up is a browser dialog window. WebDriver (Selenium 2.0) cannot interact with such dialog windows. The reason for this is because WebDriver aims solely at mimicking user interaction with websites, and browser dialog windows are currently not in that scope. JavaScript dialog windows, are part of the website, so WebDriver can handle those. In Selenium 1.0 it is possible to do basic authentication.
So how to solve this issue? 1) Authentication via URL http://username:password#website.com 2) Use a browser plugin that will handle the Basic/NTLM autentication 3) Use a local proxy that will modify the request header and pass along the username/password and 4) Make use of a robot, like AutoIt, or some Java library.
Option 1: is the easiest and has the least impact on the system/test. Option 2: has a high browser impact as your loading plugins. Also every browser uses its own plugin and it's possible that the required plugin for a certain browser is not available. Option 3: Works well with HTTP, but HTTPS requires custom certicates thus not always an option. Not much impact on both system and test. Option 4: Mimics keyboard presses, its a go to solution but prone to errors. Only works when the dialog window has focus and it is possible that this is not always the case.
I faced same issue, and got some concrete solution using robot class. Its workaround or solution, Let see , but it works.
public class DialogWindow implements Runnable {
#Override
public void run() {
try {
entercredentials();
} catch (InterruptedException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}
public void entercredentials() InterruptedException {
Thread.sleep(5000);
try {
enterText("username");
enterSpecialChar(KeyEvent.VK_TAB);
enterText("password");
enterSpecialChar(KeyEvent.VK_ENTER);
} catch (AWTException e) {
}
}
private void enterText(String text) throws AWTException {
Robot robot = new Robot();
byte[] bytes = text.getBytes();
for (byte b : bytes) {
int bytecode = b;
// keycode only handles [A-Z] (which is ASCII decimal [65-90])
if (bytecode> 96 && bytecode< 123)
bytecode = bytecode - 32;
robot.delay(40);
robot.keyPress(bytecode);
robot.keyRelease(bytecode);
}
}
private void enterSpecialChar(int s) throws AWTException {
Robot robot = new Robot();
robot.delay(40);
robot.keyPress(s);
robot.keyRelease(s);
}
}
How to call it
WebDriver driver= new FirefoxDriver()// or any other driver with capabilities and other required stuff
(new Thread(new DialogWindow())).start();
driver.get(url);
What is the point of the method FirefoxBinary#startProfile?
How would one control a browser opened using the above method?
FirefoxBinary binary = new FirefoxBinary(new File("path\\to\\binary.exe"));
String profilePath = "path\\to\\profile";
FirefoxProfile profile = new FirefoxProfile(new File(profilePath));
binary.startProfile(profile, profilePath, "");
//A browser opens at this point, but how do I send it commands or attach
//it to a WebDriver?
The purpose of the method is to do exactly what it does: start an instance of Firefox using the specified profile. WebDriver makes a copy of the profile in the temp directory and adds the WebDriver Firefox extension into this copy of the profile. Th startProfile method launches Firefox, making sure to clean up the profile so that it's usable by the new instance. If you know what port the browser extension that gets added to the profile is listening on, you could connect to it and control the browser using the WebDriver JSON wire protocol commands.
However, unless you really know what you're doing, this is the complicated way to do it. You're far better off calling the FirefoxDriver constructor, passing in the FirefoxBinary object, and letting the internals of the constructor call the startProfile method for you.