I am unable to capture request and response using browsermob(selenium+PhantomJS browser)
please refer the sample code
server = new BrowserMobProxyServer();
server.start(0);
server.newHar("contracts");
Capabilities:
DesiredCapabilities capabilities = new DesiredCapabilities();
capabilities.setJavascriptEnabled(true); capabilities.setCapability(CapabilityType.ACCEPT_SSL_CERTS, true);
capabilities.setCapability(PhantomJSDriverService.PHANTOMJS_CLI_ARGS,
new String[] {"--web-security=false", "--ssl-protocol=any",
"--ignore-ssl-errors=yes"});
capabilities.setCapability("takeScreenshot", true);
URL hubUrl = new URL("http://152.188.0.42:5555/wd/hub");
server.enableHarCaptureTypes(CaptureType.REQUEST_CONTENT, CaptureType.RESPONSE_CONTENT);
launching the application:
driver = new RemoteWebDriver(hubUrl,capabilities);
driver.get("http://www.google.com");
Creating the har file:
Har har = server.getHar();
FileOutputStream fos = new FileOutputStream("runnowNew.har");
har.writeTo(fos);
Har which is generated by using the above code:
{
"log":
{
"version":"1.2",
"creator":{"name":"BrowserMob Proxy","version":"2.1.2","comment":""},
"pages":[{"id":"contracts","startedDateTime":"2016-10-05T12:56:33.460+05:30","title":"contracts","pageTimings":{"comment":""},"comment":""}],
"entries":[],
"comment":""
}
}
I think the problem is that you don't use BrowserMob Proxy as a proxy for the Selenium traffic.
You need to set the proxy of Selenium (ip & port) to the one configured in BrowserMob Proxy.
IP is probably 127.0.0.1 since you work locally, and you can use server.getPort() to get the port that BrowserMob Proxy is listening to.
Related
I need to get headless chrome to ignore certificate errors. The option is ignored when running in headless mode, and the driver returns empty html body tags when navigating to an https resource.
<html xmlns="http://www.w3.org/1999/xhtml"><head></head><body></body></html>
This is how I am configuring my chrome driver.
ChromeOptions chromeOptions = new ChromeOptions();
chromeOptions.addArguments("--headless", "--disable-gpu", "--window-size=1920,1200","--ignore-certificate-errors");
DesiredCapabilities cap=DesiredCapabilities.chrome();
cap.setCapability(ChromeOptions.CAPABILITY, chromeOptions);
cap.setCapability(CapabilityType.ACCEPT_SSL_CERTS, true);
cap.setCapability(CapabilityType.ACCEPT_INSECURE_CERTS, true);
chromeHeadlessDriver = new ChromeDriver(cap);
This thread confirms that --ignore-certificate-errors is ignored in headless mode.
They mention about devtool protocol.
Is it something I can invoke from java? Are there any other alternatives?
There is an excellent article on medium.com by sahajamit
and i have tested the below code, it works perfectly fine with self-signed certificate https://badssl.com/
ChromeOptions options = new ChromeOptions();
options.setExperimentalOption("useAutomationExtension", false);
options.addArguments("--headless", "--window-size=1920,1200","--ignore-certificate-errors");
DesiredCapabilities crcapabilities = DesiredCapabilities.chrome();
crcapabilities.setCapability(ChromeOptions.CAPABILITY, options);
crcapabilities.setCapability(CapabilityType.ACCEPT_SSL_CERTS, true);
crcapabilities.setCapability(CapabilityType.ACCEPT_INSECURE_CERTS, true);
System.setProperty(ChromeDriverService.CHROME_DRIVER_LOG_PROPERTY, "C:\\temp\\chrome\\chromedriver.log");
System.setProperty(ChromeDriverService.CHROME_DRIVER_EXE_PROPERTY, "C:\\temp\\chrome\\chromedriver.exe");
ChromeDriverService service = null;
try {
service = new ChromeDriverService.Builder()
.usingAnyFreePort()
.withVerbose(true)
.build();
service.start();
} catch (IOException e) {
e.printStackTrace();
}
RemoteWebDriver driver = new RemoteWebDriver(service.getUrl(),crcapabilities);
driver.get("https://self-signed.badssl.com/");
System.out.println(driver.getPageSource());
driver.quit();
software/framework versions
Google Chrome Version 64.0.3282.186
Google Chrome Driver Version 64.0.3282.186
Selenium version 3.11.0
This works for me on ChromeDriver 80.
ChromeOptions option = new ChromeOptions();
option.AddArgument("--headless");
option.AcceptInsecureCertificates = true;
driver = new ChromeDriver(option);
#amila-kumara is working but, usage of DesiredCapabilities.chrome() gives warning to use ChromeOptions. Please see the updated answer.
Set the chrome option values
System.setProperty("webdriver.chrome.driver", Config.NDAC_WEBDRIVER_PATH);
ChromeOptions options = new ChromeOptions();
options.addArguments("--window-size=1920,1200");
options.setAcceptInsecureCerts(true);
options.setHeadless(true);
Start the service
ChromeDriverService service = null;
try {
service = new ChromeDriverService.Builder()
.usingAnyFreePort()
.withVerbose(true)
.build();
service.start();
remoteWebdriverUrl = service.getUrl();
System.out.println("Starting the url " + remoteWebdriverUrl);
} catch (IOException e) {
e.printStackTrace();
}
Note: I was facing the issue while closing the driver(with RemoteWebDriver), chromedriver.exe process won't close even when you use driver.quit(). To fix the issue use ChromeDriver instead of RemoteWebDriver
RemoteWebDriver driver = new ChromeDriver(service, options);
To properly close the driver, use
driver.close();
driver.quit();
I am using Browsermob proxy to catch HAR content of loaded web page thru chrome webdriver.
<dependency>
<groupId>net.lightbody.bmp</groupId>
<artifactId>browsermob-core-littleproxy</artifactId>
<version>2.1.0-beta-3</version>
</dependency>
And here is how I initialize proxy:
ChromeOptions options = new ChromeOptions();
for (String extension : DriverProperties.CHROME_EXTENSIONS.getStringArray()) {
options.addExtensions(new File(extension));
}
DesiredCapabilities capabilities = new DesiredCapabilities();
capabilities.setCapability(CapabilityType.ForSeleniumServer.ENSURING_CLEAN_SESSION, true);
capabilities.setCapability(CapabilityType.SUPPORTS_APPLICATION_CACHE, false);
capabilities.setCapability(ChromeOptions.CAPABILITY, options);
capabilities.setCapability(CapabilityType.ACCEPT_SSL_CERTS, true);
proxy = new BrowserMobProxyServer();
proxy.start(0);
proxy.setHarCaptureTypes(CaptureType.RESPONSE_CONTENT, CaptureType.RESPONSE_BINARY_CONTENT);
// get the Selenium proxy object
Proxy seleniumProxy = ClientUtil.createSeleniumProxy(proxy);
capabilities.setCapability(CapabilityType.PROXY, seleniumProxy);
proxy.newHar();
ChromeDriver driver = new ChromeDriver(capabilities);
But when I try to get url via https (for exapmle https://screen.yahoo.com) I see "This web page has a redirect loop" in the web browser. Is there any solution for this?
I am writing automation test and to capture the network call made in background, I am using browsermob-proxy.
In browsermob-proxy, I want to set cookie before making requests. How can i do it?
Below is my code:-
String strFilePath = "data.har";
// start the proxy
ProxyServer server = new ProxyServer(4444);
server.start();
server.setCaptureHeaders(true);
server.setCaptureContent(true);
// get the Selenium proxy object
Proxy proxy = server.seleniumProxy();
FirefoxProfile profile = new FirefoxProfile();
String userAgent = "Mozilla/5.0 (iPhone; U; CPU iPhone OS 4_3_2 like Mac OS X; en-us) AppleWebKit/533.17.9 (KHTML, like Gecko) Version/5.0.2 MobileWeb/8H7 Safari/6533.18.5";
profile.setPreference("general.useragent.override", userAgent);
// configure it as a desired capability
DesiredCapabilities capabilities = DesiredCapabilities.firefox();
capabilities.setCapability(CapabilityType.PROXY, proxy);
capabilities.setCapability(FirefoxDriver.PROFILE, profile);
// start the browser up
WebDriver driver = new FirefoxDriver(capabilities);
final String[] remoteHost = {null};
final String[] analytics = {null};
final String[] fetchAdjs = {null};
server.addRequestInterceptor(new RequestInterceptor()
{
int googleCount = 0;
int adjs = 0;
#Override
public void process(BrowserMobHttpRequest browserMobHttpRequest, Har har)
{
remoteHost[0] = browserMobHttpRequest.getProxyRequest().getRemoteHost();
String request = browserMobHttpRequest.getProxyRequest().getRequestURL().toString();
if (request.matches(".*google.*"))
googleCount = googleCount + 1;
if (request.matches(".*test.*"))
adjs = adjs + 1;
analytics[0] = String.valueOf(googleCount);
fetchAdjs[0] = String.valueOf(adjs);
// System.out.println(browserMobHttpRequest.getMethod().getAllHeaders()[1]); // user agent
System.out.println(browserMobHttpRequest.getProxyRequest());
}
});
// create a new HAR with the label "apple.com"
server.newHar("assertselenium.com");
// open yahoo.com
driver.get("http://test.com");
Thread.sleep(3000);
Thread.sleep(3000);
driver.get("http://test.com/316782/content/fDxL4zzv");
Thread.sleep(3000);
// get the HAR data
Har har = server.getHar();
FileOutputStream fos = new FileOutputStream(strFilePath);
// view har file here --> http://pcapperf.appspot.com/
har.writeTo(fos);
server.stop();
driver.quit();
Using a request filter is the right approach. However, I would highly recommend building the latest version of BrowserMob Proxy with LittleProxy integration. Its filters are much more reliable and easier to use. See the github page for information on building and using the latest version. The LittleProxy interceptors section will be particularly relevant.
Here's a simple example of adding a cookie to every request using the new request filters:
BrowserMobProxy proxy = new BrowserMobProxyServer();
proxy.start();
proxy.addRequestFilter(new RequestFilter() {
#Override
public HttpResponse filterRequest(HttpRequest request, HttpMessageContents contents, HttpRequest originalRequest) {
request.headers().add("Cookie", "added-cookie=added-value");
return null;
}
});
I'm a rookie test developer using selenium 2.45 and I'm trying to configure my FirefoxDriver to use my company's proxy settings. I am failing to do so :)
I am following the instruction from here to create a profile on-the-fly:
Using a Proxy for FF
My code looks like this:
public static WebDriver driver;
String usedProxy = "http://myproxy:8080";
org.openqa.selenium.Proxy proxy = new org.openqa.selenium.Proxy();
proxy.setHttpProxy(usedProxy).setFtpProxy(usedProxy).setSslProxy(usedProxy);
DesiredCapabilities cap = new DesiredCapabilities();
cap.setCapability(CapabilityType.PROXY, proxy);
driver = new FirefoxDriver(cap);
driver.manage().timeouts().implicitlyWait(10, TimeUnit.SECONDS);
driver.get("https://TestWebsite.com");
I am not receiving any kind of errors but the connection is not working for this browser. When checking Options>Advanced>Network>Connection Settings from Firefox menu, proxy is set to manual but text inputs only contain "http://"
PS: I have a feeling this is relevant but am not sure: TestWebsite.com will load via https only (it's a shopping cart)
If your goal is to test your functionality on different IP addresses, you can use the Tor Browser.
public IWebDriver Driver { get; set; }
public Process TorProcess { get; set; }
public WebDriverWait Wait { get; set; }
[TestInitialize]
public void SetupTest()
{
String torBinaryPath = #"C:\Users\aangelov\Desktop\Tor Browser\Browser\firefox.exe";
this.TorProcess = new Process();
this.TorProcess.StartInfo.FileName = torBinaryPath;
this.TorProcess.StartInfo.Arguments = "-n";
this.TorProcess.StartInfo.WindowStyle = ProcessWindowStyle.Maximized;
this.TorProcess.Start();
FirefoxProfile profile = new FirefoxProfile();
profile.SetPreference("network.proxy.type", 1);
profile.SetPreference("network.proxy.socks", "127.0.0.1");
profile.SetPreference("network.proxy.socks_port", 9150);
this.Driver = new FirefoxDriver(profile);
this.Wait = new WebDriverWait(this.Driver, TimeSpan.FromSeconds(60));
}
[TestCleanup]
public void TeardownTest()
{
this.Driver.Quit();
this.TorProcess.Kill();
}
Here is the code to refresh the Tor identity.
public void RefreshTorIdentity()
{
Socket server = null;
try
{
IPEndPoint ip = new IPEndPoint(IPAddress.Parse("127.0.0.1"), 9151);
server = new Socket(AddressFamily.InterNetwork, SocketType.Stream, ProtocolType.Tcp);
server.Connect(ip);
server.Send(Encoding.ASCII.GetBytes("AUTHENTICATE \"johnsmith\"" + Environment.NewLine));
byte[] data = new byte[1024];
int receivedDataLength = server.Receive(data);
string stringData = Encoding.ASCII.GetString(data, 0, receivedDataLength);
server.Send(Encoding.ASCII.GetBytes("SIGNAL NEWNYM" + Environment.NewLine));
data = new byte[1024];
receivedDataLength = server.Receive(data);
stringData = Encoding.ASCII.GetString(data, 0, receivedDataLength);
if (!stringData.Contains("250"))
{
Console.WriteLine("Unable to signal new user to server.");
server.Shutdown(SocketShutdown.Both);
server.Close();
}
}
finally
{
server.Close();
}
}
You can find more detailed information here: http://automatetheplanet.com/using-selenium-webdriver-tor-c-code/
The code examples are in C# but the code should be identical in Java.
Try firefox profile as well. Note this is C# code and converting to Java should be fairly simply
string usedProxy = "http://myproxy:8080";
Proxy proxy = new OpenQA.Selenium.Proxy();
proxy.HttpProxy = usedProxy;
proxy.FtpProxy = usedProxy;
proxy.SslProxy = usedProxy;
FirefoxProfile profile = new FirefoxProfile();
profile.SetProxyPreferences(proxy);
This code works for me:
FirefoxOptions options = new FirefoxOptions();
FirefoxProfile profile = new FirefoxProfile();
profile.setPreference("network.proxy.type", 1);
profile.setPreference("network.proxy.http", <proxy_url>);
profile.setPreference("network.proxy.http_port", 8080);
profile.setPreference("network.proxy.ssl", <proxy_url>);
profile.setPreference("network.proxy.ssl_port", 8080);
options.setProfile(profile);
options.setAcceptInsecureCerts(true)
options.setCapability("disable-restore-session-state", true);
options.setCapability("marionette", true);
WebDriver driver = new FirefoxDriver(options);
I followed answers from this link: Webdriver and proxy server for firefox
public class Test_One
{
public static void main(String[] args) throws Exception {
ProxyServer server = new ProxyServer(8105);
server.start();
server.setCaptureHeaders(true);
server.setCaptureContent(true);
server.newHar("test");
DesiredCapabilities capabilities = new DesiredCapabilities();
Proxy proxy = server.seleniumProxy();
FirefoxProfile profile = new FirefoxProfile();
profile.setAcceptUntrustedCertificates(true);
profile.setAssumeUntrustedCertificateIssuer(true);
profile.setPreference("network.proxy.http", "localhost");
profile.setPreference("network.proxy.http_port", 8105);
profile.setPreference("network.proxy.ssl", "localhost");
profile.setPreference("network.proxy.ssl_port", 8105);
profile.setPreference("network.proxy.type", 1);
profile.setPreference("network.proxy.no_proxies_on", "");
//profile.setProxyPreferences(proxy);
profile.setPreference(key, value)
capabilities.setCapability(FirefoxDriver.PROFILE,profile);
capabilities.setCapability(CapabilityType.PROXY, proxy);
WebDriver driver = new FirefoxDriver(capabilities);
driver.get("http://www.google.com");
Har har1 = server.getHar();
}
}
I'm new to selenium and broswermob. This is my code. Whe I try to execute this and getting error The method setProxyPreferences(Proxy) is undefined for the type FirefoxProfile. How to solve this?
You (generally) don't need to configure the FirefoxProfile settings manually. The Using With Selenium section of the readme file gives an example of how to use the proxy with Selenium:
// start the proxy
ProxyServer server = new ProxyServer(4444);
server.start();
// get the Selenium proxy object
Proxy proxy = server.seleniumProxy();
// configure it as a desired capability
DesiredCapabilities capabilities = new DesiredCapabilities();
capabilities.setCapability(CapabilityType.PROXY, proxy);
// start the browser up
WebDriver driver = new FirefoxDriver(capabilities);
// create a new HAR with the label "yahoo.com"
server.newHar("yahoo.com");
// open yahoo.com
driver.get("http://yahoo.com");
// get the HAR data
Har har = server.getHar();
If that doesn't work, it may be a problem with your specific version of Firefox and/or Selenium and/or BrowserMob Proxy. What versions are you using? A stack trace with the exact error message would also help.
Try removing profile from capabilities and passing it directly to FirefoxDriver:
driver = new FirefoxDriver(new FirefoxBinary(),profile,capabilities);