We Keep Coding

How to Conceal WebDriver in Geckodriver from BotD in Java?

I followed this post on Stackoverflow to disable Firefox WebDriver detection.
Launch Geckodriver:
System.setProperty("webdriver.gecko.driver", geckdriverExecutableFilePath);
File firefoxProfileFile = new File(fullPathOfFirefoxInstallationFolder);
FirefoxProfile firefoxProfile = null;
try {
firefoxProfile = new FirefoxProfile(firefoxProfileFile);
} catch (Exception e) {
e.printStackTrace();
}
I disabled WebDriver:
WebDriver Disabled
FirefoxOptions firefoxOptions = new FirefoxOptions();
firefoxOptions.setProfile(firefoxProfile);
// Disables WebRTC
firefoxProfile.setPreference("media.peerconnection.enabled", false);
I disabled Automation Extensions:
Automation Extension Disabled
// Disables Automation Extension
firefoxProfile.setPreference("useAutomationExtension", false);
I added Proxy:
DesiredCapabilities dc = DesiredCapabilities.firefox();
Proxy proxy = new Proxy();
proxy.setHttpProxy(ipAddress + ":" + port);
proxy.setFtpProxy(ipAddress + ":" + port);
proxy.setSslProxy(ipAddress + ":" + port);
dc.setCapability(CapabilityType.PROXY, proxy);
firefoxOptions.merge(dc);
driver = new FirefoxDriver(firefoxOptions);
Yet BotD still detects my browser as being controlled by automation tool.
BotD Detection
How can I solve this?

When using Selenium driven GeckoDriver initiated firefox Browsing Context
The webdriver-active flag is set to true when the user agent is under remote control. It is initially false.
where, webdriver returns true if webdriver-active flag is set, false otherwise.
As:
navigator.webdriver Defines a standard way for co-operating user agents to inform the document that it is controlled by WebDriver, for
example so that alternate code paths can be triggered during
automation.
Further #whimboo in his comments confirmed:
This implementation have to be conformant to this requirement. As such
we will not provide a way to circumvent that.
Conclusion
So, the bottom line is:
Selenium identifies itself
and there is no way to conceal the fact that the browser is WebDriver driven.
Recommendations
However some pundits have suggested some different approaches which can conceal the fact that the Mozilla Firefox browser is WebDriver controled through the usage of Firefox Profiles and Proxies as follows:
selenium4 compatible python code
from selenium.webdriver import Firefox
from selenium import webdriver
from selenium.webdriver.firefox.service import Service
from selenium.webdriver.firefox.options import Options
profile_path = r'C:\Users\Admin\AppData\Roaming\Mozilla\Firefox\Profiles\s8543x41.default-release'
options=Options()
options.set_preference('profile', profile_path)
options.set_preference('network.proxy.type', 1)
options.set_preference('network.proxy.socks', '127.0.0.1')
options.set_preference('network.proxy.socks_port', 9050)
options.set_preference('network.proxy.socks_remote_dns', False)
service = Service('C:\\BrowserDrivers\\geckodriver.exe')
driver = Firefox(service=service, options=options)
driver.get("https://www.google.com")
driver.quit()
Potential Solution
A potential solution would be to use the tor browser as follows:
selenium4 compatible python code
from selenium.webdriver import Firefox
from selenium import webdriver
from selenium.webdriver.firefox.service import Service
from selenium.webdriver.firefox.options import Options
import os
torexe = os.popen(r'C:\Users\username\Desktop\Tor Browser\Browser\TorBrowser\Tor\tor.exe')
profile_path = r'C:\Users\username\Desktop\Tor Browser\Browser\TorBrowser\Data\Browser\profile.default'
firefox_options=Options()
firefox_options.set_preference('profile', profile_path)
firefox_options.set_preference('network.proxy.type', 1)
firefox_options.set_preference('network.proxy.socks', '127.0.0.1')
firefox_options.set_preference('network.proxy.socks_port', 9050)
firefox_options.set_preference("network.proxy.socks_remote_dns", False)
firefox_options.binary_location = r'C:\Users\username\Desktop\Tor Browser\Browser\firefox.exe'
service = Service('C:\\BrowserDrivers\\geckodriver.exe')
driver = webdriver.Firefox(service=service, options=firefox_options)
driver.get("https://www.tiktok.com/")
References
You can find a couple of relevant detailed discussions in
How to initiate a Tor Browser 9.5 which uses the default Firefox to 68.9.0esr using GeckoDriver and Selenium through Python
How to connect to Tor browser using Python
How to use Tor with Chrome browser through Selenium

BotD detects you because you do not override navigator.webdriver attribute.
I was able to override it with this code:
((JavascriptExecutor)driver).executeScript("Object.defineProperty(navigator, 'webdriver', {get: () => undefined})");
Re-run your code with this line after driver.get("BotD url") and click on
'Start detect' on the BotD page.
It will no longer show that webdriver is detected.
I understand you are looking for a way to make it work before the initial page load.
But here are 2 things to consider:
Webdriver developers want their tool to be detected by browsers.
Gecko driver developers are not going to implement an option to disable navigator.webdriver attribute. (This is the official reply from gecko developer.)

Set capability on already running selenium webdriver

In selenium test step (like a button click) i want to prevent the selenium waiting for page finish loading. I cant throw the load Exception because then i cant work with the page anymore.
Its possible to do a simmilar thing like this:
DesiredCapabilities dr = DesiredCapabilities.chrome();
dr.setCapability("pageLoadStrategy", "none");
WebDriver driver = new RemoteWebDriver(new URL("...."), dr);
What I want is like "dr.setCapability("pageLoadStrategy", "none");" but just for one specifique step.
Does anyone know a way to do this?

Capabilities are no longer editable once the browser is launched.
One way to temporary disable the waiting is to implement your own get with a script injection.
Something like this:
//
// loads the page and stops the loading without exception after 2 sec if
// the page is still loading.
//
load(driver, "https://httpbin.org/delay/10", 2000);
public static void load(WebDriver driver, String url, int timeout) {
((JavascriptExecutor)driver).executeScript(
"var url = arguments[0], timeout = arguments[1];"
"window.setTimeout(function(){window.location.href = url}, 1);" +
"var timer = window.setTimeout(window.stop, timeout);" +
"window.onload = function(){window.clearTimeout(timer)}; "
, url, timeout);
}

As of the current implementation of Selenium once we configure the WebDriver instance with our intended configuration through DesiredCapabilities class and initialize the WebDriver session to open a Browser, we cannot change the capabilities runtime.
It is worth to mention, somehow if you are able to retrieve the runtime capabilities still you won't be able to change them back.
So, in-order to make a change in the pageLoadStrategy you have to initiate a new WebDriver session.
Here is #JimEvans clear and concise answer (as of Oct 24 '13 at 13:02) related to proxy settings capability:
When you set a proxy for any given driver, it is set only at the time WebDriver session is created; it cannot be changed at runtime. Even if you get the capabilities of the created session, you won't be able to change it. So the answer is, no, you must start a new session if you want to use different proxy settings.

Selenium WebDriver FirefoxBinary#startProfile

What is the point of the method FirefoxBinary#startProfile?
How would one control a browser opened using the above method?
FirefoxBinary binary = new FirefoxBinary(new File("path\\to\\binary.exe"));
String profilePath = "path\\to\\profile";
FirefoxProfile profile = new FirefoxProfile(new File(profilePath));
binary.startProfile(profile, profilePath, "");
//A browser opens at this point, but how do I send it commands or attach
//it to a WebDriver?

The purpose of the method is to do exactly what it does: start an instance of Firefox using the specified profile. WebDriver makes a copy of the profile in the temp directory and adds the WebDriver Firefox extension into this copy of the profile. Th startProfile method launches Firefox, making sure to clean up the profile so that it's usable by the new instance. If you know what port the browser extension that gets added to the profile is listening on, you could connect to it and control the browser using the WebDriver JSON wire protocol commands.
However, unless you really know what you're doing, this is the complicated way to do it. You're far better off calling the FirefoxDriver constructor, passing in the FirefoxBinary object, and letting the internals of the constructor call the startProfile method for you.

running vertx behind a proxy

Has anybody had a luck trying to use vertx bihind a corporate proxy? I have tried whatever possible ways that comes to my head to provide the proxy information to vertx. Nothing works so far.
set environment variable http_proxy=http://mycorporate.proxy.com:8080 - no luck
set environment variable VERTX_OPTS='-Dhttp.proxyHost=mycorporate.proxy.com -Dhttp.proxyPort=8080' - no luck
set environment variables
http.proxyHost=mycorporate.proxy.com
http.proxyPort=8080
no luck
Injecting extra echo into vertx command I can see that proxy related parameters are being passed to JVM correctly but required module still can't be downloaded ("vertx run hello.js" just stuck obviously trying to download io.vertx~lang-rhino~2.0.0-final)
Proxy itself is ok - I'm using it with no problem for maven, sbt and other different stuff requiring proxy
Same laptop being used from home can successfully run "vertx run hello.js" with downloading io.vertx~lang-rhino~2.0.0-final (for the first run)
I have just started evaluating vertx for our company needs and this is my very first choking point hindering my further attempts to make a decision. So far I have to follow next steps as workaround: 1 Run from home and get whatever required modules in sys-mods. 2 Manually upload the module(s) to sys-mods on the test server when back to the office.
Obviously this is not a normal way to run anything.

I had similar problem. I figure out that HttpClient form does not read settings from JVM_OPTS.
So solution was following:
Edit your vertx.bat(sh)
set JVM_OPTS=-Dhttp.proxyHost=xxxx -Dhttp.proxyPort=xxxx
and then in code related to httpClient try sth like this
HttpClient client = vertx.createHttpClient();
String proxyHost = System.getProperty("http.proxyHost", "none");
Integer proxyPort = Integer.valueOf(System.getProperty("http.proxyPort", "80"));
if(!"none".equalsIgnoreCase(proxyHost)){
client.setHost(proxyHost);
client.setPort(proxyPort);
}
and later in code releted to HTTP request
MultiMap map = new CaseInsensitiveMultiMap();
map.add("Host", domainName); //get domain of REQUESTED_URL
client.getNow(REQUESTED_URL, map, new new Handler<HttpClientResponse>(){...});

Building on #kamyk-pl answer. If you are using latest version of Vertx this is what you can do :
WebClientOptions webClientOptions = new WebClientOptions();
String proxyHost = System.getProperty("http.proxyHost", "none");
Integer proxyPort = Integer.valueOf(System.getProperty("http.proxyPort", "80"));
if(!"none".equalsIgnoreCase(proxyHost)){
ProxyOptions proxyOptions = new ProxyOptions();
proxyOptions.setHost(proxyHost);
proxyOptions.setPort(proxyPort);
webClientOptions.setProxyOptions(proxyOptions);
}
WebClient.create(vertx, webClientOptions)
Optionally you also want to add this if you are testing SSL connections
webClientOptions.setVerifyHost(false);
webClientOptions.setTrustAll(true);

I use vertx 3.8.0
HttpClient c = vertx.createHttpClient();
RequestOptions requestOptions = new RequestOptions();
requestOptions.setHost(ip).setPort(port).setURI("http://httpbin.org/ip");
c.request(HttpMethod.GET, requestOptions).handler(new Handler<HttpClientResponse>() {
#Override
public void handle(HttpClientResponse event) {
event.handler(new Handler<Buffer>() {
#Override
public void handle(Buffer event) {
System.out.println(event);
}
});
}
}).end();

To set the proxy, edit vertx.bat (if you are on Windows) and add
set JVM_OPTS=-Dhttp.proxyHost=xxxx -Dhttp.proxyPort=xxxx

prevent external content to be loaded in selenium webdriver test

The question:
Is is possible to tell a browser that is controlled by selenium webdriver to not load any content from external sources, or alternatively, not load resources from a given list of domains?
Backround:
I have a webpage against which I write a java based test script with selenium webdriver - I can't change the page, I just have to write the tests. There are issues with some external content that the site loads from another domain. The external stuff is some javascript code that is actually not needed for my tests, but that the page in question includes. Now the problem. Sometimes the external sources are super slow, preventing the the webdriver to load the page within the given page load timeout (20 sec). My tests actually would run fine, because the page is in fact loaded - all html is there, all internal scripts are loaded and would work.
Random thoughts about this:
There are extensions for different browsers that would do what I ask, but I need to run my tests with several browsers, namely chrome, firefox and phantomjs. And there is no such thing like phantomjs extensions. I need a solution that is purely based on the webdriver technology if possible. I am willing to program a separate solution for each browser, though.
I appreciate any idea about how to address this.

Solution is to use proxy. Webdriver integrates very well with browsermob proxy: http://bmp.lightbody.net/
private WebDriver initializeDriver() throws Exception {
// Start the server and get the selenium proxy object
ProxyServer server = new ProxyServer(proxy_port); // package net.lightbody.bmp.proxy
server.start();
server.setCaptureHeaders(true);
// Blacklist google analytics
server.blacklistRequests("https?://.*\\.google-analytics\\.com/.*", 410);
// Or whitelist what you need
server.whitelistRequests("https?://*.*.yoursite.com/.*. https://*.*.someOtherYourSite.*".split(","), 200);
Proxy proxy = server.seleniumProxy(); // Proxy is package org.openqa.selenium.Proxy
// configure it as a desired capability
DesiredCapabilities capabilities = new DesiredCapabilities();
capabilities.setCapability(CapabilityType.PROXY, proxy);
// start the driver ;
Webdriver driver = new FirefoxDriver(capabilities);
return driver;
}
EDIT:
people are often asking for http status codes, you can easily retrive them using the proxy. Code can be something like this:
// create a new har with given label
public void setHar(String label) {
server.newHar(label);
}
public void getHar() throws IOException {
// FIXME : What should be done with the this data?
Har har = server.getHar();
if (har == null) return;
File harFile = new File("C:\\localdev\\bla.har");
har.writeTo(harFile);
for (HarEntry entry : har.getLog().getEntries()) {
// Check for any 4XX and 5XX HTTP status codes
if ((String.valueOf(entry.getResponse().getStatus()).startsWith("4"))
|| (String.valueOf(entry.getResponse().getStatus()).startsWith("5"))) {
log.warn(String.format("%s %d %s", entry.getRequest().getUrl(), entry.getResponse().getStatus(),
entry.getResponse().getStatusText()));
//throw new UnsupportedOperationException("Not implemented");
}
}
}

You can chain the proxy, there isn't much documentation out there about doing so:
http://www.nerdnuts.com/2014/10/browsermob-behind-a-corporate-proxy/
We were able to use browsermob behind a corporate proxy using the following code:
// start the proxy
server = new ProxyServer(9090);
server.start();
server.setCaptureContent(true);
server.setCaptureHeaders(true);
server.addHeader(“accept-encoding”, “”);//turn off gzip
// Configure proxy server to use our network proxy
server.setLocalHost(InetAddress.getByName(“127.0.0.1″));
/**
* THIS IS THE MAJICK!
**/
HashMap<String, String> options = new HashMap<String, String>();
options.put(“httpProxy”, “172.20.4.115:8080″);
server.setOptions(options);
server.autoBasicAuthorization(“172.20.4.115″, “username”, “password”);
// get the Selenium proxy object
Proxy proxy = server.seleniumProxy();
DesiredCapabilities capabilities = DesiredCapabilities.phantomjs();
capabilities.setCapability(CapabilityType.PROXY, proxy);