We Keep Coding

How to Conceal WebDriver in Geckodriver from BotD in Java?

I followed this post on Stackoverflow to disable Firefox WebDriver detection.
Launch Geckodriver:
System.setProperty("webdriver.gecko.driver", geckdriverExecutableFilePath);
File firefoxProfileFile = new File(fullPathOfFirefoxInstallationFolder);
FirefoxProfile firefoxProfile = null;
try {
firefoxProfile = new FirefoxProfile(firefoxProfileFile);
} catch (Exception e) {
e.printStackTrace();
}
I disabled WebDriver:
WebDriver Disabled
FirefoxOptions firefoxOptions = new FirefoxOptions();
firefoxOptions.setProfile(firefoxProfile);
// Disables WebRTC
firefoxProfile.setPreference("media.peerconnection.enabled", false);
I disabled Automation Extensions:
Automation Extension Disabled
// Disables Automation Extension
firefoxProfile.setPreference("useAutomationExtension", false);
I added Proxy:
DesiredCapabilities dc = DesiredCapabilities.firefox();
Proxy proxy = new Proxy();
proxy.setHttpProxy(ipAddress + ":" + port);
proxy.setFtpProxy(ipAddress + ":" + port);
proxy.setSslProxy(ipAddress + ":" + port);
dc.setCapability(CapabilityType.PROXY, proxy);
firefoxOptions.merge(dc);
driver = new FirefoxDriver(firefoxOptions);
Yet BotD still detects my browser as being controlled by automation tool.
BotD Detection
How can I solve this?

When using Selenium driven GeckoDriver initiated firefox Browsing Context
The webdriver-active flag is set to true when the user agent is under remote control. It is initially false.
where, webdriver returns true if webdriver-active flag is set, false otherwise.
As:
navigator.webdriver Defines a standard way for co-operating user agents to inform the document that it is controlled by WebDriver, for
example so that alternate code paths can be triggered during
automation.
Further #whimboo in his comments confirmed:
This implementation have to be conformant to this requirement. As such
we will not provide a way to circumvent that.
Conclusion
So, the bottom line is:
Selenium identifies itself
and there is no way to conceal the fact that the browser is WebDriver driven.
Recommendations
However some pundits have suggested some different approaches which can conceal the fact that the Mozilla Firefox browser is WebDriver controled through the usage of Firefox Profiles and Proxies as follows:
selenium4 compatible python code
from selenium.webdriver import Firefox
from selenium import webdriver
from selenium.webdriver.firefox.service import Service
from selenium.webdriver.firefox.options import Options
profile_path = r'C:\Users\Admin\AppData\Roaming\Mozilla\Firefox\Profiles\s8543x41.default-release'
options=Options()
options.set_preference('profile', profile_path)
options.set_preference('network.proxy.type', 1)
options.set_preference('network.proxy.socks', '127.0.0.1')
options.set_preference('network.proxy.socks_port', 9050)
options.set_preference('network.proxy.socks_remote_dns', False)
service = Service('C:\\BrowserDrivers\\geckodriver.exe')
driver = Firefox(service=service, options=options)
driver.get("https://www.google.com")
driver.quit()
Potential Solution
A potential solution would be to use the tor browser as follows:
selenium4 compatible python code
from selenium.webdriver import Firefox
from selenium import webdriver
from selenium.webdriver.firefox.service import Service
from selenium.webdriver.firefox.options import Options
import os
torexe = os.popen(r'C:\Users\username\Desktop\Tor Browser\Browser\TorBrowser\Tor\tor.exe')
profile_path = r'C:\Users\username\Desktop\Tor Browser\Browser\TorBrowser\Data\Browser\profile.default'
firefox_options=Options()
firefox_options.set_preference('profile', profile_path)
firefox_options.set_preference('network.proxy.type', 1)
firefox_options.set_preference('network.proxy.socks', '127.0.0.1')
firefox_options.set_preference('network.proxy.socks_port', 9050)
firefox_options.set_preference("network.proxy.socks_remote_dns", False)
firefox_options.binary_location = r'C:\Users\username\Desktop\Tor Browser\Browser\firefox.exe'
service = Service('C:\\BrowserDrivers\\geckodriver.exe')
driver = webdriver.Firefox(service=service, options=firefox_options)
driver.get("https://www.tiktok.com/")
References
You can find a couple of relevant detailed discussions in
How to initiate a Tor Browser 9.5 which uses the default Firefox to 68.9.0esr using GeckoDriver and Selenium through Python
How to connect to Tor browser using Python
How to use Tor with Chrome browser through Selenium

BotD detects you because you do not override navigator.webdriver attribute.
I was able to override it with this code:
((JavascriptExecutor)driver).executeScript("Object.defineProperty(navigator, 'webdriver', {get: () => undefined})");
Re-run your code with this line after driver.get("BotD url") and click on
'Start detect' on the BotD page.
It will no longer show that webdriver is detected.
I understand you are looking for a way to make it work before the initial page load.
But here are 2 things to consider:
Webdriver developers want their tool to be detected by browsers.
Gecko driver developers are not going to implement an option to disable navigator.webdriver attribute. (This is the official reply from gecko developer.)

Disable popup "Restore Pages? Chrome didn't shut down correctly." in selenium webdriver Java

Though I have seen some related questions, I am putting this question again as I am still facing the issue.
I have a website that asks for login verification code on first time login and then it does not ask in further login instances. If I dont use a custom browser profile, the site asks verification code every time when selenium runs the login. Hence I have used customed browser profile as below (this is not the question)
options.addArguments("user-data-dir=C:\\Users\\user\\AppData\\Local\\Google\\Chrome\\User Data\\default");
Then, After the test is finished, I quit the browser using
driver.quit();
Problem is, next time I run the test again, the chrome browser opens with the popup "Restore Pages? Chrome didn't shut down correctly." and the site asks for verification code again after login. Entering verification code is not part my script, so the script fails. Then I close the browser manually by clicking on the X on the top corner.
Then I run the test again, chrome browser opens Normally without the popup and the site does not ask for verification code, login is successful. then, script closes browser itself by the driver.quit().
Then I run the test again, chrome browser opens with the popup and test fails again.
This shows that driver.quit() is not closing the browser correctly.
I have tried several settings, but still the popup is coming.
preferences file -> "exit_type":"none" (as well as "normal") and exit cleanly = true
site settings -> pop-ups and redirects -> Blocked (recommended)
I have also tried checking alerts exists or not, but it says "No alert". Script does not detect the popup as alert.
How can I make sure the browser is opened NORMALLY every next time after driver.quit() closes the browser in the previous test run? Please help.
Below is my code: I use chrome extension, so I cannot disable extension also.
String extFilePath = "C:\\ChromeDriver\\Salesforce_inspector.crx";
System.setProperty("webdriver.chrome.driver","C:\\chromedriver\\chromedriver.exe");
ChromeOptions options = new ChromeOptions();
options.addExtensions(new File(extFilePath));
options.addArguments("--disable-notifications");
options.addArguments("user-data-dir=C:\\Users\\user\\AppData\\Local\\Google\\Chrome\\User Data\\default");
options.addArguments("--start-maximized");
WebDriver driver = new ChromeDriver(options);
try
{
driver.switchTo().alert();
System.out.println("alert accepted");
} // try
catch (NoAlertPresentException Ex)
{
System.out.println("No alert");
}
driver.get("salesforce url");
I dont want to use incognito mode, because I need to retain the cookies and passwords to prevent the login verification code issue mentioned earlier.

I solved this issue setting the exit_type to normal in the preferences after set up the user dir.
Map<String, Object> prefs = new HashMap<String, Object>();
prefs.put("profile.exit_type", "Normal");
options.setExperimentalOption("prefs", prefs);
In your case will be something like this
String extFilePath = "C:\\ChromeDriver\\Salesforce_inspector.crx";
System.setProperty("webdriver.chrome.driver","C:\\chromedriver\\chromedriver.exe");
ChromeOptions options = new ChromeOptions();
options.addExtensions(new File(extFilePath));
options.addArguments("user-data-dir=C:\\Users\\user\\AppData\\Local\\Google\\Chrome\\User Data\\default");
options.addArguments("--start-maximized");
Map<String, Object> prefs = new HashMap<String, Object>();
prefs.put("profile.exit_type", "Normal");
options.setExperimentalOption("prefs", prefs);
WebDriver driver = new ChromeDriver(options);
try
{
driver.switchTo().alert();
System.out.println("alert accepted");
} // try
catch (NoAlertPresentException Ex)
{
System.out.println("No alert");
}
driver.get("salesforce url");

Chrome driver throwing error- Loading of unpacked extension is disabled by administrator

Yesterday my chrome version got updated to 76, I updated my chrome driver for the same- ChromeDriver 76.0.3809.12, I am facing the same error msg. I have used options.setExperimentalOption("useAutomationExtension", false); already, but still getting the same error every time before launching chrome. Please help as it is blocking my batch execution.
With last update of chrome same error was coming but go resolved with :
opts.setExperimentalOption("useAutomationExtension", false);
ChromeOptions opts = new ChromeOptions();
opts.addArguments("--disable-notifications");
opts.addArguments("no-sandbox");
opts.addArguments("--disable-extensions");
opts.setExperimentalOption("useAutomationExtension", false);
//opts.addArguments("start-maximized");
capabilities.setCapability(ChromeOptions.CAPABILITY, opts);
//webdriver = new ChromeDriver(opts);
webdriver = new ChromeDriver(capabilities);
Want to remove this warning pop up so that I can resume execution.

I got it solved by using chrome options instead of capabilities as below:
chrome_options = Options()
chrome_options.add_argument('-disable-extensions')
webdriver = webdriver.Chrome(options=chrome_options)
Have a look at one '-' instead of '--' in the argument. When I used '--disable-extensions' id didn't work. Maybe some side effect. The code is in python, but java variation could work as well.

Selenium Grid - Unable to find elements on the 2nd page in IE11

I have some selenium tests that work great in Chrome (locally and remotely) but when I run them remotely from my machine to a Windows 2012 VM they can't find any elements AFTER clicking a button to move to the next web page.
Here is a code snippet:
DesiredCapabilities caps = null;
caps = DesiredCapabilities.internetExplorer();
caps.setBrowserName("internet explorer");
caps.setCapability(InternetExplorerDriver.ENABLE_PERSISTENT_HOVERING, false);
caps.setCapability(InternetExplorerDriver.IE_ENSURE_CLEAN_SESSION, true);
caps.setCapability(CapabilityType.ACCEPT_SSL_CERTS, true);
caps.setCapability(InternetExplorerDriver.INTRODUCE_FLAKINESS_BY_IGNORING_SECURITY_DOMAINS, true);
caps.setJavascriptEnabled(true);
caps.setPlatform(Platform.WIN8);
String nodeURL = "http://0.0.0.0:5555/wd/hub"
driver = new RemoteWebDriver(new URL(nodeURL), caps);
// IE launches correctly, goes the cart page. Calculations are done on the pricing, and we click to move to the login page
driver.findElement(By.id("userName")).sendKeys("test#test.com");
// ^^ NoSuchElementException: Unable to find element with id == userName
So obviously there is a good connection made. Actions are performed and my tests move from the cart page to the login page. I even added a wait.until, hoping that it was a timing issue, but it timed out after 15 seconds. I'm watching the driver on RDP, it moved to the page. During the 15 seconds, I even opened the dev tools and checked to see if the element was there and it was.
What's going on? Thanks!

Preface: As a QA automation engineer, I hate IE.
It turns out that http://company.com was added a trusted site, but not https://company.com. Once I added that, it started working.
Did I mention I hate IE?

Error Loading Extension Could not load extension from 'C:\..\Local\Temp\scoped_dir6312_32763\internal'. Loading of unpacked extensions is disabled

When am running my webdriver script, am getting a confirmation dialog box with below message:
Error Loading Extension
Could not load extension from 'C:\Users\username\AppData\Local\Temp\scoped_dir6312_32763\internal'. Loading of unpacked extensions is disabled by the administrator.
Would you like to retry?
Yes No
Clicking "yes" lets the tests run.
I am not sure why am I getting this dialog box prompted,
I've tried the mentioned workarounds below but neither of them are working:
Replaced chrome driver with latest version.
Added below code in my script:
ChromeOptions options = new ChromeOptions();
options.addArguments("no-sandbox");
options.addArguments("disable-extensions");
driver = new ChromeDriver(options);
Below is my Test method:
public void Login() throws IOException{
test = extent.startTest("Login");
signInPage = new SignInPage(driver);
signInPage.enterMailId();
String screenShotPath = GetScreenShot.capture(driver, "enterMailId");
test.log(LogStatus.PASS, "Email id is entered successfully: " + test.addScreenCapture(screenShotPath));
signInPage.enterpwd();
//test.log(LogStatus.INFO, "Password is entered successfully");
screenShotPath = GetScreenShot.capture(driver, "enterpwd");
test.log(LogStatus.PASS, "Password is entered successfully: " + test.addScreenCapture(screenShotPath));
signInPage.clickOnLogin();
test.log(LogStatus.PASS, "User logged in successfully");
}
Below is the method which invoke the browser:
private void initChromeBrowser(){
System.setProperty("webdriver.chrome.driver", userdir +"\\chromedriver.exe");
ChromeOptions options = new ChromeOptions();
options.addArguments("test-type");
options.addArguments("no-sandbox");
//Fix for cannot get automation extension
options.addArguments("disable-extensions");
options.addArguments("start-maximized");
options.addArguments("--js-flags=--expose-gc");
options.addArguments("disable-plugins");
options.addArguments("--enable-precise-memory-info");
options.addArguments("--disable-popup-blocking");
options.addArguments("--disable-default-apps");
options.addArguments("test-type=browser");
options.addArguments("disable-infobars");
driver = new ChromeDriver(options);
launchApp();
}
Could there be anything else that I should incorporate in my script to prevent the dialog box.

You can set the useAutomationExtension capability to false.
ChromeOptions options = new ChromeOptions();
options.setExperimentalOption("useAutomationExtension", false);
WebDriver driver = new ChromeDriver(options);
This capability will help to not load Chrome Automation extension. Due to which, "Failed to load extension" popup would not appear.
But please note you will not be able to perform any window resizing/positioning operations without the Chrome automation extension.
Hope this helps!
Source : https://bugs.chromium.org/p/chromedriver/issues/detail?id=1749

This error message...
Error Loading Extension
Could not load extension from 'C:\Users\username\AppData\Local\Temp\scoped_dir6312_32763\internal'. Loading of unpacked extensions is disabled by the administrator.
Would you like to retry?
Yes No
...implies that an extension was not been loaded as it was disabled by the administrator.
As per Issue 1749: Failed to load extention from: ... Loading of unpacked extensions is disabled by the administrator ChromeDriver uses Chrome automation extension for automating various functions like window sizing, window positioning, etc.
The Failed to load extension.. popup means that this extension has not been loaded. If you manually close the popup, browser will act normally and ChromeDriver commands will continue to work as expected. But in this case if you try executing window resizing or window re-positioning commands, it will throw an error as unknown error: cannot get automation extension.
Till ChromeDriver v2.28 whenever an organizations admin policy forbidden extensions, to bypass the restriction users have used the argument disable-extensions as follows:
ChromeOptions options = new ChromeOptions();
options.addArguments("--disable-extensions");
WebDriver driver = new ChromeDriver(options);
and it worked perfecto.
ChromeDriver v2.28 onwards, whenever disable-extensions flag is passed by test, ChromeDriver implicitly passes disable-extensions-except flag which in turn loads Chrome automation extension. This extension helps Chromedriver to perform window sizing and window re-positioning operations.
So, if your organizational admin policy blocks extensions, display of popup Failed to load extension from: ... Loading of unpacked extensions is an expected behavior.
This issue had a dependency on Selenium support for headless.
As an alternative, you can set the useAutomationExtension capability to false as follows:
ChromeOptions options = new ChromeOptions();
options.setExperimentalOption("useAutomationExtension", false);
WebDriver driver = new ChromeDriver(options);
This capability inturn will help to not load Chrome Automation extension and Failed to load extension popup would not appear. But you will not be able to perform any window resizing/positioning operations without the Chrome automation extension.
Now, Selenium support for headless being resolved ChromeDriver will no longer require this extension and you shouldn't have seen this error/popup.
Solution
The simplest solution would be to use the latest version of ChromeDriver and Chrome combination among either of the following:
If you are using Chrome version 73, please download ChromeDriver 73.0.3683.20
If you are using Chrome version 72, please download ChromeDriver 2.46 or ChromeDriver 72.0.3626.69
If you are using Chrome version 71, please download ChromeDriver 2.46 or ChromeDriver 71.0.3578.137
For older version of Chrome, please see this discussion.
Alternative
Some other alternatives are:
Add the Registry Key ExtensionInstallWhitelist to whitelist
Remove the Registry Key ExtensionInstallBlacklist containing a string key 1 with value *

I encountered this same issue after upgrading to ChromeDriver v2.29. I have Chrome v58.0. It looks like an open issue: https://bugs.chromium.org/p/chromedriver/issues/detail?id=639#c26
Depending on your versions, YMMV, in my case, I had to downgrade to ChromeDriver v2.27.

If you go to chrome://version/ you can see under the Command:
C:\Program Files (x86)\Google\Chrome\Application\chrome.exe" --disable-background-networking --disable-client-side-phishing-detection --disable-default-apps --disable-extensions --disable-extensions-except="C:\Users\Inno3\AppData\Local\Temp\scoped_dir80288_6333\internal" --disable-hang-monitor --disable-popup-blocking --disable-prompt-on-repost --disable-sync --disable-web-resources --enable-automation --enable-logging --force-fieldtrials=SiteIsolationExtensions/Control --ignore-certificate-errors --log-level=0 --metrics-recording-only --no-first-run --password-store=basic --remote-debugging-port=12354 --safebrowsing-disable-auto-update --start-maximized --test-type=webdriver --use-mock-keychain --user-data-dir="C:\Users\Inno3\AppData\Local\Temp\scoped_dir80288_30914" --flag-switches-begin --flag-switches-end data:,
This is why it throw error, I don't know why it give error, maybe user policy or Chrome updates?
--disable-extensions-except="C:\Users\Inno3\AppData\Local\Temp\scoped_dir80288_6333\internal"
I believe the argument is added by Selenium, you need the following command to tell selenium to not add it.
In C#:
chromeOptions = OpenQA.Selenium.Chrome.ChromeOptions();
chromeOptions.AddAdditionalCapability("useAutomationExtension", false);
river = new ChromeDriver(chromeOptions);
launchApp();
In Java:
chromeOptions.AddAdditionalCapability("useAutomationExtension", false)

//Set the system property for chrome browser location
System.setProperty("webdriver.chrome.driver", Global.sChromeDriverPath);
//Set the Chrome capabilities
ChromeOptions options = new ChromeOptions();
options.addArguments("test-type");
options.addArguments("start-maximized");
options.addArguments("--js-flags=--expose-gc");
options.addArguments("--enable-precise-memory-info");
options.addArguments("--disable-popup-blocking");
options.addArguments("--disable-default-apps");
options.addArguments("--enable-automation");
options.addArguments("test-type=browser");
options.addArguments("disable-infobars");
options.addArguments("disable-extensions");
options.setExperimentalOption("useAutomationExtension", false);
Global.driver = new ChromeDriver(options);

Below code is working fine for me chrome driver 2.41 and browser version 68.0.3440.84
public class patCheck {
WebDriver driver;
#Test
public void f() {
System.setProperty("webdriver.chrome.driver", "C:\\Users\\shirish.nagar\\Work\\Selenium\\Web\\Drivers\\chromedriver.exe");
ChromeOptions options = new ChromeOptions();
options.setExperimentalOption("useAutomationExtension", false);
driver = new ChromeDriver(options);
driver.manage().window().maximize();
driver.get("https://www.google.com");
}
}
It successfully invokes the chrome browser without any pop-up of "loading unpacked extension disabled by administrator"

ChromeOptions options = new ChromeOptions();
System.setProperty("webdriver.chrome.driver", "C:\\drivers\\chromedriver.exe");
options.setExperimentalOption("useAutomationExtension", false);
driver = new ChromeDriver(options);

On my company we have a GPO that block all extensions on chrome with the ExtensionInstallBacklist.
So to avoid this we change the blocked extentensions registry key from * (all) to a random value (foobar).
To do that you can create a .reg file with this content:
Windows Registry Editor Version 5.00
[HKEY_LOCAL_MACHINE\SOFTWARE\Policies\Google\Chrome\ExtensionInstallBlacklist]
"1"="lala"

I tried a bunch of things like removing the * entry of Chrome blacklist in Windows registry (which is a painful hack because it will be reversed back couple times a week by the company group policy). I finally came up with the working solution. With the following code, the "error loading extension" pop up isn't showing up any more.
${options}= Evaluate sys.modules['selenium.webdriver'].ChromeOptions() sys, selenium.webdriver
Call Method ${options} add_argument --start-maximized
Call Method ${options} add_experimental_option useAutomationExtension ${False}
Create WebDriver Chrome chrome_options=${options}

Below code works for me:
Set useAutomationExtension as false
options.setExperimentalOption("useAutomationExtension", false);
Fulll code:
System.setProperty("webdriver.chrome.driver", "C:\\Selenium Drivers\\chromedriver.exe");
ChromeOptions options = new ChromeOptions();
DesiredCapabilities capabilities = DesiredCapabilities.chrome();
capabilities.setAcceptInsecureCerts(true);
options.merge(capabilities);
options.addArguments("--test-type");
options.addArguments("start-maximized");
options.addArguments("--js-flags=--expose-gc");
options.addArguments("--enable-precise-memory-info");
options.addArguments("--disable-popup-blocking");
options.addArguments("--disable-default-apps");
options.addArguments("--enable-automation");
options.addArguments("disable-extensions");
options.setExperimentalOption("useAutomationExtension", false);
options.addArguments("--disable-dev-shm-usage");
options.addArguments("--no-sandbox");
options.addArguments("disable-infobars");
driver = new ChromeDriver(options);
driver.get("https://www.google.com/");

This below code works for me with the addition of - options.setExperimentalOption("useAutomationExtension", false) :
...
System.setProperty("webdriver.chrome.driver", "chromedriver path");
ChromeOptions options = new ChromeOptions();
options.addArguments("test-type");
options.addArguments("start-maximized");
options.addArguments("--js-flags=--expose-gc");
options.addArguments("--enable-precise-memory-info");
options.addArguments("--disable-popup-blocking");
options.addArguments("--disable-default-apps");
options.addArguments("--enable-automation");
options.addArguments("test-type=browser");
options.addArguments("disable-infobars");
options.addArguments("disable-extensions");
options.setExperimentalOption("useAutomationExtension", false);
driver.new ChromeDriver(options);
driver.get(ur url);