So I am looking to do some simple data collection on a few .onion sites. I am going about this by using selenium webdriver to call Tor as part of the Firefox webdriver. However, I can't seem to figure out how to get firefox to successfully go to .onion sites. Here is the code.
public static void main(String[] args) throws InterruptedException, IOException {
File torProfileDir = new File("C:\\Users\\Chambers\\Desktop\\Tor Browser\\Browser\\TorBrowser\\Data\\Browser\\profile.default");
FirefoxBinary binary = new FirefoxBinary(new File("C:\\Users\\Chambers\\Desktop\\Tor Browser\\Browser\\firefox.exe"));
FirefoxProfile torProfile = new FirefoxProfile(torProfileDir);
torProfile.setPreference("webdriver.load.strategy", "unstable");
try {
binary.startProfile(torProfile, torProfileDir, "");
} catch (IOException e) {
e.printStackTrace();
}
FirefoxProfile profile = new FirefoxProfile();
profile.setPreference("network.proxy.type", 1);
profile.setPreference("network.proxy.socks", "127.0.0.1");
profile.setPreference("network.proxy.socks_port", 9150);
FirefoxDriver driver = new FirefoxDriver(profile);
int firstCheck = "0";
while (firstCheck == 0) {
driver.navigate().to("onion site here");
......
The problem I am having is that I end up with a firefox browser that cannot connect to .onion sites. If I change FirefoxDriver driver = new FirefoxDriver(profile); to FirefoxDriver driver = new FirefoxDriver(binary, profile); then I am left with a blank Tor window that I can't seem to control with the webdriver.
Anyone have any ideas on how to fix this? any help would be appreciated!
Figured it out. Needed to add a lot of permissions to the new profile. Here is the fixed code for those that are interested. It allows you to browse the dark web with Firefox as if you were using Tor. The commands for controlling the webdriver don't change.
File torProfileDir = new File("C:\\Users\\Chambers\\Desktop\\Tor Browser\\Browser\\TorBrowser\\Data\\Browser\\profile.default");
FirefoxBinary binary = new FirefoxBinary(new File("C:\\Users\\Chambers\\Desktop\\Tor Browser\\Browser\\firefox.exe"));
FirefoxProfile torProfile = new FirefoxProfile(torProfileDir);
torProfile.setPreference("webdriver.load.strategy", "unstable");
try {
binary.startProfile(torProfile, torProfileDir, "");
} catch (IOException e) {
e.printStackTrace();
}
FirefoxProfile profile = new FirefoxProfile();
profile.setPreference("network.proxy.type", 1);
profile.setPreference("network.proxy.socks", "127.0.0.1");
profile.setPreference("network.proxy.socks_port", 9150);
profile.setPreference("network.proxy.socks_version", 5);
profile.setPreference("places.history.enabled", false);
profile.setPreference("privacy.clearOnShutdown.offlineApps", true);
profile.setPreference("privacy.clearOnShutdown.passwords", true);
profile.setPreference("privacy.clearOnShutdown.siteSettings", true);
profile.setPreference("privacy.sanitize.sanitizeOnShutdown", true);
profile.setPreference("signon.rememberSignons", false);
profile.setPreference("network.cookie.lifetimePolicy", 2);
profile.setPreference("network.dns.disablePrefetch", true);
profile.setPreference("network.http.sendRefererHeader", 0);
profile.setPreference("network.proxy.socks_remote_dns", true);
FirefoxDriver driver = new FirefoxDriver(profile);
String firstCheck = "";
while (firstCheck == 0) {
driver.get("http://kbhpodhnfxl3clb4.onion/");
.........................
Related
I want to automate the web application on all browsers.My code is running fine for chrome,firefox but when i tried it on Edge,it showing the ssl certificate error.
how i can handle this.tried so many workarounds but failed to achieve it.
if((browser.equalsIgnoreCase("Edge"))){
//browserName = "";
browser = "Edge";
//set path to Edge.exe
System.setProperty("webdriver.edge.driver","C:\\edgedriver.exe");
if(enableProxy == true) {
proxy = new ProxyServer();
proxy.setTrustAllServers(true);
proxy.start();
System.out.println( proxy.getPort());
Proxy seleniumProxy = new Proxy();
EdgeOptions options = new EdgeOptions();
String hostIp = Inet4Address.getLocalHost().getHostAddress();
seleniumProxy.setHttpProxy(hostIp + ":" + proxy.getPort());
seleniumProxy.setSslProxy(hostIp + ":" + proxy.getPort());
seleniumProxy = ClientUtil.createSeleniumProxy(proxy);
options.setProxy(seleniumProxy);
options.setCapability(CapabilityType.ACCEPT_SSL_CERTS, true);
// DesiredCapabilities capabilities = new DesiredCapabilities();
// capabilities.setCapability(CapabilityType.PROXY, proxy);
proxy.enableHarCaptureTypes(CaptureType.REQUEST_CONTENT, CaptureType.RESPONSE_CONTENT);
proxy.newHar();
driver = new EdgeDriver(EdgeDriverService.createDefaultService(),options);
//driver = new EdgeDriver(capabilities);
driver.manage().window().maximize();
}else {
//create Edge instance
driver = new EdgeDriver();
driver.manage().window().maximize();
}
}
You can set ACCEPT_SSL_CERTS to true in DesiredCapabilities:
DesiredCapabilities capabilities = new DesiredCapabilities();
capabilities.setCapability(CapabilityType.ACCEPT_SSL_CERTS, true);
System.setProperty("webdriver.edge.driver", "C://EdgeDriver.exe");
WebDriver driver = new EdgeDriver(capabilities);
I need to get headless chrome to ignore certificate errors. The option is ignored when running in headless mode, and the driver returns empty html body tags when navigating to an https resource.
<html xmlns="http://www.w3.org/1999/xhtml"><head></head><body></body></html>
This is how I am configuring my chrome driver.
ChromeOptions chromeOptions = new ChromeOptions();
chromeOptions.addArguments("--headless", "--disable-gpu", "--window-size=1920,1200","--ignore-certificate-errors");
DesiredCapabilities cap=DesiredCapabilities.chrome();
cap.setCapability(ChromeOptions.CAPABILITY, chromeOptions);
cap.setCapability(CapabilityType.ACCEPT_SSL_CERTS, true);
cap.setCapability(CapabilityType.ACCEPT_INSECURE_CERTS, true);
chromeHeadlessDriver = new ChromeDriver(cap);
This thread confirms that --ignore-certificate-errors is ignored in headless mode.
They mention about devtool protocol.
Is it something I can invoke from java? Are there any other alternatives?
There is an excellent article on medium.com by sahajamit
and i have tested the below code, it works perfectly fine with self-signed certificate https://badssl.com/
ChromeOptions options = new ChromeOptions();
options.setExperimentalOption("useAutomationExtension", false);
options.addArguments("--headless", "--window-size=1920,1200","--ignore-certificate-errors");
DesiredCapabilities crcapabilities = DesiredCapabilities.chrome();
crcapabilities.setCapability(ChromeOptions.CAPABILITY, options);
crcapabilities.setCapability(CapabilityType.ACCEPT_SSL_CERTS, true);
crcapabilities.setCapability(CapabilityType.ACCEPT_INSECURE_CERTS, true);
System.setProperty(ChromeDriverService.CHROME_DRIVER_LOG_PROPERTY, "C:\\temp\\chrome\\chromedriver.log");
System.setProperty(ChromeDriverService.CHROME_DRIVER_EXE_PROPERTY, "C:\\temp\\chrome\\chromedriver.exe");
ChromeDriverService service = null;
try {
service = new ChromeDriverService.Builder()
.usingAnyFreePort()
.withVerbose(true)
.build();
service.start();
} catch (IOException e) {
e.printStackTrace();
}
RemoteWebDriver driver = new RemoteWebDriver(service.getUrl(),crcapabilities);
driver.get("https://self-signed.badssl.com/");
System.out.println(driver.getPageSource());
driver.quit();
software/framework versions
Google Chrome Version 64.0.3282.186
Google Chrome Driver Version 64.0.3282.186
Selenium version 3.11.0
This works for me on ChromeDriver 80.
ChromeOptions option = new ChromeOptions();
option.AddArgument("--headless");
option.AcceptInsecureCertificates = true;
driver = new ChromeDriver(option);
#amila-kumara is working but, usage of DesiredCapabilities.chrome() gives warning to use ChromeOptions. Please see the updated answer.
Set the chrome option values
System.setProperty("webdriver.chrome.driver", Config.NDAC_WEBDRIVER_PATH);
ChromeOptions options = new ChromeOptions();
options.addArguments("--window-size=1920,1200");
options.setAcceptInsecureCerts(true);
options.setHeadless(true);
Start the service
ChromeDriverService service = null;
try {
service = new ChromeDriverService.Builder()
.usingAnyFreePort()
.withVerbose(true)
.build();
service.start();
remoteWebdriverUrl = service.getUrl();
System.out.println("Starting the url " + remoteWebdriverUrl);
} catch (IOException e) {
e.printStackTrace();
}
Note: I was facing the issue while closing the driver(with RemoteWebDriver), chromedriver.exe process won't close even when you use driver.quit(). To fix the issue use ChromeDriver instead of RemoteWebDriver
RemoteWebDriver driver = new ChromeDriver(service, options);
To properly close the driver, use
driver.close();
driver.quit();
I can't connect my BrowserFactory to Selenium Grid. Any ideas why the following code won't work?
public static WebDriver getDriver() throws Exception {
try {
// Load the driver selected by user
Properties p = new Properties();
FileInputStream fi = new FileInputStream(Constant.CONFIG_PROPERTIES_DIRECTORY);
p.load(fi);
if(p.getProperty("use_grid").equalsIgnoreCase("true")) {
DesiredCapabilities desiredCapabilities = new DesiredCapabilities();
desiredCapabilities.getBrowserName();
desiredCapabilities.setPlatform(Platform.WINDOWS);
return new RemoteWebDriver(new URL("http://localhost:4444/wd/hub"), desiredCapabilities);
}
String browserName = p.getProperty("browser");
switch (browserName) {
case "firefox":
if (null == webdriver) {
System.setProperty("webdriver.gecko.driver", Constant.GECKO_DRIVER_DIRECTORY);
webdriver = new FirefoxDriver();
}
break;
I have the node and grid up and running successfully.
Thanks.
It looks like you're not setting the browser when using selenium grid. Try changing this line:
desiredCapabilities.getBrowserName();
to this:
desiredCapabilities.setBrowserName("firefox");
or this if you're properties are in the correct format and that code is working correctly:
desiredCapabilities.setBrowserName(p.getProperty("browser"));
public static void main(String[] args)
{
try
{
WebDriver driver = new FirefoxDriver(getFProfile());
driver.get("http://www.energy.umich.edu/sites/default/files/pdf-sample.pdf");
WebDriver driver1 = new FirefoxDriver(getFProfile());
driver1.get("http://samplecsvs.s3.amazonaws.com/SalesJan2009.csv");
}
catch (Exception e)
{
e.printStackTrace();
}
}
public static FirefoxProfile getFProfile()
{
FirefoxProfile firefoxProfile = new FirefoxProfile();
firefoxProfile.setPreference("browser.download.folderList", 2);
firefoxProfile.setPreference("browser.download.manager.showWhenStarting", false);
firefoxProfile.setPreference("browser.download.dir", "${user.home}\\Downloads"); //C:\\download
//For PDF
firefoxProfile.setPreference("browser.helperApps.neverAsk.saveToDisk", "application/pdf");
//For CSV
firefoxProfile.setPreference("browser.helperApps.neverAsk.saveToDisk", "text/csv");
firefoxProfile.setPreference("pdfjs.disabled", true);
firefoxProfile.setPreference("plugin.scan.Acrobat", "99.0");
firefoxProfile.setPreference("plugin.scan.plid.all", false);
return firefoxProfile;
}
Above code is working for .pdf files only but for .csv prompt was displayed.
How can we auto save both using single profile preference setting.
please help me on this.
Use the below modified code for single profile:
firefoxProfile.setPreference("browser.helperApps.neverAsk.saveToDisk", "application/pdf, text/csv");
hope this helps !!
I'm a rookie test developer using selenium 2.45 and I'm trying to configure my FirefoxDriver to use my company's proxy settings. I am failing to do so :)
I am following the instruction from here to create a profile on-the-fly:
Using a Proxy for FF
My code looks like this:
public static WebDriver driver;
String usedProxy = "http://myproxy:8080";
org.openqa.selenium.Proxy proxy = new org.openqa.selenium.Proxy();
proxy.setHttpProxy(usedProxy).setFtpProxy(usedProxy).setSslProxy(usedProxy);
DesiredCapabilities cap = new DesiredCapabilities();
cap.setCapability(CapabilityType.PROXY, proxy);
driver = new FirefoxDriver(cap);
driver.manage().timeouts().implicitlyWait(10, TimeUnit.SECONDS);
driver.get("https://TestWebsite.com");
I am not receiving any kind of errors but the connection is not working for this browser. When checking Options>Advanced>Network>Connection Settings from Firefox menu, proxy is set to manual but text inputs only contain "http://"
PS: I have a feeling this is relevant but am not sure: TestWebsite.com will load via https only (it's a shopping cart)
If your goal is to test your functionality on different IP addresses, you can use the Tor Browser.
public IWebDriver Driver { get; set; }
public Process TorProcess { get; set; }
public WebDriverWait Wait { get; set; }
[TestInitialize]
public void SetupTest()
{
String torBinaryPath = #"C:\Users\aangelov\Desktop\Tor Browser\Browser\firefox.exe";
this.TorProcess = new Process();
this.TorProcess.StartInfo.FileName = torBinaryPath;
this.TorProcess.StartInfo.Arguments = "-n";
this.TorProcess.StartInfo.WindowStyle = ProcessWindowStyle.Maximized;
this.TorProcess.Start();
FirefoxProfile profile = new FirefoxProfile();
profile.SetPreference("network.proxy.type", 1);
profile.SetPreference("network.proxy.socks", "127.0.0.1");
profile.SetPreference("network.proxy.socks_port", 9150);
this.Driver = new FirefoxDriver(profile);
this.Wait = new WebDriverWait(this.Driver, TimeSpan.FromSeconds(60));
}
[TestCleanup]
public void TeardownTest()
{
this.Driver.Quit();
this.TorProcess.Kill();
}
Here is the code to refresh the Tor identity.
public void RefreshTorIdentity()
{
Socket server = null;
try
{
IPEndPoint ip = new IPEndPoint(IPAddress.Parse("127.0.0.1"), 9151);
server = new Socket(AddressFamily.InterNetwork, SocketType.Stream, ProtocolType.Tcp);
server.Connect(ip);
server.Send(Encoding.ASCII.GetBytes("AUTHENTICATE \"johnsmith\"" + Environment.NewLine));
byte[] data = new byte[1024];
int receivedDataLength = server.Receive(data);
string stringData = Encoding.ASCII.GetString(data, 0, receivedDataLength);
server.Send(Encoding.ASCII.GetBytes("SIGNAL NEWNYM" + Environment.NewLine));
data = new byte[1024];
receivedDataLength = server.Receive(data);
stringData = Encoding.ASCII.GetString(data, 0, receivedDataLength);
if (!stringData.Contains("250"))
{
Console.WriteLine("Unable to signal new user to server.");
server.Shutdown(SocketShutdown.Both);
server.Close();
}
}
finally
{
server.Close();
}
}
You can find more detailed information here: http://automatetheplanet.com/using-selenium-webdriver-tor-c-code/
The code examples are in C# but the code should be identical in Java.
Try firefox profile as well. Note this is C# code and converting to Java should be fairly simply
string usedProxy = "http://myproxy:8080";
Proxy proxy = new OpenQA.Selenium.Proxy();
proxy.HttpProxy = usedProxy;
proxy.FtpProxy = usedProxy;
proxy.SslProxy = usedProxy;
FirefoxProfile profile = new FirefoxProfile();
profile.SetProxyPreferences(proxy);
This code works for me:
FirefoxOptions options = new FirefoxOptions();
FirefoxProfile profile = new FirefoxProfile();
profile.setPreference("network.proxy.type", 1);
profile.setPreference("network.proxy.http", <proxy_url>);
profile.setPreference("network.proxy.http_port", 8080);
profile.setPreference("network.proxy.ssl", <proxy_url>);
profile.setPreference("network.proxy.ssl_port", 8080);
options.setProfile(profile);
options.setAcceptInsecureCerts(true)
options.setCapability("disable-restore-session-state", true);
options.setCapability("marionette", true);
WebDriver driver = new FirefoxDriver(options);
I followed answers from this link: Webdriver and proxy server for firefox