Unable to run headless selenium on heroku with javascript enabled - java

I am trying to scrape some twitter data with headless selenium running in a Spring Boot application on Heroku. The following 2 buildpacks are included in my Heroku app:
1) https://github.com/heroku/heroku-buildpack-chromedriver.git
2) https://github.com/heroku/heroku-buildpack-google-chrome.git
These two buildpacks are designed for running headless selenium in a heroku app.
After the selenium script enters my twitter login data and hit the login button, my script keeps crashing. Why? I examined the page source and found the following hint from twitter:
We've detected that JavaScript is disabled in your browser. Would you
like to proceed to legacy Twitter?
So obviously javascript seems to be disabled which is very bad. And i don't want to proceed in legacy mode (without javascript).
When initializing the webdriver, i tried the following two options to make sure javascript is enabled. But it seems to be ignored:
Option 1)
ChromeOptions chromeOptions = new ChromeOptions();
chromeOptions.addArguments("--headless", "--enable-javascript");
WebDriver webdriver = ChromeDriver(chromeOptions);
Option 2)
ChromeOptions chromeOptions = new ChromeOptions();
chromeOptions.addArguments("--headless");
DesiredCapabilities capabilities = new DesiredCapabilities();
capabilities.setJavascriptEnabled(true);
capabilities.setCapability(ChromeOptions.CAPABILITY, chromeOptions);
WebDriver webdriver = ChromeDriver(capabilities);
Any ideas?

I had a similar problem. I solved it by removing some options.
This is my driver options (python) currently:
options = Options()
options.binary_location = os.getenv("GOOGLE_CHROME_BIN")
options.add_argument("--headless")
options.add_argument("--disable-gpu")
options.add_argument("--no-sandbox")
options.add_argument("--window-size=1920x1080")
driver = webdriver.Chrome(options=options, executable_path=os.getenv("CHROMEDRIVER_PATH"))

Related

Can I run Selenium using geckodriver without a webpage detecting marionette?

I want to use FirefoxDriver with Selenium but I keep getting detected by webpages. When I add the following code
System.setProperty("webdriver.gecko.driver", "../../../../../../../usr/bin/geckodriver");
FirefoxOptions opt = new FirefoxOptions();
opt.setCapability("marionette", false);
driver = new FirefoxDriver(opt);
The webpages cant't detect I'm using geckodriver but I can't use Selenium automation and that's my problem. I need the automation without detection.
QUESTIONS:
Can I change the setCapabilites on/off while the driver is running?
Is it easier to do this using ChromeDriver?
Using FirefoxDriver with Selenium but getting detected is quite common now as:
Selenium identifies itself
You can find a detailed discussion in How to make Selenium script undetectable using GeckoDriver and Firefox through Python?
Marionette
As per the documentation, Marionette, is the automation driver for Mozilla’s Gecko engine. It can remotely control the UI and the internal JavaScript of a Gecko platform, such as Mozilla Firefox. It can control both the chrome (i.e. menus and functions) or the content (the webpage loaded inside the browsing context), giving a high level of control and ability to replicate user actions. In addition to performing actions on the browser, Marionette can also read the properties and attributes of the DOM. Now, marionette shares much of the same API as Selenium/WebDriver, with additional commands to interact with Gecko’s chrome interface. Its goal is to replicate what Selenium does for web content, i.e. to enable the tester to have the ability to send commands to remotely control a user agent.
We have also discussed in details about Why Firefox requires GeckoDriver? within this thread
Finally, in the discussion Difference between webdriver.firefox.marionette & webdriver.gecko.driver we discussed about initializing Firefox sessions using legacy Firefox 47.x browsers and GeckoDriver enabled Firefox >47.x browsers. The conclusion was when using Firefox browsers > v77.x you have to mandatorily use GeckoDriver which extensively uses the marionette. So configuring marionette as false won't help us out. While using the latest version of geckodriver, selenium and firefox, you have to use the marionette by default.
If you still want to initialise a Firefox browsing session without using marionette you need to configure "marionette" to false as follows:
System.setProperty("webdriver.gecko.driver", "C://path//to//geckodriver.exe");
DesiredCapabilities dc = new DesiredCapabilities();
dc.setCapability("marionatte", false);
FirefoxOptions opt = new FirefoxOptions();
opt.merge(dc);
FirefoxDriver driver = new FirefoxDriver(opt);
driver.get("https://stackoverflow.com");
System.out.println("Application opened");
System.out.println("Page Title is : "+driver.getTitle());
driver.quit();
You can find a couple of relevant discussions in:
org.openqa.selenium.SessionNotCreatedException: Unable to find a matching set of capabilities while initiating Firefox v37 through Selenium v3.11.0
selenium.common.exceptions.SessionNotCreatedException: Message: Unable to find a matching set of capabilities with Firefox 46 through Selenium
How can Geckodriver/Firefox work without Marionette? (running python selenium 3 against FF 53)
The other questions:
Can I change the setCapabilites on/off while the driver is running?: The short answer is No, you can't change the capabilites while the session initiated by the webdriver is In Progress and you can find a couple of detailed discussions in:
Change ChromeOptions in an existing webdriver
Set capability on already running selenium webdriver
Is it easier to do this using ChromeDriver?: Again the precise answer is No, ChromeDriver also gets detected and you can find a couple of detailed discussions in:
Is there a version of selenium that is not detectable ? can selenium be truly undetectable?
Webpage Is Detecting Selenium Webdriver with Chromedriver as a bot
Outro
Here you can find a detailed discussion on Which Firefox browser versions supported for given Geckodriver version?

chrome extension - how to invoke webDriver with chrome extension (not in a new tab)

i am trying to open webDriver with chrome extension (Modify header value) using below code
ChromeOptions options = new ChromeOptions();
options.addExtensions(new File("/path/to/extension.crx"));
DesiredCapabilities capabilities = new DesiredCapabilities();
capabilities.setCapability(ChromeOptions.CAPABILITY, options);
ChromeDriver driver = new ChromeDriver(capabilities);
but this code is opening the webDriver instance with the extention in another window can any one know how to invoke it with chrome extension not in a new tab.
You may refer with this thread. Be noted that Selenium supports interaction with web view only. You can check this tutorial on how to test your Chrome Extension with selenium. However, you can also use of sikuli (Automation tool that make use of image recognition), to click the chrome add-on. After that add-on popup will be another browser window, so use switch window to perform actions on add-on popup. Hope this helps!

Install/Add chrome extension from Chrome Web Store using Selenium

I am developing test automation using Selenium with Java. I would like to install/add extension to chrome instance launched through Selenium. One way to install/add extension is using .crx file as follow.
ChromeOptions options = new ChromeOptions();
options.addExtensions(new File(ext_path));
DesiredCapabilities capabilities = DesiredCapabilities.chrome();
capabilities.setCapability(ChromeOptions.CAPABILITY, options);
driver = new ChromeDriver(capabilities);
But I would like to install/add extension from Chrome Web Store. Can anyone please suggest how to achieve this.
Thanks.
You can install extension (packed/unpacked) to chrome with options.addExtensions/addArguments as mentioned here:- https://sites.google.com/a/chromium.org/chromedriver/extensions
alternative: Chromedriver every time opens new chrome instance with temporary profile for the execution. To avoid this you can use existing profile with desired chrome extension installed OR you can send commands to already running instance of chrome which has desir d extension installed.
usually, inline/webstore installation of a Chrome extension can't be tested with ChromeDriver/Selenium alone, because of confirmation dialog.
you can use platform-specific hacks to click on the button (e.g. via AutoIt),

How to work with mobile chrome and testing app by Appium

I'm automationg mobile application, but in some cases I need to click on link and check that approproate link was opened by browser. For example link Terms of Service on SignUP page of my application.So the question is -HOW to switch between testing app and browser.Here is my code
File apk = new File("src/test/resources/myapp.apk");
DesiredCapabilities AppCapabilities = new DesiredCapabilities();
DesiredCapabilities chromeCapabilities=new DesiredCapabilities();
AppCapabilities.setCapability("app", apk.getAbsolutePath());
AppCapabilities.setCapability("platformName", "Android");
AppCapabilities.setCapability("automationName", "Appium");
AppCapabilities.setCapability("deviceName", "MyDevice");
AppCapabilities.setCapability("platformVersion", "4.3");
//capabilities.setCapability(key, value);
AppCapabilities.setCapability("newCommandTimeout", 400);
androidDriver = new AppiumDriver(new URL("http://127.0.0.1:4723/wd/hub"), AppCapabilities);
but something like adroidDriver.getCurrentURL() doesn't work.
You are missing a few capabilities.
Please look at the server flag and capabilities for anything related to "Chrome" http://appium.io/slate/en/master/?ruby#server-flags
And then please take a look at the documentation for automation using ChromeDriver with Appium.
http://appium.io/slate/en/master/?ruby#mobile-chrome-on-emulator-or-real-device
Solution found. Just add one more appium node and define additional driver for mobile browser

Accepting "Sharing Location" browser popups through Selenium WebDriver

I've Googled, searched SO, and looked at the Selenium Javadocs but haven't found an answer to this question.
Is there any way, using Selenium WebDriver, to accept those "Share Location" popups that get displayed by browsers (Chrome, Firefox, IE) when a website is trying to track your location?
Also, I'm writing up my Selenium code in Java and using the Selenium Client Drivers.
I've tried running the Selenium IDE (the Firefox add-on) and recording me accepting the "Share Location" popup that Firefox popsup but nothing gets recorded in the Selenium IDE.
If the popup is part of the browser then the answer is no. If you want to interact with something displayed in a second window ( not part of the browser) then yes. There should be a .switch_to method.
Although if you can have this page remember your settings you can load a profile when you create a new instance of the browser.
In windows you can create a profile in firefox by going to the start menu and typing "firefox.exe -p" here you can create a new profile where you you would have a "remember me check box" checked.
As far as loading a profile in Java + Selenium I'm not sure how you would do this. I use Ruby + Selenium and this is how I do it.
$driver = Selenium::WebDriver.for :firefox, :profile => "MyProfile"
hope this helps
I have got the solution to this in firefox but still working on chrome. The below code should help and fetch your exact location on maps:
ProfilesIni profile = new ProfilesIni();
FirefoxProfile myprofile = profile.getProfile("default");
myprofile.setPreference("geo.prompt.testing", true);
myprofile.setPreference("geo.prompt.testing.allow", true);
myprofile.setPreference("geo.wifi.uri", "https://location.services.mozilla.com/v1/geolocate?key=%MOZILLA_API_KEY%");
DesiredCapabilities capabilities = DesiredCapabilities.firefox();
capabilities.setCapability(FirefoxDriver.PROFILE,myprofile);
After this just pass the FirefoxProfile instance in the driver method just as below(for the above code):
driver = new FirefoxDriver(myprofile);
That should be fine, write all the other stuff you want and execute!!

Categories