I am trying to get data from a webpage (http://steamcommunity.com/id/Winning117/games/?tab=all) using a specific tag but I keep getting null. My desired result is to get the "hours played" for a specific game - Cluckles' Adventure in this case.
import org.jsoup.Jsoup;
import org.jsoup.nodes.Document;
import org.jsoup.nodes.Element;
public class TestScrape {
public static void main(String[] args) throws Exception {
String url = "http://steamcommunity.com/id/Winning117/games/?tab=all";
Document document = Jsoup.connect(url).get();
Element playTime = document.select("div#game_605250").first();
System.out.println(playTime);
}
}
Edit: How can I tell if a webpage is using JavaScript and is therefore unable to be parsed by Jsoup?
To execute javascript in java code there is Selenium :
Selenium-WebDriver makes direct calls to the browser using each
browser’s native support for automation.
To include it with maven use this dependency:
<dependency>
<groupId>org.seleniumhq.selenium</groupId>
<artifactId>selenium-server</artifactId>
<version>3.4.0</version>
</dependency>
Next I give you code of simple JUnit test that creates instance of WebDriver and goes to given url and executes simple script to get rgGames .
File chromedriver you have to download at https://sites.google.com/a/chromium.org/chromedriver/downloads.
package SeleniumProject.selenium;
import java.io.File;
import java.io.IOException;
import java.util.ArrayList;
import java.util.Map;
import org.junit.After;
import org.junit.AfterClass;
import org.junit.Before;
import org.junit.BeforeClass;
import org.junit.Test;
import org.junit.runner.RunWith;
import org.junit.runners.JUnit4;
import org.openqa.selenium.By;
import org.openqa.selenium.JavascriptExecutor;
import org.openqa.selenium.WebDriver;
import org.openqa.selenium.WebElement;
import org.openqa.selenium.chrome.ChromeDriverService;
import org.openqa.selenium.chrome.ChromeOptions;
import org.openqa.selenium.remote.DesiredCapabilities;
import org.openqa.selenium.remote.RemoteWebDriver;
import org.openqa.selenium.support.ui.ExpectedCondition;
import org.openqa.selenium.support.ui.WebDriverWait;
import junit.framework.TestCase;
#RunWith(JUnit4.class)
public class ChromeTest extends TestCase {
private static ChromeDriverService service;
private WebDriver driver;
#BeforeClass
public static void createAndStartService() {
service = new ChromeDriverService.Builder()
.usingDriverExecutable(new File("D:\\Downloads\\chromedriver_win32\\chromedriver.exe"))
.withVerbose(false).usingAnyFreePort().build();
try {
service.start();
} catch (IOException e) {
System.out.println("service didn't start");
// TODO Auto-generated catch block
e.printStackTrace();
}
}
#AfterClass
public static void createAndStopService() {
service.stop();
}
#Before
public void createDriver() {
ChromeOptions chromeOptions = new ChromeOptions();
DesiredCapabilities capabilities = DesiredCapabilities.chrome();
capabilities.setCapability(ChromeOptions.CAPABILITY, chromeOptions);
driver = new RemoteWebDriver(service.getUrl(), capabilities);
}
#After
public void quitDriver() {
driver.quit();
}
#Test
public void testJS() {
JavascriptExecutor js = (JavascriptExecutor) driver;
// Load a new web page in the current browser window.
driver.get("http://steamcommunity.com/id/Winning117/games/?tab=all");
// Executes JavaScript in the context of the currently selected frame or
// window.
ArrayList<Map> list = (ArrayList<Map>) js.executeScript("return rgGames;");
// Map represent properties for one game
for (Map map : list) {
for (Object key : map.keySet()) {
// take each key to find key "name" and compare its vale to
// Cluckles' Adventure
if (key instanceof String && key.equals("name") && map.get(key).equals("Cluckles' Adventure")) {
// print all properties for game Cluckles' Adventure
map.forEach((key1, value) -> {
System.out.println(key1 + " : " + value);
});
}
}
}
}
}
As you can see selenium loads page at
driver.get("http://steamcommunity.com/id/Winning117/games/?tab=all");
And to get data of all games by Winning117 it returns rgGames variable:
ArrayList<Map> list = (ArrayList<Map>) js.executeScript("return rgGames;");
The page you want to scrape is load by js,and there is not any #game_605250 element that jsoup get.All datas are write in page by using js.
But when I print document to a file ,I see some data like this:
<script language="javascript">
var rgGames = [{"appid":224260,"name":"No More Room in Hell","logo":"http:\/\/cdn.steamstatic.com.8686c.com\/steamcommunity\/public\/images\/apps\/224260\/670e9aba35dc53a6eb2bc686d302d357a4939489.jpg","friendlyURL":224260,"availStatLinks":{"achievements":true,"global_achievements":true,"stats":false,"leaderboards":false,"global_leaderboards":false},"hours_forever":"515","last_played":1492042097},{"appid":241540,"name":"State of Decay","logo":"http:\/\/....
then,you can extract 'rgGames' by some StringTools and format it to json obj.
It't not a clerver method,but it worked
try this :
public class TestScrape {
public static void main(String[] args) throws Exception {
String url = "http://steamcommunity.com/id/Winning117/games/?tab=all";
Document document = Jsoup.connect(url).get();
Element playTime = document.select("div#game_605250");
Elements val = playTime.select(".hours_played");
System.out.println(val.text());
}
}
Related
I use a Maven project:
<dependency>
<groupId>net.lightbody.bmp</groupId>
<artifactId>browsermob-core</artifactId>
<version>2.1.5</version>
</dependency>
I want go to html site with different user (by headers).
example:
1: get home page with header user:foo
2: get home page with header user:bar
3: get home page with header user:goo
I try with this code but my chrome is open with a white page only:
Here is my code
import org.junit.After;
import org.junit.Assert;
import org.junit.Before;
import org.junit.Test;
import org.openqa.selenium.By;
import org.openqa.selenium.Proxy;
import org.openqa.selenium.WebDriver;
import org.openqa.selenium.chrome.ChromeDriver;
import org.openqa.selenium.chrome.ChromeOptions;
import org.openqa.selenium.remote.CapabilityType;
import net.lightbody.bmp.BrowserMobProxy;
import net.lightbody.bmp.BrowserMobProxyServer;
import net.lightbody.bmp.client.ClientUtil;
public class Sof {
WebDriver driver;
#Before
public void setUp() {
// start the proxy
BrowserMobProxy proxy = new BrowserMobProxyServer();
// put our custom header to each request
proxy.addRequestFilter((request, contents, messageInfo) -> {
request.headers().add("my-test-header", "my-test-value");
System.out.println("addRequestFilter: " + request.headers().entries().toString());
return null;
});
proxy.start();
// get the Selenium proxy object
Proxy seleniumProxy = ClientUtil.createSeleniumProxy(proxy);
// Setting up Proxy for chrome
ChromeOptions opts = new ChromeOptions();
opts.setCapability(CapabilityType.PROXY, seleniumProxy);
System.setProperty("webdriver.chrome.driver", "robot\\src\\test\\resources\\drivers\\windows\\googlechrome\\64bit\\chromedriver.exe");
driver = new ChromeDriver(opts);
}
#Test
public void testProxifying() {
driver.get("https://noraui.github.io/demo/logogame/v1/");
Assert.assertEquals(driver.findElement(By.xpath("//body")).getText(), "{\"SUCCESS\"}");
}
#After
public void tearDown() {
if (driver != null) {
driver.quit();
System.out.println("Driver was instantiated. Quitting..");
} else {
System.out.println("Driver was null so nothing to do");
}
}
}
My result is:
I have 100+ button in the page and I have to click on each button and to verify the link and page which opens after clicking.
so how to click without writing xpath for all 100+ button, i just need to click them and for each button different page will open.
Pls help me to proceed my selenium test
After you click the first button a new DOM is loaded, so click on the second (and third ...) will end with StaleElementReferenceException. You need to get all the href values first and then try them one by one. Just tried on web bbc.com with this code:
package navi;
import java.util.ArrayList;
import java.util.List;
import org.junit.After;
import org.junit.AfterClass;
import org.junit.Before;
import org.junit.BeforeClass;
import org.junit.Test;
import org.openqa.selenium.By;
import org.openqa.selenium.WebDriver;
import org.openqa.selenium.WebElement;
import org.openqa.selenium.firefox.FirefoxDriver;
import org.openqa.selenium.firefox.FirefoxOptions;
import org.openqa.selenium.firefox.FirefoxProfile;
import org.openqa.selenium.firefox.internal.ProfilesIni;
import org.openqa.selenium.support.ui.ExpectedConditions;
import org.openqa.selenium.support.ui.WebDriverWait;
public class Himanshu {
public static WebDriver driver;
#BeforeClass
public static void setUpClass() {
FirefoxOptions options = new FirefoxOptions();
ProfilesIni allProfiles = new ProfilesIni();
FirefoxProfile selenium_profile = allProfiles.getProfile("selenium_profile");
options.setProfile(selenium_profile);
options.setBinary("C:\\Program Files (x86)\\Mozilla Firefox\\firefox.exe");
System.setProperty("webdriver.gecko.driver", "C:\\Users\\pburgr\\Desktop\\geckodriver-v0.20.0-win64\\geckodriver.exe");
driver = new FirefoxDriver(options);
driver.manage().window().maximize();
}
#Before public void setUp() {} #After public void tearDown() {}
#AfterClass public static void tearDownClass() {driver.quit();}
#Test
public void lots_of_buttons() throws InterruptedException {
driver.get("http://www.bbc.com/");
// wait to page is fully loaded
wait_sec(driver, 5).until(ExpectedConditions.elementToBeClickable(By.xpath("//*[#id=\"orb-header\"]/div[1]/div[1]/a/img")));
// get all the desired webelements
List<WebElement> header_links = driver.findElements(By.xpath("//a[contains(#href,'x')]"));
// create new arraylist
ArrayList<String> hrefs = new ArrayList<String>();
// collect all the href values
for (WebElement link: header_links) {
hrefs.add(link.getAttribute("href"));
}
// visit all the URLs
for (String url: hrefs) {
driver.get(url);
Thread.sleep(1000);
// do some stuff here
}
}
// modified wait method
public WebDriverWait wait_sec(WebDriver driver, int sec) {
return new WebDriverWait(driver, sec);
}
}
I am fairly new to selenium, and I was trying to use some of the scripts being used in tutorials for my practice. I downloaded all the required .JAR files (Chrome drivers, Selenium Java and Stand Alone server) and added it to the path in Eclipse.
Below is the Code which I am trying to run to access a Weblink and then trying to verify if a user is able to log in successfully or not.
package learnautomation;
import org.openqa.selenium.By;
import org.openqa.selenium.WebDriver;
import org.openqa.selenium.chrome.ChromeDriver;
public class practice {
public static void main(String[] args) {
System.setProperty("webdriver.chrome.driver", "Mypath\\chromedriver.exe");
WebDriver driver = new ChromeDriver();
driver.get("http://www.gcrit.com/build3/admin/");
driver.findElement(By.name("username")).sendKeys("admin");
driver.findElement(By.name("password")).sendKeys("admin#123");
driver.findElement(By.id("tdb1")).click();
String url = driver.getCurrentUrl();
if (url.equals("http://www.gcrit.com/build3/admin/index.php")){
System.out.println("Successful");
}
else {
System.out.println("Unsuccessful");
}
driver.close();
}
}
While doing this I am getting this error:
"Error occurred during initialization of boot layer
java.lang.module.FindException: Module seleniumnew not found"
Also, import org.openqa.selenium.chrome.ChromeDriver; it says this is not accessible as well when I just hover over it.
try this, just edit paths and package name.
package navi;
import org.junit.After;
import org.junit.AfterClass;
import org.junit.Before;
import org.junit.BeforeClass;
import org.junit.Test;
import org.openqa.selenium.By;
import org.openqa.selenium.WebDriver;
import org.openqa.selenium.chrome.ChromeDriver;
import org.openqa.selenium.chrome.ChromeOptions;
import org.openqa.selenium.support.ui.ExpectedConditions;
import org.openqa.selenium.support.ui.WebDriverWait;
public class Avi {
public static WebDriver driver;
WebDriverWait wait5s = new WebDriverWait(driver,5);
#BeforeClass
public static void setUpClass() {
System.setProperty("webdriver.chrome.driver", "C:\\Users\\pburgr\\Desktop\\chromedriver\\chromedriver.exe");
ChromeOptions options = new ChromeOptions();
options.addArguments("user-data-dir=C:\\Users\\pburgr\\AppData\\Local\\Google\\Chrome\\User Data");
driver = new ChromeDriver(options);
driver.manage().window().maximize();}
#Before
public void setUp() {}
#After
public void tearDown() {}
#AfterClass
public static void tearDownClass() {driver.close();driver.quit();}
#Test
public void avi() throws InterruptedException {
driver.get("http://www.gcrit.com/build3/admin/");
wait5s.until(ExpectedConditions.elementToBeClickable(By.name("username"))).sendKeys("admin");
driver.findElement(By.name("password")).sendKeys("admin#123");
driver.findElement(By.id("tdb1")).click();
// wait to resolve the click before checking url
Thread.sleep(1000);
String url = driver.getCurrentUrl();
if (url.equals("http://www.gcrit.com/build3/admin/index.php")){
System.out.println("Successful");
}
else {
System.out.println("Unsuccessful");
}
}
}
I'm trying to get the link name of all the links present on a particular page, but some link get visible after hovering over a particular link, like there is a parent child relation between those links and Selenium WebDriver is unable to identify the link names for the child link.
Is there any possible way to get all the link names including the parent and the child?
package test;
import static org.junit.Assert.*;
import java.io.IOException;
import java.net.HttpURLConnection;
import java.net.MalformedURLException;
import java.net.URL;
import java.util.List;
import java.util.concurrent.TimeUnit;
import mx4j.tools.remote.http.HTTPConnectionManager;
import org.junit.After;
import org.junit.Before;
import org.junit.Test;
import org.openqa.selenium.By;
import org.openqa.selenium.WebDriver;
import org.openqa.selenium.WebElement;
import org.openqa.selenium.ie.InternetExplorerDriver;
public class SparshLink {
WebDriver driver;
#Before
public void setUp() throws Exception {
String basePath = System.getProperty("user.dir");
String finalPath = basePath + "\\IEDriver\\IEDriverServer.exe";
System.setProperty("webdriver.ie.driver", finalPath);
driver = new InternetExplorerDriver();
driver.navigate().to("http://sparshv2/");
}
#After
public void tearDown() throws Exception {
driver.quit();
}
#Test
public void test() throws Exception {
List<WebElement> list;
list = driver.findElements(By.tagName("a"));
System.out.println("Number of links : " + list.size());
driver.manage().timeouts().implicitlyWait(2, TimeUnit.SECONDS);
for (int i = 0; i < list.size(); i++) {
System.out.println(i);
URL url = new URL(list.get(i).getAttribute("href"));
HttpURLConnection connection = (HttpURLConnection) url.openConnection();
if (connection.getResponseCode() != 404) {
System.out.println(list.get(i).getText() + "is working properly.");
} else {
System.out.println(list.get(i).getText() + "is not working properly.");
}
connection.disconnect();
}
}
}
I'm sorry i wont be able to share the code behind the application.
'a' will be the tag but I believe all your URL should be there in href attribute.
You should refer below:-
Finding all "A" tags with specific strings in the attribute href?
I was trying to run Sikuli WebDriver based tests on Sauce On Demand infrastructure.
But I have a problem with RemoteWebDriver.
I have this BaseSikuliWebDriver class
package com.pitito.sikuli.base;
import java.lang.reflect.Method;
import java.net.MalformedURLException;
import java.net.URL;
import java.util.HashMap;
import java.util.Map;
import org.openqa.selenium.Platform;
import org.openqa.selenium.remote.Augmenter;
import org.openqa.selenium.remote.CapabilityType;
import org.openqa.selenium.remote.DesiredCapabilities;
import org.openqa.selenium.remote.RemoteWebDriver;
import org.testng.ITestResult;
import org.testng.annotations.AfterMethod;
import org.testng.annotations.BeforeMethod;
import com.pitito.core.basetests.BaseLoggingTest;
import com.pitito.selenium.webdriver.RemoteWebDriverSession;
import com.pitito.selenium.webdriver.WebDriverScreenshooter;
import com.pitito.sikuli.webdriver.SikuliFirefoxDriver;
/**
* Base class for all Sikuli WebDriver tests.
*
* #author guillem.hernandez
*/
public abstract class BaseSikuliWebDriverTest {
Map<String, Object> sauceJob = new HashMap<String, Object>();
private static SikuliFirefoxDriver sikuliDriver;
protected SikuliFirefoxDriver driver() {
return getDriver();
}
public static SikuliFirefoxDriver getDriver() {
return sikuliDriver;
}
public static void setDriver(SikuliFirefoxDriver driver) {
BaseSikuliWebDriverTest.sikuliDriver = driver;
}
#Override
#BeforeMethod(alwaysRun = true)
protected void setup(Method method, Object[] testArguments) {
super.setup(method, testArguments);
String sessionId = method.getName() + "_" + testArguments.hashCode();
DesiredCapabilities caps = DesiredCapabilities.firefox();
caps.setCapability("id", sessionId);
caps.setCapability("name", sessionId);
caps.setCapability(CapabilityType.BROWSER_NAME, "firefox");
caps.setCapability("platform", Platform.XP);
caps.setCapability("version", "21");
try {
sikuliDriver = (SikuliFirefoxDriver) new Augmenter().augment(new RemoteWebDriver(new URL("http://"
+ RemoteWebDriverSession.USER + ":" + RemoteWebDriverSession.APIKEY
+ "#ondemand.saucelabs.com:80/wd/hub"), caps));
} catch (MalformedURLException e) {
e.printStackTrace();
}
setDriver(sikuliDriver);
}
#Override
#AfterMethod(alwaysRun = true)
protected void teardown(ITestResult tr, Method method) {
if ((logger() != null) && (tr.getStatus() == ITestResult.FAILURE)) {
logUnexpectedException(tr.getThrowable());
}
super.teardown(tr, method);
sikuliDriver.quit();
}
#Override
protected void logScreenshot(String screenshotName) {
logResource(new WebDriverScreenshooter(driver(), screenshotName).getScreenshot());
}
}
The test I implemented is the Sikuli WebDriver example and the code is as follows:
package com.pitito.sikuli.tests;
import java.net.MalformedURLException;
import java.net.URL;
import org.openqa.selenium.By;
import org.openqa.selenium.Keys;
import org.openqa.selenium.WebElement;
import org.testng.annotations.Test;
import com.pitito.sikuli.base.BaseSikuliWebDriverTest;
import com.pitito.sikuli.webdriver.ImageElement;
/**
* Sikuli Firefox WebDriver Automated Test Example.
*
* #author guillem.hernandez
*/
public class SikuliGoogleCodeTest extends BaseSikuliWebDriverTest {
#Test(groups = { "ES" }, description = "Use Sikuli to search on Google Maps")
public void testSikuliWebDriverPassingExample_ES() {
verifySikuliWebDriverPassingTest();
}
private void verifySikuliWebDriverPassingTest() {
// visit Google Map
driver().get("https://maps.google.com/");
// enter "Denver, CO" as search terms
WebElement input = driver().findElement(By.id("gbqfq"));
input.sendKeys("Denver, CO");
input.sendKeys(Keys.ENTER);
ImageElement image;
// find and click on the image of the lakewood area
try {
image = driver().findImageElement(new URL("https://dl.dropbox.com/u/5104407/lakewood.png"));
image.doubleClick();
// find and click on the image of the kendrick lake area
image =
driver().findImageElement(new URL("https://dl.dropbox.com/u/5104407/kendrick_lake.png"));
image.doubleClick();
// find and click the Satellite icon to switch to the satellite view
image = driver().findImageElement(new URL("https://dl.dropbox.com/u/5104407/satellite.png"));
image.click();
// find and click the plus button to zoom in
image = driver().findImageElement(new URL("https://dl.dropbox.com/u/5104407/plus.png"));
image.click();
// find and click the link button
WebElement linkButton = driver().findElement(By.id("link"));
linkButton.click();
} catch (MalformedURLException e) {
e.printStackTrace();
}
}
}
When I try to run the test, the error I get is this one:
[Invoker 18958118] Invoking #BeforeMethod BaseSikuliWebDriverTest.setup(java.lang.reflect.Method, [Ljava.lang.Object;)[pri:0, instance:com.pitito.sikuli.tests.SikuliGoogleCodeTest#137008a]
Failed to invoke configuration method com.pitito.sikuli.base.BaseSikuliWebDriverTest.setup:org.openqa.selenium.remote.RemoteWebDriver$$EnhancerByCGLIB$$52a1cf6f cannot be cast to com.pitito.sikuli.webdriver.SikuliFirefoxDriver
The problem resides here:
sikuliDriver = (SikuliFirefoxDriver) new Augmenter().augment(new RemoteWebDriver(new URL("http://"
+ RemoteWebDriverSession.USER + ":" + RemoteWebDriverSession.APIKEY
+ "#ondemand.saucelabs.com:80/wd/hub"), caps));
How can I use SikuliFirefoxDriver remotely? How can I cast RemoteWebDriver with SikuliFirefoxDriver? Can I do it?
As far as I know, the Selenium Grid server does not have the capability of passing Sikuli commands (and binary screenshots for comparison purposes) through its JSON api. Not even SauceLabs has this capability. Hopefully, its on the radar to be implemented someday. On the SauceLabs forum, there is someone that asked this question (and I answered that one also with this same answer).
I know that there is a project in-progress called Marionette that is supposed to be able to automate browser/Firefox menus and native dialogs.
I implemented a remote driver version of sikuli driver. You can use that to do this action.
Please feel free to fork:
https://github.com/AJ-72/SikuliRemoteWebdriver
My guess is that SikuliFirefoxDriver cannot be augmented because it was not invoked as RemoteWebdriver. Try invoke it as Remote webdriver with sikuli as desired capabilities.
Please, post here if it worked (I could not find proofs if it is possible, but still worth a shot)