I have been working on automated tests and are trying to skip the login page, by logging in one time, saving the cookies on a text file and then finally reading and adding the files on a new browser instance.
It works fine when I just create one browser at a time, but if I create multiple browser parallel, only one page receives the cookies (thus skipping the login page as intended).
Here is the code:
My Setup(). This works perfect. I just login, saves the cookie data in text file and close the browser.
#BeforeClass
public void cookie_setup(){
//Firefox
System.setProperty("webdriver.gecko.driver", "C:\\SeleniumGecko/geckodriver.exe");
webdriver = new FirefoxDriver();
File file = new File("Cookies.data");
try
{
file.delete();
file.createNewFile();
FileWriter fileWrite = new FileWriter(file);
BufferedWriter Bwrite = new BufferedWriter(fileWrite);
for ( Cookie ck: webdriver.manage().getCookies())
{
Bwrite.write((ck.getName()+";"+ck.getValue()+";"+ck.getDomain()+";"+ck.getPath()+";"+ck.getExpiry()+";"+ck.isSecure()));
Bwrite.newLine();
}
Bwrite.close();
fileWrite.close();
}catch (Exception ex)
{
ex.printStackTrace();
}
cookies = webdriver.manage().getCookies();
webdriver.quit();
}
Then I create a new firefox browser
Reads the cookie data in the text file
Add the the cookies to the page with:`webdriver.manage().addCookie(ck);
#Test(threadPoolSize = 4, invocationCount = 4)
public void search() throws InterruptedException {
System.setProperty("webdriver.gecko.driver",
"C:\\SeleniumGecko/geckodriver.exe");
webdriver.navigate().to("http://pagethatrequirelogin/");
try {
File file = new File("Cookies.data");
FileReader fileReader = new FileReader(file);
BufferedReader Buffreader = new BufferedReader(fileReader);
String strline;
while ((strline = Buffreader.readLine()) != null) {
StringTokenizer token = new StringTokenizer(strline, ";");
while (token.hasMoreTokens()) {
String name = token.nextToken();
String value = token.nextToken();
String domain = token.nextToken();
String path = token.nextToken();
Date expiry = null;
String val;
if (!(val = token.nextToken()).equals("null")) {
expiry = new Date(val);
}
Boolean isSecure = Boolean.valueOf(token.nextToken());
Cookie ck = new Cookie(name, value, domain, path, expiry, isSecure);
System.out.println(ck);
webdriver.manage().addCookie(ck);
}
}
}
catch(Exception ex){
}
}
I use TestNG and with the parameters:
#Test(threadPoolSize = 4, invocationCount = 4)
I create 4 firefox browers at the same time, but only one of browser receives the cookies and skip login page.
It works perfectly fine if I just create one browser at a time multiple times.
The issue is because you are perhaps having your #Test methods overwrite the webdriver instance. Here's a sample that shows how a thread-safe version of it would look like:
import org.openqa.selenium.Cookie;
import org.openqa.selenium.WebDriver;
import org.openqa.selenium.firefox.FirefoxDriver;
import org.testng.annotations.AfterMethod;
import org.testng.annotations.BeforeClass;
import org.testng.annotations.BeforeMethod;
import org.testng.annotations.Test;
import java.io.IOException;
import java.nio.file.Files;
import java.nio.file.Paths;
import java.text.DateFormat;
import java.text.ParseException;
import java.util.ArrayList;
import java.util.Date;
import java.util.List;
import java.util.StringTokenizer;
public class CookiesSample {
private static final ThreadLocal<WebDriver> drivers = new ThreadLocal<>();
private List<Cookie> cookies = new ArrayList<>();
#BeforeClass
public void beforeClass() throws IOException {
String text = "Cookies.data";
List<String> lines = Files.readAllLines(Paths.get(text));
lines.forEach(line -> {
StringTokenizer token = new StringTokenizer(line, ";");
while (token.hasMoreTokens()) {
String name = token.nextToken();
String value = token.nextToken();
String domain = token.nextToken();
String path = token.nextToken();
Date expiry = null;
String val = token.nextToken();
if (!val.equals("null")) {
expiry = asDate(val);
}
Boolean isSecure = Boolean.valueOf(token.nextToken());
Cookie ck = new Cookie(name, value, domain, path, expiry, isSecure);
cookies.add(ck);
}
});
}
#BeforeMethod
public void beforeMethod() {
WebDriver driver = new FirefoxDriver();
cookies.forEach(driver.manage()::addCookie);
drivers.set(driver);
}
#AfterMethod
public void afterMethod() {
drivers.get().quit();
drivers.remove();
}
#Test(threadPoolSize = 4, invocationCount = 4)
public void testMethod() {
drivers.get().get("http://pagethatrequirelogin/");
}
private static Date asDate(String text) {
try {
return DateFormat.getInstance().parse(text);
} catch (ParseException e) {
return new Date();
}
}
}
Related
I'm trying to find a file based on the first 8 numbers i pull from an excel sheet for each iteration. Whenever I use the code below I get the error message "Local variable CaseID defined in an enclosing scope must be final or effectively final". I'm still new so i'm not sure how to fix this even though it sounds like a simple fix. The issue is happening at the caseID variable towards the bottom of the code.
package Chrome;
//CHROME
import java.io.File;
import java.io.FileFilter;
import java.io.FileInputStream;
import java.io.FileOutputStream;
import java.io.FilenameFilter;
import org.apache.poi.ss.usermodel.Cell;
import org.apache.poi.ss.usermodel.DataFormatter;
import org.apache.poi.xssf.usermodel.XSSFCell;
import org.apache.poi.xssf.usermodel.XSSFSheet;
import org.apache.poi.xssf.usermodel.XSSFWorkbook;
import org.openqa.selenium.By;
import org.openqa.selenium.WebDriver;
import org.openqa.selenium.chrome.ChromeDriver;
import org.openqa.selenium.chrome.ChromeOptions;
public class ValidateInput {
public static void main(String[] args) throws Throwable {
// TODO Auto-generated method stub
// String filePath=System.getProperty("user.dir")+"\\UP.xlsx";
//File src = new File(filePath);
//System.out.println(filePath);
File src = new File("C:\\Users\\Z246379\\Documents\\TestDataFolder\\ValidateInput.xlsx");
FileInputStream fis = new FileInputStream(src);
XSSFWorkbook wb = new XSSFWorkbook(fis);
XSSFSheet Sheet1 = wb.getSheetAt(0);
int i1 = 2;
//Username
String data0 = Sheet1.getRow(2).getCell(0).getStringCellValue();
//Password
String data01 = Sheet1.getRow(2).getCell(1).getStringCellValue();
//Case ID
String caseID = Sheet1.getRow(i1).getCell(2).getStringCellValue();
// Description
String Desc = Sheet1.getRow(2).getCell(3).getStringCellValue();
//Internal Claim File
String ICF = Sheet1.getRow(1).getCell(4).getStringCellValue();
String CRPT = Sheet1.getRow(1).getCell(5).getStringCellValue();
// final String caseID1 = caseID1;
//Chrome driver code
System.out.println("Called openBrowser");
String exePath = System.getProperty("user.dir")+"\\chromedriver.exe";
System.setProperty("webdriver.chrome.driver", exePath);
ChromeOptions options = new ChromeOptions();
options.setExperimentalOption("useAutomationExtension", false);
options.addArguments("start-maximized");
options.addArguments("--no-sandbox");
options.addArguments("--disable-extensions-except");
options.addArguments("disable-extensions");
//options.setExperimentalOption("useAutomationExtension", false);
WebDriver driver = new ChromeDriver(options);
driver.get("https://client.abc.com");
//Logging in
driver.findElement(By.xpath("//input[#id='userid']")).sendKeys(data0);
driver.findElement(By.xpath("//body[#class='no-nav']/div[#id='wrapper']/header[#class='header']/div[#class='container']/div[#class='menu']/div[#class='drop']/div[#class='login-form']/form[#method='POST']/div[2]/input[1]")).sendKeys(data01);
driver.findElement(By.xpath("//input[#value='Sign In']")).click();
Thread.sleep(2000);
//Navigate to Validate Input
driver.findElement(By.xpath("//span[#class='wpsNavLevel2'][contains(text(),'EZ-Test')]")).click();
Thread.sleep(3000);
driver.get("https://ezt.abc.com/Test/inputFiles/selectValidateInput/selectValidateInput.xhtml");
while (caseID != null)
{
caseID = Sheet1.getRow(i1).getCell(2).getStringCellValue();
//Input Validate Input stuff
driver.findElement(By.xpath("//input[#id='mainForm:caseId']")).sendKeys(caseID);
driver.findElement(By.xpath("//input[#id='mainForm:testBedDesc']")).sendKeys(Desc);
driver.findElement(By.xpath("//select[#id='mainForm:customRptOptions']")).sendKeys(ICF);
driver.findElement(By.xpath("//span[#id='mainForm:customReportLabel']")).click();
File dir = new File("C:\\Users\\Z333379\\Documents\\Test beds");
FilenameFilter filter = new FilenameFilter() {
public boolean accept (File dir, String name) {
return name.startsWith(caseID); //this is where i get the error. its not letting me make case ID a variable. but i really need the starts with stuff
}
};
String[] children = dir.list(filter);
if (children == null) {
System.out.println("Either dir does not exist or is not a directory");
} else {
for (int i=0; i< children.length; i++) {
String filename = children[i];
System.out.println(filename);
driver.findElement(By.xpath("//input[#id='mainForm:comprehensive']")).sendKeys("C:\\Users\\Z246379\\Documents\\Test beds\\" + filename);
}
}
driver.findElement(By.xpath("//button[#id='mainForm:reset']")).click();
i1=i1+1;
}
}
}
Here:
FilenameFilter filter = new FilenameFilter() {
public boolean accept (File dir, String name) {
return name.startsWith(caseID);
}
};
You cannot use a mutable variable in the definition of an inner local class. That's why you receive the error compiler.
Create a temporary variable and assign the value of caseID there, use this variable in your inner local class:
final String localCaseID = caseID;
FilenameFilter filter = new FilenameFilter() {
public boolean accept (File dir, String name) {
return name.startsWith(localCaseID);
}
};
You can use inner class instead of anonymous one. Define it like:
private static class CaseIDFilenameFilter implements FilenameFilter {
private final String caseID;
private CaseIDFilenameFilter(String caseID) {
this.caseID = caseID;
}
#Override
public boolean accept(File dir, String name) {
return name.startsWith(caseID);
}
}
And then use it in your code like:
FilenameFilter filter = new CaseIDFilenameFilter(str);
/I am using the following code to get the cookie value but I am only getting 1st and 2nd part.But not getting 3rd and 4th part(null as you can see).
Please help me with this.I have attached the screenshot of the cookies i get manually/
WebDriver driver;
System.setProperty("webdriver.ie.driver",
"C:\\Users\\MR049860\\Documents\\Selenium\\IEDriverServer\\IEDriverServer.exe");
driver = new InternetExplorerDriver();
driver.get("https://www.example.com");
// Input Email id and Password If you are already Register
driver.findElement(By.name("j_username")).sendKeys("publisher");
driver.findElement(By.name("j_password")).sendKeys("Passw0rd");
WebDriverWait wait = new WebDriverWait(driver, 5);
WebElement element = wait.until(ExpectedConditions.elementToBeClickable(By.id("BtnButton__")));
// WebElement ele = driver.findElement(By.id("ctllogonBtnButton__"));
element.sendKeys(Keys.ENTER);
// create file named Cookies to store Login Information
File file = new File("madhu.data");
try
{
// Delete old file if exists
file.delete();
file.createNewFile();
FileWriter fileWrite = new FileWriter(file);
BufferedWriter Bwrite = new BufferedWriter(fileWrite);
// loop for getting the cookie information
// loop for getting the cookie information
for(Cookie ck : driver.manage().getCookies())
{
Bwrite.write((ck.getName()+";"+ck.getValue()+";"+ck.getDomain()+";"+ck.getPath()+";"+ck.getExpiry()+";"+ck.isSecure()));
Bwrite.newLine();
}
Bwrite.close();
fileWrite.close();
}
catch(Exception ex)
{
ex.printStackTrace();
}
Output : -
ASPSESSIONIDSZQPRQCS;NFPFIAGDBMJNOMKPCPKESHDC;null;/;null;true
You can get only name and value. You are not allowed to get cookies of other domains/paths or expiry date. That's the security policy of web browsers.
Below code can be used to get the cookie value
package utility;
import java.util.Set;
import org.openqa.selenium.By;
import org.openqa.selenium.Capabilities;
import org.openqa.selenium.Cookie;
import org.openqa.selenium.Keys;
import org.openqa.selenium.WebElement;
import org.openqa.selenium.ie.InternetExplorerDriver;
import org.openqa.selenium.remote.DesiredCapabilities;
import org.openqa.selenium.remote.RemoteWebDriver;
import org.openqa.selenium.support.ui.ExpectedConditions;
import org.openqa.selenium.support.ui.WebDriverWait;
public class CookieUtility {
public static InternetExplorerDriver getDriver() throws InterruptedException {
InternetExplorerDriver driver;
System.setProperty("webdriver.ie.driver",
"C:\\Documents\\Selenium\\IEDriverServer\\IEDriverServer.exe");
driver = new InternetExplorerDriver();
driver.get("https://example.com");
// Input Email id and Password If you are already Register
driver.findElement(By.name("username")).sendKeys("password");
driver.findElement(By.name("password")).sendKeys("Paswrd");
WebDriverWait wait = new WebDriverWait(driver, 5);
WebElement element = wait.until(ExpectedConditions.elementToBeClickable(By.id("logonButton")));
// WebElement ele = driver.findElement(By.id("ctllogonBtnButton__"));
element.sendKeys(Keys.ENTER);
for (int i = 0; i < 2 && driver.findElements(By.id("textbox")).size() == 0; i++) {
Thread.sleep(10000);
}
element.sendKeys(Keys.F5);
return driver;
}
public static String[] getCookieValues(InternetExplorerDriver driver) throws InterruptedException {
// create file named Cookies to store Login Information
Set<Cookie> cks = driver.manage().getCookies();
String[] cookieValues = new String[cks.size()];
int i = 0;
for (Cookie ck : cks) {
cookieValues[i] = ck.getValue();
i++;
}
i = 0;
return cookieValues;
}
public static String getSessionId(InternetExplorerDriver driver) {
String sessionId = driver.getSessionId().toString();
return sessionId;
}
public static void main(String args[]) throws InterruptedException {
InternetExplorerDriver driver = getDriver();
String[] values = getCookieValues(driver);
String sessionId = getSessionId(driver);
}
public static String getcookiestring(String sessionId, String cookie1, String cookie2, String cookie3) {
String cookie = "JSESSIONID=" + sessionId + "; hi.session.co.entity=" + cookie2 + "; hi.session.id.identifier="
+ cookie1 + "; hi.session.client.identifier=" + cookie3;
return cookie;
}
}
I have created a web scraper which brings the market data of share rates from the website of stock exchange. www.psx.com.pk in that site there is a hyperlink of Market Summary. From that link I have to scrap the data. I have created a program which is as follows.
package com.market_summary;
import java.io.File;
import java.io.FileWriter;
import java.io.IOException;
import java.text.ParseException;
import java.text.SimpleDateFormat;
import java.util.ArrayList;
import java.util.Date;
import java.util.Iterator;
import java.util.Locale;
import org.jsoup.Jsoup;
import org.jsoup.nodes.Document;
import org.jsoup.nodes.Element;
import org.jsoup.select.Elements;
public class ComMarket_summary {
boolean writeCSVToConsole = true;
boolean writeCSVToFile = true;
boolean sortTheList = true;
boolean writeToConsole;
boolean writeToFile;
public static Document doc = null;
public static Elements tbodyElements = null;
public static Elements elements = null;
public static Elements tdElements = null;
public static Elements trElement2 = null;
public static String Dcomma = ",";
public static String line = "";
public static ArrayList<Elements> sampleList = new ArrayList<Elements>();
public static void createConnection() throws IOException {
System.setProperty("http.proxyHost", "191.1.1.202");
System.setProperty("http.proxyPort", "8080");
String tempUrl = "http://www.psx.com.pk/index.php";
doc = Jsoup.connect(tempUrl).get();
System.out.println("Successfully Connected");
}
public static void parsingHTML() throws Exception {
File fold = new File("C:\\market_smry.csv");
fold.delete();
File fnew = new File("C:\\market_smry.csv");
for (Element table : doc.getElementsByTag("table")) {
for (Element trElement : table.getElementsByTag("tr")) {
trElement2 = trElement.getElementsByTag("td");
tdElements = trElement.getElementsByTag("td");
FileWriter sb = new FileWriter(fnew, true);
if (trElement.hasClass("marketData")) {
for (Iterator<Element> it = tdElements.iterator(); it.hasNext();) {
if (it.hasNext()) {
sb.append("\r\n");
}
for (Iterator<Element> it2 = trElement2.iterator(); it.hasNext();) {
Element tdElement2 = it.next();
final String content = tdElement2.text();
if (it2.hasNext()) {
sb.append(formatData(content));
sb.append(" | ");
}
}
System.out.println(sb.toString());
sb.flush();
sb.close();
}
}
System.out.println(sampleList.add(tdElements));
}
}
}
private static final SimpleDateFormat FORMATTER_MMM_d_yyyy = new SimpleDateFormat("MMM d, yyyy", Locale.US);
private static final SimpleDateFormat FORMATTER_dd_MMM_yyyy = new SimpleDateFormat("dd-MMM-YYYY", Locale.US);
public static String formatData(String text) {
String tmp = null;
try {
Date d = FORMATTER_MMM_d_yyyy.parse(text);
tmp = FORMATTER_dd_MMM_yyyy.format(d);
} catch (ParseException pe) {
tmp = text;
}
return tmp;
}
public static void main(String[] args) throws IOException, Exception {
createConnection();
parsingHTML();
}
}
Now, the problem is when I execute this program it should create a .csv file but what actually happens is it's not creating any file. When I debug this code I found that program is not entering in the loop. I don't understand that why it is doing so. While when I run the same program on the other website which have slightly different page structure it is running fine.
What I understand that this data is present in the #document which is a virtual element and doesn't mean anything that's why program can't read it while there is no such thing in other website. Kindly, help me out to read the data inside the #document element.
Long Story Short
Change your temp url to http://www.psx.com.pk/phps/index1.php
Explanation
There is no table in the document of http://www.psx.com.pk/index.php.
Instead it is showing it's content in two frameset.
One is dummy with url http://www.psx.com.pk/phps/blank.php.
Another one is the real page which is showing actual data and it's url is
http://www.psx.com.pk/phps/index1.php
When i am using this code it gives error:
2011-08-10 13:18:13.368::WARN: EXCEPTION
java.lang.IllegalAccessError: tried to access method org.mortbay.util.Utf8StringBuffer.(I)V from class org.mortbay.jetty.HttpURI
Code:
package com.google.api.client.sample.docs.v3;
import com.google.api.client.auth.oauth.OAuthCredentialsResponse;
import com.google.api.client.auth.oauth.OAuthHmacSigner;
import com.google.api.client.auth.oauth.OAuthParameters;
import com.google.api.client.googleapis.auth.oauth.GoogleOAuthAuthorizeTemporaryTokenUrl;
import com.google.api.client.googleapis.auth.oauth.GoogleOAuthGetAccessToken;
import com.google.api.client.googleapis.auth.oauth.GoogleOAuthGetTemporaryToken;
import com.google.api.client.http.HttpTransport;
import com.google.api.client.sample.docs.v3.model.DocsUrl;
import java.awt.Desktop;
import java.awt.Desktop.Action;
import java.net.URI;
public class Auth {
private static final String APP_NAME ="Google Documents List Data API Java Client Sample";
private static OAuthHmacSigner signer;
private static OAuthCredentialsResponse credentials;
static void authorize(HttpTransport transport) throws Exception {
// callback server
LoginCallbackServer callbackServer = null;
String verifier = null;
String tempToken = null;
try {
callbackServer = new LoginCallbackServer();
callbackServer.start();
// temporary token
GoogleOAuthGetTemporaryToken temporaryToken =new GoogleOAuthGetTemporaryToken();
signer = new OAuthHmacSigner();
signer.clientSharedSecret = "anonymous";
temporaryToken.signer = signer;
temporaryToken.consumerKey = "in.gappsdemo.in";
//temporaryToken.scope ="https://apps-apis.google.com/a/feeds/user/";
temporaryToken.scope = DocsUrl.ROOT_URL;
temporaryToken.displayName = APP_NAME;
temporaryToken.callback = callbackServer.getCallbackUrl();
System.out.println("temporaryToken.callback: "+temporaryToken.callback);
OAuthCredentialsResponse tempCredentials = temporaryToken.execute();
signer.tokenSharedSecret = tempCredentials.tokenSecret;
System.out.println("signer.tokenSharedSecret: " + signer.tokenSharedSecret);
// authorization URL
GoogleOAuthAuthorizeTemporaryTokenUrl authorizeUrl =new GoogleOAuthAuthorizeTemporaryTokenUrl();
authorizeUrl.temporaryToken = tempToken = tempCredentials.token;
String authorizationUrl = authorizeUrl.build();
System.out.println("Go to this authorizationUrl: " + authorizationUrl);
// launch in browser
boolean browsed = false;
if (Desktop.isDesktopSupported()) {
Desktop desktop = Desktop.getDesktop();
if (desktop.isSupported(Action.BROWSE)) {
System.out.println("In if browsed condition:");
desktop.browse(URI.create(authorizationUrl));
browsed = true;
}
}
if (!browsed) {
String browser = "google-chrome";
Runtime.getRuntime().exec(new String[] {browser, authorizationUrl});
System.out.println("In if !browsed condition1:");
}
System.out.println("tempToken: "+tempToken);
verifier = callbackServer.waitForVerifier(tempToken);
System.out.println("GoogleOAuthGetAccessToken: ");
} finally {
System.out.println("server Stop:");
if (callbackServer != null) {
System.out.println("server Stop:");
callbackServer.stop();
}
}
System.out.println("GoogleOAuthGetAccessToken: ");
GoogleOAuthGetAccessToken accessToken = new GoogleOAuthGetAccessToken();
accessToken.temporaryToken = tempToken;
accessToken.signer = signer;
accessToken.consumerKey = "in.gappsdemo.in";
accessToken.verifier = verifier;
credentials = accessToken.execute();
signer.tokenSharedSecret = credentials.tokenSecret;
System.out.println("signer.tokenSharedSecret: ");
createOAuthParameters().signRequestsUsingAuthorizationHeader(transport);
}
static void revoke() {
if (credentials != null) {
try {
GoogleOAuthGetAccessToken.revokeAccessToken(createOAuthParameters());
} catch (Exception e) {
e.printStackTrace(System.err);
}
}
}
private static OAuthParameters createOAuthParameters() {
OAuthParameters authorizer = new OAuthParameters();
authorizer.consumerKey = "in.gappsdemo.in";
authorizer.signer = signer;
authorizer.token = credentials.token;
return authorizer;
}
}
Here are two potentially related, posts. Notice you're getting an Error, not an Exception, so that is interesting. You might try running your JVM with -verbose to see where each class is getting loaded from, to see if maybe you're compiling with one class/jar, but accidentally running with another.
Also notice from the error that the packages are slightly different "org.mortbay.util" vs. "org.mortbay.jetty", so the UTF8StringBuffer constructor would need to be more visible. But again, normally a compiler would catch that.
I realize this isn't a full answer, to be sure, but at least a couple places to start looking.
I am wanting to take a screenshot of a page using HtmlUnitDriver I came across this Link where this guy has made a custom HTML unit driver to take the screenshot.
But unfortunately, while implementing that I am getting an exception.
"Exception in thread "main" java.lang.ClassCastException: [B cannot be cast to java.io.File
at Test.main(Test.java:39)"
My code is as follows-
import java.io.File;
import java.io.IOException;
import org.openqa.selenium.OutputType;
import org.openqa.selenium.WebDriver;
import com.gargoylesoftware.htmlunit.BrowserVersion;
public class Test extends ScreenCaptureHtmlUnitDriver {
public static void main(String[] args) throws InterruptedException, IOException {
WebDriver driver = new ScreenCaptureHtmlUnitDriver(BrowserVersion.FIREFOX_38);
driver.get("https://www.google.com/?gws_rd=ssl");
try{
File scrFile = ((ScreenCaptureHtmlUnitDriver) driver).getScreenshotAs(OutputType.FILE);
FileUtils.copyFile(scrFile, new File("D:\\TEMP.PNG"));
}catch (Exception e) {
e.printStackTrace();
}
}
}
HtmlUnit driver which I am using(the one which is in the link) is this-
import java.io.ByteArrayOutputStream;
import java.io.IOException;
import java.net.URL;
import java.util.Collections;
import java.util.HashMap;
import java.util.Iterator;
import java.util.LinkedList;
import java.util.List;
import java.util.Map;
import java.util.regex.Matcher;
import java.util.regex.Pattern;
import java.util.zip.ZipEntry;
import java.util.zip.ZipOutputStream;
import org.apache.commons.io.FilenameUtils;
import org.apache.commons.io.IOUtils;
import org.openqa.selenium.Capabilities;
import org.openqa.selenium.OutputType;
import org.openqa.selenium.TakesScreenshot;
import org.openqa.selenium.WebDriverException;
import org.openqa.selenium.htmlunit.HtmlUnitDriver;
import org.openqa.selenium.internal.Base64Encoder;
import org.openqa.selenium.remote.CapabilityType;
import org.openqa.selenium.remote.DesiredCapabilities;
import com.gargoylesoftware.htmlunit.BrowserVersion;
import com.gargoylesoftware.htmlunit.WebClient;
import com.gargoylesoftware.htmlunit.WebRequest;
import com.gargoylesoftware.htmlunit.WebWindow;
import com.gargoylesoftware.htmlunit.html.HtmlElement;
import com.gargoylesoftware.htmlunit.html.HtmlPage;
public class ScreenCaptureHtmlUnitDriver extends HtmlUnitDriver implements TakesScreenshot {
private static Map<String, byte[]> imagesCache = Collections.synchronizedMap(new HashMap<String, byte[]>());
private static Map<String, String> cssjsCache = Collections.synchronizedMap(new HashMap<String, String>());
// http://stackoverflow.com/questions/4652777/java-regex-to-get-the-urls-from-css
private final static Pattern cssUrlPattern = Pattern.compile("background(-image)?[\\s]*:[^url]*url[\\s]*\\([\\s]*([^\\)]*)[\\s]*\\)[\\s]*");// ?<url>
public ScreenCaptureHtmlUnitDriver() {
super();
}
public ScreenCaptureHtmlUnitDriver(boolean enableJavascript) {
super(enableJavascript);
}
public ScreenCaptureHtmlUnitDriver(Capabilities capabilities) {
super(capabilities);
}
public ScreenCaptureHtmlUnitDriver(BrowserVersion version) {
super(version);
DesiredCapabilities var = ((DesiredCapabilities) getCapabilities());
var.setCapability(CapabilityType.TAKES_SCREENSHOT, true);
}
//#Override
#SuppressWarnings("unchecked")
public <X> X getScreenshotAs(OutputType<X> target) throws WebDriverException {
byte[] archive = new byte[0];
try {
archive = downloadCssAndImages(getWebClient(), (HtmlPage) getCurrentWindow().getEnclosedPage());
} catch (Exception e) {
}
if(target.equals(OutputType.BASE64)){
return target.convertFromBase64Png(new Base64Encoder().encode(archive));
}
if(target.equals(OutputType.BYTES)){
return (X) archive;
}
return (X) archive;
}
// http://stackoverflow.com/questions/2244272/how-can-i-tell-htmlunits-webclient-to-download-images-and-css
protected byte[] downloadCssAndImages(WebClient webClient, HtmlPage page) throws Exception {
WebWindow currentWindow = webClient.getCurrentWindow();
Map<String, String> urlMapping = new HashMap<String, String>();
Map<String, byte[]> files = new HashMap<String, byte[]>();
WebWindow window = null;
try {
window = webClient.getWebWindowByName(page.getUrl().toString()+"_screenshot");
webClient.getPage(window, new WebRequest(page.getUrl()));
} catch (Exception e) {
window = webClient.openWindow(page.getUrl(), page.getUrl().toString()+"_screenshot");
}
String xPathExpression = "//*[name() = 'img' or name() = 'link' and (#type = 'text/css' or #type = 'image/x-icon') or #type = 'text/javascript']";
List<?> resultList = page.getByXPath(xPathExpression);
Iterator<?> i = resultList.iterator();
while (i.hasNext()) {
try {
HtmlElement el = (HtmlElement) i.next();
String resourceSourcePath = el.getAttribute("src").equals("") ? el.getAttribute("href") : el
.getAttribute("src");
if (resourceSourcePath == null || resourceSourcePath.equals(""))
continue;
URL resourceRemoteLink = page.getFullyQualifiedUrl(resourceSourcePath);
String resourceLocalPath = mapLocalUrl(page, resourceRemoteLink, resourceSourcePath, urlMapping);
urlMapping.put(resourceSourcePath, resourceLocalPath);
if (!resourceRemoteLink.toString().endsWith(".css")) {
byte[] image = downloadImage(webClient, window, resourceRemoteLink);
files.put(resourceLocalPath, image);
} else {
String css = downloadCss(webClient, window, resourceRemoteLink);
for (String cssImagePath : getLinksFromCss(css)) {
URL cssImagelink = page.getFullyQualifiedUrl(cssImagePath.replace("\"", "").replace("\'", "")
.replace(" ", ""));
String cssImageLocalPath = mapLocalUrl(page, cssImagelink, cssImagePath, urlMapping);
files.put(cssImageLocalPath, downloadImage(webClient, window, cssImagelink));
}
files.put(resourceLocalPath, replaceRemoteUrlsWithLocal(css, urlMapping)
.replace("resources/", "./").getBytes());
}
} catch (Exception e) {
}
}
String pagesrc = replaceRemoteUrlsWithLocal(page.getWebResponse().getContentAsString(), urlMapping);
files.put("page.html", pagesrc.getBytes());
webClient.setCurrentWindow(currentWindow);
return createZip(files);
}
String downloadCss(WebClient webClient, WebWindow window, URL resourceUrl) throws Exception {
if (cssjsCache.get(resourceUrl.toString()) == null) {
cssjsCache.put(resourceUrl.toString(), webClient.getPage(window, new WebRequest(resourceUrl))
.getWebResponse().getContentAsString());
}
return cssjsCache.get(resourceUrl.toString());
}
byte[] downloadImage(WebClient webClient, WebWindow window, URL resourceUrl) throws Exception {
if (imagesCache.get(resourceUrl.toString()) == null) {
imagesCache.put(
resourceUrl.toString(),
IOUtils.toByteArray(webClient.getPage(window, new WebRequest(resourceUrl)).getWebResponse()
.getContentAsStream()));
}
return imagesCache.get(resourceUrl.toString());
}
public static byte[] createZip(Map<String, byte[]> files) throws IOException {
ByteArrayOutputStream bos = new ByteArrayOutputStream();
ZipOutputStream zipfile = new ZipOutputStream(bos);
Iterator<String> i = files.keySet().iterator();
String fileName = null;
ZipEntry zipentry = null;
while (i.hasNext()) {
fileName = i.next();
zipentry = new ZipEntry(fileName);
zipfile.putNextEntry(zipentry);
zipfile.write(files.get(fileName));
}
zipfile.close();
return bos.toByteArray();
}
List<String> getLinksFromCss(String css) {
List<String> result = new LinkedList<String>();
Matcher m = cssUrlPattern.matcher(css);
while (m.find()) { // find next match
result.add( m.group(2));
}
return result;
}
String replaceRemoteUrlsWithLocal(String source, Map<String, String> replacement) {
for (String object : replacement.keySet()) {
// background:url(http://org.com/images/image.gif)
source = source.replace(object, replacement.get(object));
}
return source;
}
String mapLocalUrl(HtmlPage page, URL link, String path, Map<String, String> replacementToAdd) throws Exception {
String resultingFileName = "resources/" + FilenameUtils.getName(link.getFile());
replacementToAdd.put(path, resultingFileName);
return resultingFileName;
}
}
UPDATE
Code provided by Andrew works- but I wanted to know if there is a way by which we can download only selected resources. For eg this website I would like to download only captcha image those id is "//*[#id='cimage']" because downloading all the resources will take a long time. Is there a way by which we can download only the specific resource. Because with the existing code provided
below all the resources get downloaded.
byte[] zipFileBytes = ((ScreenCaptureHtmlUnitDriver) driver).getScreenshotAs(OutputType.BYTES);
FileUtils.writeByteArrayToFile(new File("D:\\TEMP.PNG"), zipFileBytes);
The error says that the code is trying to convert a byte[] to a File. It's easy to see why if you just strip out the unused paths from getScreenshotAs:
public <X> X getScreenshotAs(OutputType<X> target) throws WebDriverException {
byte[] archive = new byte[0];
try {
archive = downloadCssAndImages(getWebClient(), (HtmlPage) getCurrentWindow().getEnclosedPage());
} catch (Exception e) {
}
return (X) archive;
}
There's no way you can get a File out of that. OutputType.FILE is not supported, so you have to handle file output yourself. Luckily, that's easy. You can change your code to:
byte[] zipFileBytes = ((ScreenCaptureHtmlUnitDriver) driver).getScreenshotAs(OutputType.BYTES);
FileUtils.writeByteArrayToFile(new File("D:\\TEMP.PNG"), zipFileBytes);
See FileUtils.writeByteArrayToFile() for more.
Check this out this may helpful for you
File scrFile = ((TakesScreenshot)driver).getScreenshotAs(OutputType.FILE);
FileUtils.copyFile(scrFile, new File("C:/Users/home/Desktop/screenshot.png"));// copy it somewhere