copyURLToFile behavior in multithreading - java

I'm using copyURLToFile to download a file from url.
When I use it sequentially, in a loop, about 74% of the iterations sucsseds.
But when I use it in multithreading and there are 100 concurrently running Threads, it drops down to below 5%.
Interestingly enough, when i debug the program, it get back to 74% sucsses and even higher.
Every unsuccessful attempt is caused by an exception thrown:
java.net.MalformedURLException: no protocol:
My question is whether FileUtils uses some limited resource which cause a race condition, or any other reason it fails in significantly higher rate in multithreaded environment.
I have checked the documentation of copyURLToFile, but there is no refernce there to this issue.
I have also checked the source code of copyURLToFile), but couldn't spot any thing to give a hint of why those exceptions are thrown.
public void doDownload () {
try (final WebClient webClient = new WebClient()) {
// Get the page
final HtmlPage page1 = webClient.getPage(YouTubeConvertor);
// Get the form that we are dealing with and within that form,
// find the submit button and the field that we want to change.
final List<HtmlForm> form = page1.getForms();
final HtmlTextInput textField = form.get(0).getInputByName(InputName);
final HtmlSubmitInput button = form.get(0).getInputByValue(InputValue);
// Change the value of the text field
textField.setValueAttribute(this.link);
// Now submit the form by clicking the button and get back the second page.
final HtmlPage page2 = button.click();
HtmlElement down = page2.getHtmlElementById(FileElement);
String name = down.getAttribute(HrefElement);
URL linkUrl = new URL(name);
File f = new File (Dir , this.saveName + Mp3Ending);
FileUtils.copyURLToFile(linkUrl, f);
} catch(Exception ex) {
System.out.println("Some form of error happened! " + ex.toString());
// timeout based on attempts and time elapsed
if (times++ < 10 && TimeUnit.SECONDS.convert
(System.nanoTime()-this.startTime, TimeUnit.NANOSECONDS) < 60) {
doDownload ();
}
}

Related

Check loading time in WebDriver test during execution

I use Selenium WebDriver 3.14 and test is executed in Chrome browser. I need to measure response time of a page in execution time to check it is under a predefined value. If it is greater than this value some additional actions should be done. So I need different solution than System.currentTimeMillis(), because check of this value should be done automatically in background. It is an AJAX like window, so when loading takes too long time, it should be closed by script. Window example:
The typical solution to this is a try/catch against a wait. E.g. if the next step is to click a button that shows once loading completes:
WebDriverWait wait = new WebDriverWait(driver, LOADING_TIMEOUT);
WebElement webElement;
try {
webElement = wait.until(elementToBeClickable(By.id(id)));
} catch (TimeoutException ex) {
// Close loading window
return;
}
webElement.click();
However, there is a common problem if you are using implicit timeouts in Selenium. This doesn't work too well, particularly if the implicit timeout is longer than the LOADING_TIMEOUT, as this slows down the polling cycle in the wait.until().
In this case, the simplest solution is to temporarily reduce the implicit timeout:
WebDriverWait wait = new WebDriverWait(driver, LOADING_TIMEOUT);
WebElement webElement;
try {
driver.manage().timeouts().implicitlyWait(1, TimeUnit.SECONDS);
webElement = wait.until(elementToBeClickable(By.id(id)));
} catch (TimeoutException ex) {
// Delay any further interaction until the timeout has been restored
webElement = null;
} finally {
driver.manage().timeouts().implicitlyWait(DEFAULT_TIMEOUT,
TimeUnit.SECONDS);
}
if (webElement != null)
webElement.click();
else
// Close loading window
If I understand correctly, you could decrease time in selenium.waitForPageToLoad("100000"); to a wanted predefined value, let us say 20 seconds. So if you want the page loading to stop if it is not loaded in 20 seconds, try something like this:
long start = System.currentTimeMillis();
try {
selenium.waitForPageToLoad("20000");
System.out.println("The page load is too long!");
} catch {
long timeToLoad= (System.currentTimeMillis()-start);
System.out.println("The page loaded in " +timeToLoad+ " seconds.");
}
You should try setting Logging Preferences through capability CapabilityType.LOGGING_PREFS for performance-log.
For example:
LoggingPreferences logs = new LoggingPreferences();
logs .enable(LogType.PERFORMANCE, Level.ALL);
caps.setCapability(CapabilityType.LOGGING_PREFS, logs);
you can get performance log entries as below.
for (LogEntry entry : driver.manage().logs().get(LogType.PERFORMANCE)) {
System.out.println(entry.toString());
//do the needful
}
I think you're looking for API testing not Automation Testing.
Postman API Testing Setup and Usage Tutorial
Hopefully this will help
edit:
Alternatively, a more lightweight solution for API testing:
Online API tester

Unable to handle alert using phantomJS in Java

I have a Java code as below and when I am running through PhantomJs getting "Unsupported Command Exception" but it is working fine if I run through firefox and chrome:-
Note: With phantomJs we could able to execute till 3rd step in below code.I searched in many blogs but those answers didn't solve my problem.
1. cvvField.sendKeys(cvcData);
2. proceedToPayBtn.click();
3. Reporter.log("Card details are submitted from payment UI page");
4. Alert a1=driver.switchTo().alert();
5. Reporter.log("Alert with text:"+a1.getText());
6. a1.accept();
Here cvvField and proceedToPayBtn are WebElements and cvcData have value as "111".
Error log:-
org.openqa.selenium.UnsupportedCommandException: Invalid Command Method -
{"headers":{"Accept-Encoding":"gzip,deflate","Cache-Control":"no-cache","Connection":"Keep-Alive","Host":"localhost:30462","User-Agent":"Apache-HttpClient/4.5.1 (Java/1.8.0_101)"},"httpVersion":"1.1","method":"GET","url":"/alert_text","urlParsed":{"anchor":"","query":"","file":"alert_text","directory":"/","path":"/alert_text","relative":"
/alert_text","port":"","host":"","password":"","user":"","userInfo":"","authority":"","protocol":"","source":"/alert_text","queryKey":{},"chunks":["alert_text"]},"urlOriginal":"/session/9e392a50-ce79-11e6-b24a-2b12cf1ec4d6/alert_text"}
Command duration or timeout: 31 milliseconds
I have edited above code as below but same error is coming.Please suggest
if (driver instanceof PhantomJSDriver)
{
JavascriptExecutor je = (JavascriptExecutor) driver;
je.executeScript("window.alert = function(){};");
je.executeScript("window.confirm = function(){return true;};");
System.out.println("Alert has been handled");
} else {
Alert a1 = driver.switchTo().alert();
a1.accept();
}
I am getting "Alert has been handled" in output console but alert is not handled.
Some issue due to wait time can be the source of your problem
The code above can help to wait until element is visible (since the wait of ngWebDriver or Selenium Webdriver are not compatible with PhantomJS)
public static String waitJSResponse(PhantomJSDriver driver, String script) {
String ReturnedValue = null;
int sleeper = 10;
Boolean flag = false;
int timeOut = 30000;
int i = 0;
while ((!flag) & ((i*sleeper)<timeOut)) {
try {
Thread.sleep(sleeper);
ReturnedValue = (String) driver.executeScript(script);
} catch (Exception e) {
flag = false;
i++;
}
if (ReturnedValue != null) {
flag = true;
System.out.println("Overall wait time is : "+(i * sleeper)+" ms \n\r");
break;
}
}
return ReturnedValue;
}
This code will wait 10ms then verify that the element is visble, if there is an exception, it will loop again.
The returned value must be a text, an object or anything that is not null.
the script value must be your JS script to get the correct element.
Hope it work.
I tried the above code by:-
1.Creating a class "Test" and writing above method in it.
2.Above method is called by creating an object(TestObject) as
TestObject.waitJSResponse((PhantomJSDriver) driver, "window.confirm = function(){return true;};");
But ReturnedValue in
try
{
Thread.sleep(sleeper);
ReturnedValue = (String) driver.executeScript(script);
System.out.println(ReturnedValue);
}
returns null.So Can u please help with this?

HtmlUnit Does not get content after click

I am tying to parse a website with html unit. The process basically;
WebClient client = new WebClient(BrowserVersion.CHROME);
client.waitForBackgroundJavaScript(5 * 1000);
HtmlPage page = client.getPage("http://www.exapmle.com"); //here it waits to run js code.
HtmlUnorderedList ul = (HtmlUnorderedList) page.getByXPath("//ul[contains(#class, 'class-name')]").get(0);
HtmlListItem li = (HtmlListItem) ul.getChildNodes().get(1); // I want to click li and get result page. But it takes a little time to execute.
li.click();
client.waitForBackgroundJavaScript(5 * 1000); //At here it does not do what I want.
After that when I check the page, I see that its content does not change.
What can I do to get correct page result?
Thanks.
You could try polling for a javascript condition to be true
int attempts = 20;
int pollMillis = 500;
boolean success = false;
for (int i = 0; i < attempts && !success; i++) {
TimeUnit.MILLISECONDS.sleep(pollMillis);
if (someJavascriptCondition == true) {
success = true;
}
}
if (!success) throw new RuntimeException(String.format("Condition not met after %s millis", attempts * pollMillis);
A similar technique discussed here
WebClient client = new WebClient;
HtmlPage page = client.getPage("http://www.exapmle.com");
client.waitForBackgroundJavaScript(5 * 1000);
Thread.sleep(10*1000);// this code will waite to 10 seconds
HtmlUnorderedList ul = (HtmlUnorderedList) page.getByXPath("//ul[contains(#class, 'class-name')]").get(0);
HtmlListItem li = (HtmlListItem) ul.getChildNodes().get(1); // I want to click li and get result page. But it takes a little time to execute.
li.click();
client.waitForBackgroundJavaScript(5 * 1000);
// this code will waite to 10 seconds
Thread.sleep(10*1000);
use Thread.sleep() instead of waitForBackgroundJavaScript
works for me!
You can use a JavaScriptJobManager to check the amount of JavaScript jobs that have yet to complete. Try the following code after you call click().
JavaScriptJobManager manager = page.getEnclosingWindow().getJobManager();
while (manager.getJobCount() > 0) {
System.out.printlin("Jobs remaining: " + manager.getJobCount());
Thread.sleep(1000);
}
You may want to add another way to end the while loop in case your JavaScript jobs never finish. Personally, I start terminating jobs manually with:
JavaScriptJob job = manager.getEarliestJob();
System.out.println("Stopping job: " + job.getId());
manager.stopJob(job.getId());
Hope this helps.

StateElementReferenceException in Selenium

I am getting a StateElementReferenceException when I try to perform a drag-and-drop action, see the code snippets below. Can anyone please help to solve this with appropriate suggestions and explanations?
#Test(priority=4)
public void addUserBySearch() {
driver.findElement(By.xpath(OR.getProperty("btnUserGroup"))).click();
driver.findElement(By.xpath(OR.getProperty("btnCreateUG"))).click();
driver.findElement(By.xpath(OR.getProperty("textUGName"))).sendKeys("UserGroup:" + Utils.randGen());
driver.findElement(By.xpath(OR.getProperty("userSearchField"))).sendKeys("testuser2", Keys.ENTER);
source = driver.findElement(By.xpath(OR.getProperty("searchedUserSource")));
destination = driver.findElement(By.xpath(OR.getProperty("userDestination")));
waitUntilElementVisible(30, source);
dragAndDrop(source, destination);
driver.findElement(By.xpath(OR.getProperty("btnScheduleNow"))).click();
}
public void waitUntilElementVisible(int seconds, WebElement element) {
WebDriverWait explicitWait = new WebDriverWait(driver, seconds);
explicitWait.until(ExpectedConditions.visibilityOf(element));
}
public void dragAndDrop(WebElement sourceElement, WebElement destinationElement){
try {
if (sourceElement.isDisplayed() && destinationElement.isDisplayed()) {
Actions action = new Actions(driver);
action.dragAndDrop(sourceElement, destinationElement).build().perform();
}
else {
System.out.println("Element not found to drag and drop");
}
}
catch (StaleElementReferenceException exc) {
exc.printStackTrace();
}
catch (NoSuchElementException exc) {
exc.printStackTrace();
}
}
btnUserGroup = //*[text()='User Groups']
btnCreateUG = //*[#id='stage']/div/div[2]/div/div/div[2]/div[3]/a
textUGName = //input[#id='user_group_name']
btnScheduleNow = //*[text()='Schedule Now']
userDestination = //*[#class='ConnectedList ConnectedListAdded']/div[2]/ul
userSearchField = //div[#class='ConnectedListConnectedListSelect']/div[2]/div[1]/ul/li[1]/input
searchedUserSource = //div[#class='ConnectedList ConnectedListSelect']/div[2]/ul/li/span[5]
Exception:
org.openqa.selenium.StaleElementReferenceException: Element not found in the cache - perhaps the page has changed since it was looked up
Command duration or timeout: 10.28 seconds
A StaleReferenceException occurs for mainly one of two reasons:
The element has been deleted entirely.
The element is no longer attached to the DOM.
One possible solution to your problem is to surround dragAndDrop() in a while loop, and continually try and dragAndDrop() until you succeed. So something like the following:
boolean draggedAndDropped = false;
while(!draggedAndDropped ) {
try{
source = driver.findElement(By.xpath(OR.getProperty("searchedUserSource")));
destination = driver.findElement(By.xpath(OR.getProperty("userDestination")));
dragAndDrop(source, destination);
draggedAndDropped = true;
} catch(StaleElementReferenceException e) {
System.out.println("StaleElementReferenceException caught, trying again");
}
}
Of course its never a good idea idea to give allow a loop to loop infinitely, so you will want to incorporate a max attempts into the while loop.
As stated by the Selenium project's documentation, a StaleElementReferenceException can be thrown for two reasons:
The element has been deleted entirely.
The element is no longer attached to the DOM.
As such, I suspect that a race condition is occurring at this point:
driver.findElement(By.xpath(OR.getProperty("userSearchField"))).sendKeys("testuser2", Keys.ENTER);
source = driver.findElement(By.xpath(OR.getProperty("searchedUserSource")));
After you've entered "testuser2" in the "userSearchField" WebElement you've used Keys.ENTER to presumably submit the search query - the problem may be occurring because WebDriver is retrieving a reference to the "searchedUserSource" WebElement before the page has fully refreshed.
To get around this, you could try using the WebElement.submit() method to submit the search as it blocks the WebDriver's execution until the page has finished changing (i.e. finished displaying the search result).
Below is an example of how you could implement this:
driver.findElement(By.xpath(OR.getProperty("userSearchField"))).sendKeys("testuser2").submit();
source = driver.findElement(By.xpath(OR.getProperty("searchedUserSource")));

Java ProgramCall.run hangs

Busy trying to Call RPG function from Java and got this example from JamesA. But now I am having trouble, here is my code:
AS400 system = new AS400("MachineName");
ProgramCall program = new ProgramCall(system);
try
{
// Initialise the name of the program to run.
String programName = "/QSYS.LIB/LIBNAME.LIB/FUNNAME.PGM";
// Set up the 3 parameters.
ProgramParameter[] parameterList = new ProgramParameter[2];
// First parameter is to input a name.
AS400Text OperationsItemId = new AS400Text(20);
parameterList[0] = new ProgramParameter(OperationsItemId.toBytes("TestID"));
AS400Text CaseMarkingValue = new AS400Text(20);
parameterList[1] = new ProgramParameter(CaseMarkingValue.toBytes("TestData"));
// Set the program name and parameter list.
program.setProgram(programName, parameterList);
// Run the program.
if (program.run() != true)
{
// Report failure.
System.out.println("Program failed!");
// Show the messages.
AS400Message[] messagelist = program.getMessageList();
for (int i = 0; i < messagelist.length; ++i)
{
// Show each message.
System.out.println(messagelist[i]);
}
}
// Else no error, get output data.
else
{
AS400Text text = new AS400Text(50);
System.out.println(text.toObject(parameterList[1].getOutputData()));
System.out.println(text.toObject(parameterList[2].getOutputData()));
}
}
catch (Exception e)
{
//System.out.println("Program " + program.getProgram() + " issued an exception!");
e.printStackTrace();
}
// Done with the system.
system.disconnectAllServices();
The application Hangs at this lineif (program.run() != true), and I wait for about 10 minutes and then I terminate the application.
Any idea what I am doing wrong?
Edit
Here is the message on the job log:
Client request - run program QSYS/QWCRTVCA.
Client request - run program LIBNAME/FUNNAME.
File P6CASEL2 in library *LIBL not found or inline data file missing.
Error message CPF4101 appeared during OPEN.
Cannot resolve to object YOBPSSR. Type and Subtype X'0201' Authority
FUNNAME insert a row into table P6CASEPF through a view called P6CASEL2. P6CASEL2 is in a different library lets say LIBNAME2. Is there away to maybe set the JobDescription?
Are you sure FUNNAME.PGM is terminating and not hung with a MSGW? Check QSYSOPR for any messages.
Class ProgramCall:
NOTE: When the program runs within the host server job, the library list will be the initial library list specified in the job description in the user profile.
So I saw that my problem is that my library list is not setup, and for some reason, the user we are using, does not have a Job Description. So to over come this I added the following code before calling the program.run()
CommandCall command = new CommandCall(system);
command.run("ADDLIBLE LIB(LIBNAME)");
command.run("ADDLIBLE LIB(LIBNAME2)");
This simply add this LIBNAME, and LIBNAME2 to the user's library list.
Oh yes, the problem is Library list not set ... take a look at this discussion on Midrange.com, there are different work-around ...
http://archive.midrange.com/java400-l/200909/msg00032.html
...
Depe

Categories