We Keep Coding

How to call post method after setting the value of the form for screen scraping using java

Background : I have a webpage (.aspx) which have few dropdown lists.The list value is getting populated using Ajax call based on the selection of previous dropdown. After selecting the value of all drop down lists we can click on download button and the data will be downloaded based on the downloaded data we need to perform some other operations.
what i already did: I am able to set the drop down data via calling the ajax correctly but sending a post request is a problem. Here is the code snippet/pseudo Code.
Feel free to use any tool along with java
public static void main(String[] args) throws FailingHttpStatusCodeException, IOException {
final WebClient webClient = new WebClient(BrowserVersion.FIREFOX_17);
WebRequest request = new WebRequest(new URL(DataDownloader.MY_URL),HttpMethod.POST);
webClient.getOptions().setThrowExceptionOnScriptError(false);
webClient.setJavaScriptTimeout(10000);
webClient.getOptions().setJavaScriptEnabled(true);
webClient.setAjaxController(new NicelyResynchronizingAjaxController());
webClient.getOptions().setTimeout(10000);
HtmlPage page = webClient.getPage(request);
HtmlSelect firstDd = (HtmlSelect) page.getElementById("dd1_id");
List<HtmlOption> firstOption = firstDd.getOptions();
firstDd.setSelectedAttribute(firstOption.get(2), true);
webClient.waitForBackgroundJavaScript(3000);
HtmlPage pgAfterFirstDd = (HtmlPage) webClient.getCurrentWindow().getEnclosedPage();
HtmlSelect secondDd = (HtmlSelect) pgAfterFirstDd.getElementById("dd2_id");
List<HtmlOption> secondOption = secondDd.getOptions();
secondDd.setSelectedAttribute(secondOption.get(2), true);
webClient.waitForBackgroundJavaScript(10000);
//set the value for all other dropdowns
HtmlPage finalpage = (HtmlPage) webClient.getCurrentWindow().getEnclosedPage();
HtmlForm form = finalpage.getHtmlElementById("aspnetForm");
webClient.waitForBackgroundJavaScript(10000);
request.setRequestBody("REQUESTBODY");
Page redirectPage = webClient.getPage(request);
// HtmlSubmitInput submitInput=form.getInputByName("btnSubmit");
// submitInput.click();
/*HtmlButton submitButton = (HtmlButton) pageAfterWard.createElement("btnSubmit");
submitButton.setAttribute("type", "submit");
form.appendChild(submitButton);
HtmlPage nextPage = (HtmlPage) submitButton.click();*/
}

Why you hide your error details? Is there any secret? If you like helpful answers you have to provide as many information as possible.
So i do a wild guess...
submitInput.click();
will return a PDF. In this case you have to do something like
Page pdfPage = submitInput.click();
WebResponse resp = pdfPage.getWebResponse();
if("application/pdf".equals(resp.getContentType())) {
.... process the bytes
.... resp.getContentAsStream()
}
HtmlUnit has four kind of pages HtmlPage/XmlPage/TextPage and UnexpectedPage. Binary content like PDF or office documents are handled as UnexpectedPage. Processing this content is up to you.

as you mentioned in the comment under RBRi's Answer that you were getting the typecast error.can you please mention
what the exact error you were getting
what type of file/response you were expecting.
Because the code looks good to me and it should work perfectly..

final WebClient webClient = new WebClient(BrowserVersion.FIREFOX_17);
Looks like you are using an old version, please use the latest one.
WebRequest request = new WebRequest(new URL(DataDownloader.MY_URL),HttpMethod.POST);
With HtmlUnit you usually do not work with requests. The idea is to work more 'browser like'. Use something like getPage(final URL url).
List<HtmlOption> firstOption = firstDd.getOptions();
firstDd.setSelectedAttribute(firstOption.get(2), true);
Do your work more 'browser like'
firstOption.get(2)setSelected(true);
This will do all the background work for you like deselection the other options and event processing for you.
Regarding submitting the form your idea of
HtmlSubmitInput submitInput=form.getInputByName("btnSubmit");
HtmlPage nextPage = submitInput.click();
looks correct. Maybe your have to wait after that also.
If you still have problems you have to provide the URL you are working with to enable us to reproduce/debug your case.

Read alerts in HTML unit

I am working on HTML unit with Java to read HTML pages. I have a scenario where I have to read messages from the popup/ alert window. I have an index page page = form.getInputByName("index").click();
After I click on the index page I get the response page. But before that I get an alert some thing like
I want to read the above message and then proceed with ok.
I tried with alert handlers like `
ConfirmHandler okHandler = new ConfirmHandler(){
#Override
public boolean handleConfirm(Page page, String message) {
System.out.println(">>>>>>>>>>>>>>>>>>> message--"+message);
return true;
}
};
webClient.setConfirmHandler(okHandler);`
But this is not working for me.

You should be using the CollectingAlertHandler instead:
CollectingAlertHandler alertHandler = new CollectingAlertHandler();
webClient.setAlertHandler(alertHandler);
/*Your browsing codes here*/
List<String> alertmsgs = new ArrayList<String>();
alertmsgs = alertHandler.getCollectedAlerts();
Then you can use the message obtained as you want. Reminder: You do not need to click the OK button.

HTMLUnit Wait For JS Issue

I'm using HTMLUnit in order to put text in a input box and then clicking on a link which is actually a JS call.
The problem comes up when I put text in an input , using inputBox.setValueAttribute("example input"); . In this case , after clicking the button the page does not change at all.
On the other hand , once I delete inputBox.setValueAttribute("example input"); and then click the button, the page content does change and includes an error for empty input.
Below is the code i've used in order to put text in the relevant input and then click the button.
public void addressToBlockPlot(){
WebClient client = new WebClient(BrowserVersion.FIREFOX_24);
client.getOptions().setThrowExceptionOnScriptError(false);
client.getOptions().setThrowExceptionOnScriptError(false);
client.setJavaScriptTimeout(10000);
client.getOptions().setJavaScriptEnabled(true);
client.setAjaxController(new NicelyResynchronizingAjaxController());
client.getOptions().setTimeout(10000);
try {
HtmlPage page = client.getPage("http://mapi.gov.il/Pages/LotAddressLocator.aspx");
HtmlInput inputBox = (HtmlInput)page.getHtmlElementById("AddressInput");
final HtmlAnchor a = (HtmlAnchor) page.getElementById("helkaSubmit");
inputBox.setValueAttribute("example input");
a.click();
client.waitForBackgroundJavaScript(2000);
HtmlPage page2= (HtmlPage) client.getCurrentWindow().getEnclosedPage();
System.out.println(page2.asXml());
} catch (Exception e) {
}
}
Any ideas for solving this issue ?
EDIT:
I've tried using inputBox.setValueAttribute(""); , which resulted in receiving the same error that I got when no input text was set at all.

I have almost similar issue, where JS take some time to load, I have solved this issue by setting wait and retries until page load successfully by adding a condition,
int input_length = page.getByXPath("//input").size();
int tries = 5;
while (tries > 0 && input_length < 12) {
tries--;
synchronized (page) {
page.wait(2000);
}
input_length = page.getByXPath("//input").size();
System.out.println("page loaded successfully");
}
here input_length determines that whether required page is loaded or not. you can use similar condition according to structure of your page.

HtmlUnit, trying to get 2 forms, get exception at second form

Currently using HtmlUnit.
Getting first login page is no problem, succesfully logging in, getting next page, "clicking" the link to get the "MyDetails" page.
After getting "MyDetails" page, i want to get the same way as im getting the first login form.
Why i need to get the form is that i want to change the password, and the fields are in a form.
When im trying to get the second form, it gives me exception as follows:
com.gargoylesoftware.htmlunit.ElementNotFoundException: elementName=[form] attributeName=[name] attributeValue=[form2]
Gives exception at this line of code:
HtmlForm form2 = page3.getFormByName("form2");
Note: the first form name is "form1" & second form name is "form2".
Is this a problem with HtmlUnit?
Code:
try {
WebClient webclient = new WebClient(BrowserVersion.FIREFOX_3_6);
HtmlPage page1 = webclient.getPage("http://www.highveld.mobi/pages/clubvip/login.aspx");
HtmlForm form = page1.getFormByName("form1");
HtmlSubmitInput buttonLogin = form.getInputByName("cmdLogin");
HtmlTextInput cellLogin = form.getInputByName("txtCellNumber");
HtmlPasswordInput passLogin = form.getInputByName("txtLoginPassword");
cellLogin.setValueAttribute(change);
passLogin.setValueAttribute(oldPass);
HtmlPage page2 = buttonLogin.click();
HtmlAnchor link = page2.getAnchorByHref("updateprofile.aspx");
HtmlPage page3 = link.click();
System.out.println(page3.getUrl());
HtmlForm form2 = page3.getFormByName("form2");
HtmlPasswordInput pass = form2.getInputByName("txtPassword");
HtmlPasswordInput passConfirm = form2.getInputByName("txtConfirmPassword");
HtmlSubmitInput button = form2.getInputByName("cmdUpdate");
pass.setValueAttribute(newPass);
passConfirm.setValueAttribute(newPass);
HtmlPage page4 = button.click();
}

First of all, please update to HtmlUnit 2.9 in case you are using an old version.
Secondly, replace this with this:
System.out.println(page3.getUrl());
HtmlForm form2 = page3.getFormByName("form2");
With this:
System.out.println(page3.getUrl());
System.out.println(page3.asXml());
HtmlForm form2 = page3.getFormByName("form2");
And check for the existance of the form2 element, which I'm pretty sure it shouldn't be there as it is throwing a ElementNotFoundException.
I usually use XPath instead of getFormByName, you can give it a try too.
Hope this helps!

How to use HtmlUnit in Java to select an element from a drop down box?

I'm using HtmlUnit in Java to navigate to a web page. From that webpage i need to log in and then go from there. I know how to type in the user name and password but then there is a dropdown box where i need to select one of the options. How do i select an option from a dropdown box in HtmlUnit?
Thanks

You can navigate and manipulate the page <select> elements using HtmlSelect:
WebClient client = ...
Page page = client.getPage(url);
HtmlSelect select = (HtmlSelect) page.getElementById(mySelectId);
HtmlOption option = select.getOptionByValue(desiredOptionValue);
select.setSelectedAttribute(option, true);
The JavaDoc shows that there are a lot of flexible API methods for doing things like this.

Add the follwoing lines:
protected void selectOption(WebElement el, String option)
{
Select select = new Select(el);
select.selectByVisibleText(option);
}
protected WebElement elById(String id)
{
return driver.findElement(By.id(id));
}
// "title" is your drop-down HTML id
public void populateForm(String elValue)
{
selectOption(elById("title"), elValue);
}

Following code:
HtmlSelect select = page.getElementById(mySelectId);
should be:
HtmlSelect select = (HtmlSelect)page.getElementById(mySelectId);