I'm trying to fetch data from this webpage: http://www.atm-mi.it/en/Giromilano/Pages/default.aspx. Basically I'm using HtmlUnit in Java to interact with the "Route and timetable finder" in the middle of the left column, looping through each option in the select, clicking on "Find" and gathering the data I need from the resulting pages.
I've had no problem extracting data for urban routes, but can't seem to handle the radio buttons above: clicking on "Underground" in a browser, for example, should bring a new page with different options in the select below.
But I keep getting the same Select as before; to be more precise, I keep getting the same page (page2 has the same HTML code as page).
Clearly something must be going wrong in the .click() function, but what?
This is a simple version of my code:
WebClient webClient = new WebClient(BrowserVersion.FIREFOX_3_6);
webClient.setThrowExceptionOnScriptError(false);
HtmlPage page = webClient.getPage("http://www.atm-mi.it/en/Giromilano/Pages/default.aspx");
HtmlRadioButtonInput radioButton2 = (HtmlRadioButtonInput) page.getElementById("ctl00_SPWebPartManager1_g_e31ad29e_62a8_401c_43ae_eb61300b4fc0_lines_type_rbl_0");
HtmlPage page2 = radioButton2.click();
HtmlSelect lineSelect = (HtmlSelect) page2.getElementById("ctl00_SPWebPartManager1_g_e31ad29e_62a8_401c_43ae_eb61300b4fc0_txt_dp_lines");
int size = lineSelect.getOptionSize();
System.out.println(size);
This is the radio button input HTML:
<input id="ctl00_SPWebPartManager1_g_e31ad29e_62a8_401c_43ae_eb61300b4fc0_lines_type_rbl_0" type="radio" name="ctl00$SPWebPartManager1$g_e31ad29e_62a8_401c_43ae_eb61300b4fc0$lines_type_rbl" value="0" onclick="javascript:setTimeout('__doPostBack(\'ctl00$SPWebPartManager1$g_e31ad29e_62a8_401c_43ae_eb61300b4fc0$lines_type_rbl$0\',\'\')', 0)" />
<label for="ctl00_SPWebPartManager1_g_e31ad29e_62a8_401c_43ae_eb61300b4fc0_lines_type_rbl_0">Underground</label>
The select:
<select name="ctl00$SPWebPartManager1$g_e31ad29e_62a8_401c_43ae_eb61300b4fc0$txt_dp_lines" id="ctl00_SPWebPartManager1_g_e31ad29e_62a8_401c_43ae_eb61300b4fc0_txt_dp_lines" class="dplinee">
EDIT:
Ok, so I've tried a different approach: since it looked like some kind of JavaScript engine problem, I figured I could try and disable JavaScript, carrying out the onclick action myself. This is the original JavaScript function:
var theForm = document.forms['aspnetForm'];
function __doPostBack(eventTarget, eventArgument) {
if (!theForm.onsubmit || (theForm.onsubmit() != false)) {
theForm.__EVENTTARGET.value = eventTarget;
theForm.__EVENTARGUMENT.value = eventArgument;
theForm.submit();
}
}
And this is what I did:
HtmlForm aspnetForm = (HtmlForm) page.getElementById("aspnetForm");
HtmlHiddenInput eventTarget = (HtmlHiddenInput) page.getElementById("__EVENTTARGET");
HtmlHiddenInput eventArgument = (HtmlHiddenInput) page.getElementById("__EVENTARGUMENT");
eventTarget.setValueAttribute("ctl00$SPWebPartManager1$g_e31ad29e_62a8_401c_43ae_eb61300b4fc0$lines_type_rbl$0");
eventArgument.setValueAttribute("");
HtmlElement submitButton = (HtmlElement) page.createElement("button");
submitButton.setAttribute("type", "submit");
aspnetForm.appendChild(submitButton);
HtmlPage page2 = submitButton.click();
All good, except I still keep getting the same page with the same old Select.
I know this is quite a long and boring question, but I thought I could update it anyway. I hope somebody will eventually have the patience to try this out (and at least confirm I'm not doing some obvious mistake).
I finally found a way to make this work. The second approach was almost right. I was correctly submitting the form, but with a difference from normal browsing: I didn't actually check the radio button. Apparently, the destination page used that information too. By adding this
HtmlRadioButtonInput radioButton = (HtmlRadioButtonInput) page.getElementById("ctl00_SPWebPartManager1_g_e31ad29e_62a8_401c_43ae_eb61300b4fc0_lines_type_rbl_0");
radioButton.setChecked(true);
to my previous attempt the submit action worked perfectly. I still don't know why the .click() method didn't work as expected, though, but this is good enough for me.
Related
My question is how do i click the radio button of html web table?
Additional info:
My code works just fine however I cant click the radio button of selected column/row using xpath. The xpath I am using within the loop is not working. However the xpaths I am using work individually in the Chrome browser. I am thinking feel I need to join both together but that did not work probably because i am not an expert in using xpath. I have been working on this for the past 24 hours and no solution. I have search for a solution on stackoverflow but have not found anything similar.
radio button xpath:
//TBODY[#id='changeStartWeekGrid_rows_tbody']/TR[7]/TD[1]/DIV[1]/DIV[1]/DIV[1]
my code is:
List < WebElement > payDates = driver.findElements(By.xpath("//table[#id='changeStartWeekGrid_rows_table']//tr[position()>1]/td[position()=5]"));
//** Begin third inner for-loop****
for (WebElement pd: payDates) {
System.out.println("sample1-> " + pd.getText());
SimpleDateFormat dateFormat = new SimpleDateFormat("MM/dd/yyyy");
Date payDate = dateFormat.parse(pd.getText());
System.out.println("sample-> " + dateFormat.format(payDate));
if (payDate.after(new Date())) {
System.out.println("inside for loop");
String radiobutton = "//TBODY[#id='changeStartWeekGrid_rows_tbody']/TR[7]/TD[1]/DIV[1]/DIV[1]/DIV[1]";
WebElement calrow = driver.findElement(By.xpath(pd + radiobutton));
calrow.click();
pd in the loop should already contain the correct link i am thinking because it is looping through all the dates in payDates.
HTML code i loop through this is just a sample.
<td id="changeStartWeekGrid_row_1_cell_4" style="" align="left" class="table-grid-cell OUTPUT_TEXT" title=""><span>03/02/2018</span></td>
HTML radio button
<div class="revitRadioButtonIcon"></div>
I actually found what i needed this was my answer //table[#id='changeStartWeekGrid_rows_table']//span[text()='"+dateFormat.format(payDate)+"']/parent::td/preceding-sibling::td//div[#class='revitRadioButtonIcon']
If found out that you use preceding-sibling::td// to go back and td/following-sibling::td// to go forward here is another example
element = driver.findElement(By.xpath("//td//*[contains(text(),'Regular Hours')]//ancestor::td/following-sibling::td//*[contains(#class,'reactTextBox')]"));
i am new to Htmlunit and trying to extract data from a website http://capitaline.com/new/index.asp. I have logged into the website successfully. When we log into website there are three frames.
One on the top to search for the company(like ACC ltd.) for which we are extracting data.
2nd frame has a tree which provide links to various data we want to look at.
3rd frame has the resulted data outcome on the basis of link you clicked in frame.
I managed to get the frame i need below:
HtmlPage companyAtGlanceTopWindow =(HtmlPage)companyAtGlanceLink.click().getEnclosingWindow().getTopWindow().getEnclosedPage();
HtmlPage companyAtGlanceFrame = (HtmlPage)companyAtGlanceTopWindow.getFrameByName("mid2").getEnclosedPage();
System.out.println(companyAtGlanceFrame.toString()); // This line returns the frame URL as i can see in my browser.
Output of print statement is
HtmlPage(http://capitaline.com/user/companyatglance.asp?id=CGO&cocode=6)#1194282974
Now i want my code to navigate down to the table inside this frame and for that i am using getByXPath() but it gives me nullPointerException. Here is the code for that.
HtmlTable companyGlanceTable1 = companyAtGlanceFrame.getFirstByXPath("/html/body/table[4]/tbody/tr/td/table/tbody/tr/td[1]/table");
My XPath for the current webpage(after i clicked the link)from which i am trying to extract table is seems correct, as it is copied from chrome element inspect. Please suggest some way to extract the table. I have done this type of extraction before but there i had id of table so, i used it.
Here is the HTML code for the table in the webpage.
<table width="100%" class = "tablelines" border = "0" >
I want to know that can you see the inner contents of each iframes in console (print asXml()), are they nested iframes?
well try this
List<WebWindow> windows = webClient.getWebWindows();
for(WebWindow w : windows){
HtmlPage hpage = (HtmlPage) w.getEnclosedPage();
System.out.println(hpage.asXml());
}
once you can see the contents,
HtmlPage hpage = (HtmlPage)webClient.getWebWindowByName(some_name).getEnclosedPage();
then using xpath grab your table contents(make sure your xpath is correct). It will work.(worked for me)
Thank you RDD for your feedback.
I solved the problem. Actually issue was not with the frame but with the XPath provided by chrome.
XPath Provided by chrome is:
/html/body/**table[4]**/tbody/tr/td/table/tbody/tr/td[1]/table
But the XPath worked for me is:
/html/body/**table[3]**/tbody/tr/td/table/tbody/tr/td[1]/table
It seems as, XPath provided by chrome has some glitch when there is a table within the path(Or may be some bug in htmlunit itself). I did many experiments and found that chrome always gives ../../table[row+1]/.. as XPath, while working XPath for htmlunit is ../../table[row]/..
SO, this code is working fine for me
HtmlTable companyGlanceTable1 = companyAtGlanceFrame.getFirstByXPath("/html/body/table[3]/tbody/tr/td/table/tbody/tr/td[1]/table");
I'm trying to get the textbox with u_0_1e as id, from the page wall but HtmlUnit does not find anything. The last line prints null.
Here's the code:
java.util.logging.Logger.getLogger("com.gargoylesoftware").setLevel(java.util.logging.Level.OFF);
WebClient client = new WebClient(BrowserVersion.CHROME);
JavaScriptEngine engine = new JavaScriptEngine(client);
client.setJavaScriptEngine(engine);
HtmlPage home = client.getPage("https://www.facebook.com/login.php");
HtmlSubmitInput login = (HtmlSubmitInput) home.getElementById("u_0_1");
HtmlTextInput name = (HtmlTextInput) home.getElementById("email");
HtmlPasswordInput pass = (HtmlPasswordInput) home.getElementById("pass");
name.setValueAttribute("myname");
pass.setValueAttribute("mypass");
HtmlPage page = login.click();
HtmlPage wall = client.getPage("https://www.facebook.com/");
System.out.println(wall.getElementById("u_0_1e"));
I have some comments about your issue.
First of all, you have disabled HtmlUnit's logging. So if you have any JavaScript issue then you are not going to see it. If you are actually getting a JavaScript error then the JavaScript code won't be fully executed. If the element you're trying to fetch was dynamically fetched from the server (probably using AJAX) then the JavaScript errors, if any, might result in that element not being fetched.
If you are webscraping, which is clearly the case, then you don't have any control over the JS so you can only accept it as not working or disable JS and manually processing the AJAX requests.
Of course, you will see the page perfectly working on a real browser but take into consideration that the JavaScript engine HtmlUnit uses is different from the real browsers.
Secondly, the two lines containing the word engine are absolutely unneeded.
Thirdly, as I mentioned in a previous question of yours, this will be more suitable to be handled by means of the Facebook API.
Finally, you might find this other answer useful:
JavaScript not being properly executed in HtmlUnit
I am trying to make a application which would connect to a site with the login provided by the user. I don't have any experience in interacting with websites in Java so I googled some and found hmtlunit to fit my needs.
But I ran into an error when trying to click the submit button for the login form:
public static boolean attempt_login(String username, String password) throws ElementNotFoundException, IOException {
WebClient webClient = new WebClient(BrowserVersion.FIREFOX_3_6);
webClient.setJavaScriptEnabled(false);
webClient.setThrowExceptionOnScriptError(false);
webClient.setRefreshHandler(new RefreshHandler() {
public void handleRefresh(Page page, URL url, int arg) throws IOException {
System.out.println("handleRefresh");
}
});
HtmlPage page = (HtmlPage) webClient.getPage(Config.LOGIN_PAGE);
List<HtmlForm> forms = page.getForms();
HtmlForm form = null;
for(HtmlForm f : forms){
if(f.getId().equals("login_form")){
form = f;
}
}
if(form == null){
throw new NullPointerException("Could not find form!");
}
form.getInputByName("username").setValueAttribute(username);
form.getInputByName("password").setValueAttribute(password);
page = (HtmlPage) form.getInputByValue("Login Now!").click();
System.out.println(page.asText());
return false;}
Somehow it fails to find the submitButton to login
com.gargoylesoftware.htmlunit.ElementNotFoundException: elementName=[input] attributeName=[value] attributeValue=[Login Now!]
at com.gargoylesoftware.htmlunit.html.HtmlForm.getInputByValue(HtmlForm.java:737)
at domain.Helper.attempt_login(Helper.java:41)
at TesterStartUp.main(TesterStartUp.java:15)
The html source code:
<button type="submit" value="Login Now!" onmouseover="this.style.backgroundPosition='bottom';" onmouseout="this.style.backgroundPosition='top';" onclick="return SetFocus();">Login Now!</button>
When I googled for a solution, I found something about disabling javascript would help. So i told the webclient to disabled it ( webClient.setJavaScriptEnabled(false);) but still had the exception.
At first I had the same issue with trying to select the form ("login_form") but there was a method where I could get the list of all forms and then just see if one matched the list. I couldn't find a way around it for the submit button, So I hoped someone else knows a solution to this problem.
Thanks in advance,
Sir Troll
The HtmlUnit getInputByValue() method operating on a <form> will only return types of HtmlInput, and the only Input Button type -- HtmlButtonInput -- represents <input type="button"> and NOT <button>. You will need to change your HTML or use a different HtmlUnit method call.
I had same kind of issue that I resolved myself:
When you try to login using html unit first check if login box is appearing onclick using jquery or not. If login box/div is appearing after click a button then you need to refresh the page (web client) to access new input elements
HtmlPage page = webClient.getPage("https://www.yourwebsite.com/#");
HtmlAnchor link=page.getElementByName("link");
link.click();
page.refresh();
If your site has a button instead of anchor link change it for html button, After refreshing may you can found the element.
Hope it will help and solve your issue
I am trying to get all the main links, then click on them and navigation to the page:
WebClient client = new WebClient();
HtmlPage page = client.getPage(url);
// Get all links with a href of www.example.com/pages/1_
List<HtmlAnchor> links = (List<HtmlAnchor>) page.getByXPath("//a[href='www.example.com/pages/1_*'");
links[0].click();
After calling click, does it return a HtmlPage? (The NetBeans documentation is not telling me.)
Does the XPath expression look OK?
I don't know how documentation works in NetBeans but the documentation is all available online, and if you go to it, you'll see that the return type is <P extends Page> which will probably be HtmlPage most of the time, but could also be XmlPage or something like that.
Simulates clicking on this element,
returning the page in the window that
has the focus after the element has
been clicked. Note that the returned
page may or may not be the same as the
original page, depending on the type
of element being clicked, the presence
of JavaScript action listeners, etc.