We Keep Coding

Going to next page on an aspx form with JSoup

I'm trying to go to the next page on an aspx form using JSoup.
I can find the next button itself. I just don't know what to do with it.
The idea is that, for that particular form, if the next button exists, we would simulate a click and go to the next page. But any other solution other than simulating a click would be fine, as long as we get to the next page.
I also need to update the results once we go to the next page.
// Connecting, entering the data and making the first request
...
// Submitting the form
Document searchResults = form.submit().cookies(resp.cookies()).post();
// reading the data. Everything up to this point works as expected
...
// finding the next button (this part also works as expected)
Element nextBtn = searchResults.getElementById("ctl00_MainContent_btnNext");
if (nextBtn != null) {
// click? I don't know what to do here.
searchResults = ??? // updating the search results to include the results from the second page
}
The page itself is www.somePage.com/someForm.aspx, so I can't use the solution stated here:
Android jsoup, how to select item and go to next page
I was unable to find any other suggestions.
Any ideas? What am I missing? Is simulating a click even possible with JSoup? The documentation says nothing about it. But I'm sure people are able to navigate these type of forms.
Also, I'm working with Android, so I can't use HtmlUnit, as stated here:
importing HtmlUnit to Android project
Thank you.

This is not Jsoup work! Jsoup is a parser with a nice DOM API that allows you to deal with wild HTML as if it were well-formed and not crippled with errors and nonsenses.
In your specific case you may be able to scrape the target site directly from your app by finding links and retrieving HTML pages recursively. Something like
private void scrape(String url) {
Document doc = Jsoup.connect(url).get();
// Analyze current document content here...
// Then continue
for (Element link : doc.select(".ctl00_MainContent_btnNext")) {
scrape(link.attr("href"));
}
}
But in the general case what you want to do requires far more functionality that Jsoup provides: a user agent capable of interpreting HTML, CSS and Javascript with a scriptable API that you can call from your app to simulate a click. For example Selenium:
WebDriver driver = new FirefoxDriver();
driver.findElement(By.name("next_page")).click();
Selenium can't be bundled in an Android app, so I suggest you put your Selenium code on a server and make it accessible with some REST API.

Pagination on ASPX can be a pain. The best thing you can do is to use your browser to see the data parameters it sends to the server, then try to emulate this in code.
I've written a detailed tutorial on how to handle it here but it uses the univocity HTML parser (which is commercial closed source) instead of JSoup.
In short, you should try to get a <form> element with id="aspnetForm", and read the form elements to generate a POST request for the next page. The form data usually comes out with stuff such as this:
__EVENTTARGET =
__EVENTARGUMENT =
__VIEWSTATE = /wEPDwUKMTU0OTkzNjExNg8WBB4JU29ydE9yZ ... a very long string
__VIEWSTATEGENERATOR = 32423F7A
... and other gibberish
Then you need to look at each one of these and compare with what your browser sends. Sometimes you need to get values from other elements of the page to generate a similar POST request. You may have to REMOVE some of the parameters you get - again, make your code behave exactly the same as your browser
After some (frustrating) trial and error you will get it working. The server should return a pipe-delimited result, which you can break down and parse. Something like:
25081|updatePanel|ctl00_ContentPlaceHolder1_pnlgrdSearchResult|
<div>
<div style="font-weight: bold;">
... more stuff
|__EVENTARGUMENT||343908|hiddenField|__VIEWSTATE|/wEPDwU... another very long string ...1Pni|8|hiddenField|__VIEWSTATEGENERATOR|32423F7A| other gibberish
From THAT sort of response you need to generate new POST requests for the subsequent pages, for example:
String viewState = substringBetween(ajaxResponse, "__VIEWSTATE|", "|");
Then:
request.setDataParameter("__VIEWSTATE", viewState);
There are will be more data parameters to get from each response. But a lot depends on the site you are targeting.
Hope this helps a little.

Selenium: how to get the value of hidden element which has all div tags

I would like to get the value of all div tags specified in attached. I have tried with all possible locators like classname etc, which is showing null. and tried with JavaScript also which is returning null.
Please see the screen shot and I need the selected text which is in blue color starts with "Enables enterprise IT to deploy networking services"

You need to research creating selectors as this isn't a difficult one. There are numerous approaches for this element, but here's one for you: $$("#offers-popover .description"). Obviously this is a CSS selector based on the $$ and you use getText from the Selenium API in order to scrape the element text, which is what I assume you are intending to do.
driver.findElement(By.css("#offers-popover .description")).getText();

Since your element is not visible you can try this:
String divText = driver.findElement(By.className("description")).getAttribute("textContent");
Or, if this is not the only element on the page with the class description:
WebElement popElement = driver.findElement(By.id("offers-popover"));
String divText = popElement.findElement(By.className("description")).getAttribute("textContent");

JSoup check if <HTML>,<HEAD> and <BODY> tags are present

Hi I am using JSoup to parse a HTML file. After parsing, I want to check if the file contains the tag. I am using the following code to check that,
htmlDom = parser.parse("<p>My First Heading</p>clk");
Elements pe = htmlDom.select("html");
System.out.println("size "+pe.size());
The output I get is "size 1" even though there is no HTML tag present. My guess is that it is because the HTML tag is not mandatory and that it is implicit. Same is the case for Head and Body tag. Is there any way I could check for sure if these tags are present in the input file?
Thank you.

It does not return 1 because the tag is implicit, but because it is present in the Document object htmlDom after you have parsed the custom HTML.
That is because Jsoup will try to conform the HTML5 Parsing Rules, and thus adds missing elements and tries to fix a broken document structure. I'm quite sure you would get a 1 in return if you were to run the following aswell:
Elements pe = htmlDom.select("head");
System.out.println("size "+pe.size());
To parse the HTML without Jsoup trying to clean or make your HTML valid, you can instead use the included XMLParser, as below, which will parse the HTML as it is.
String customHtml = "<p>My First Heading</p>clk";
Document customDoc = Jsoup.parse(customHtml, "", Parser.xmlParser());
So, as opposed to your assumption in the comments of the question, this is very much possible to do with Jsoup.

How to get HTML tag After rendering the html on webpage using java or javascript or xslt

how to get html source code which was rendered by a javascript in webpage. How can i proceed this? Using xsl or javascript or java.

Get entire HTML in current page:
function getHTML(){
var D=document,h=D.getElementsByTagName('html')[0],e;
if(h.outerHTML)return h.outerHTML;
e=D.createElement('div');
e.appendChild(h.cloneNode(true));
return e.innerHTML;
}
outerHTML is non-standard property thus might not supported in some browser (i.e., Firefox), in this case this function mimic the outerHTML feature by cloning the html node into unattached element and read it's innerHTML property.

Javascript provides
document.getElementByTagName('')
You can get any tag from this line. Moreover if you want to do any operation to this tag then assign any id to that tag. then you can use document.getElementById('') to do any operation on it.
These will give you source code.

How to jump to specific page number using display tag in Spring framework

I want to jump to a specific page number using display tag with a textbox and a "go" button.
On the click of GO button calls a javascript in which it should go to that specific page through that .htm which is not happening.
please suggest an argument for this particular way of getting a specific page or else alternate suggestions are always welcome
Below are the arguments in displaytag.properties which i know so far
enter code here
{0}: numbered pages list
{1}: link to the first page
{2}: link to the previous page
{3}: link to the next page
{4}: link to the last page
{5}: current page
{6}: total number of pages
Below is the javascript function which is being called on click of GO button
function selectPage(){
alert("pageNo:" +document.portalDisplayform.selPageNo[0].value);
alert("pageNo:" +document.portalDisplayform.selPageNo[1].value);
var pageNo = document.portalDisplayform.selPageNo[0].value;
var pageNo = document.portalDisplayform.selPageNo[1].value;
document.portalDisplayform.action = '<%=request.getContextPath()% >'+"/portalAccessdisplay.htm?tokenId="+'<%=cachetoken%>'+pageNo;
document.portalDisplayform.submit();
}

It's not so easy, because the displaytag encodes parameters (to avoid naming conflicts with functional parameters), and uses a unique ID per table, in case several tables are on the same page. I suggest you download the source code of displaytag, have a look at the ParamEncoder and Pagination classes (and their callers) to discover how a link to a specific page is constructed by the displaytag. You'll have to use similar code to generate the URL, and you'll have to modify the value of the appropriate (encoded) parameter in your JavaScript code.

The very easiest way, you can modify the TableTag.java which is in the display-tag jar file. In this file modify the initParameters() method. Inside the method
place the below 4 line code.
After this line which is in initParameters() method.
this.pageNumber = (pageNumberParameter == null) ? 1 : pageNumberParameter.intValue();
Place the below code
if((request.getParameter("pageno") != null) && (request.getParameter("pageno") != ""))
{
this.pageNumber=Integer.parseInt(request.getParameter("pageno"))
}
And use a TextBox with the name pageno in your DisplayTag page. Also include the name in your DisplayTag property excludedparam.