As per the html the source code is:
{result.data}
While requesting the URL result.data is set with 100 and am able to see the value as 100 in the browser. Where as while I am trying to execute the java program with the same url request I am unable to see the value as I have seen in browser.
URL url = new URL(site)
url.openConnection() etc..
I wanted to get the same content as I have seen in the browser through java program.
Your question is not very descriptive, but I guess you are trying to scrape data from the site.
You can use the following libraries for this task:
Jaunt (http://jaunt-api.com)
Jsoup (http://jsoup.org/cookbook/extracting-data/dom-navigation)
HTMLUnit
To what i understand, you want to do one of the below things :
Instead of reading the result line by line, you want to parse it as an XML to as to traverse to div(s) and other html tags.
For this purpose i would suggest you to use jsoup library.
When you hit the URL: www.abcd.com/number=500 in browser, it loads an empty div and on load it fetches data from somewhere, this data which it fetches on load, you want to fetch this using java ?
For this, there must be some js in the resulting page, which is fetching data by hitting some service on page load, you will need to look up in the page to know the service details and instead of hitting this URL (www.abcd.com/number=500) you will need to hit that service to get data.
Related
I am trying to build my new App and I need some data for it.
so I was wondering if I can get JSON data (or query link or any source file) of any site for now and for future.
I need to get data from this site:
https://study.ekb.eg/ - https://www.ekb.eg/
Note: https://www.ekb.eg/ is the main source data for https://study.ekb.eg/
Thanks
The site needs to be specifically programmed to return JSON data for you. Regular, normal sites are not like this. They generally just return HTML
So the site owner would need to set up an API for you to consume
I have a project that requires me to use JSOUP for web scraping. I was able to get the data from the main page of the website that I want to scrape. but, as I scrape deeper into the page by looping into the hyperlink and accessing it, I get the following errors:
java.io.IOException: Input is binary and unsupported
at org.jsoup.UncheckedIOException.<init>(UncheckedIOException.java:11)
at org.jsoup.parser.CharacterReader.<init>(CharacterReader.java:38)
at org.jsoup.parser.CharacterReader.<init>(CharacterReader.java:43)
at org.jsoup.parser.TreeBuilder.initialiseParse(TreeBuilder.java:38)
at org.jsoup.parser.HtmlTreeBuilder.initialiseParse(HtmlTreeBuilder.java:65)
at org.jsoup.parser.TreeBuilder.parse(TreeBuilder.java:46)
at org.jsoup.parser.Parser.parseInput(Parser.java:35)
at org.jsoup.helper.DataUtil.parseInputStream(DataUtil.java:169)
at org.jsoup.helper.HttpConnection$Response.parse(HttpConnection.java:835)
at org.jsoup.helper.HttpConnection.get(HttpConnection.java:285)
when I inspect the website, there are parts of the website that contains a commented binary data and I think it caused the problem. I've tried using this code:
Document docs2 = Jsoup.connect("https://www.kiatravels.co.id/group_tour/index?TOUR_ID=1467&ID=15803").ignoreContentType(true).get();
but still didn't work.
Here's hoping some brainy code master can help!
It looks like you navigated to the "Download Itinerary" link, which opens a pdf. Before parsing the link with Jsoup, you'll want to check the content-type of the url response.
Connection.Response res = Jsoup.connect(url).execute();
String contentType = res.contentType();
You'll probably want to ignore MIME types that are not text/html.
I have a URI that returns 5 pages while I run it on Google Chrome REST API Client.
I have to write a code in java to read all the pages and create a JSON File. Simple solution is to give the page size(attribute) in the URI as 1000 and extract all the 5 pages. But, the requirement is that - I have to check if there is any "NEXT" keyword present in the result obtained after running the URI. And if, the "Next" keyword is found, I need to extract the link from that line and run it to get the result from next page.
JSON file
{"PageSize":100,"PageNumber":1,"TotalRecords":423,"TotalPages":5,
"Items":
[{"CampaignID":1,
"Created":"2013-10-23T23:00:00Z",
"ScheduledDate":"2013-10-24T00:15:00Z",
"Name":"Halloween Deals Campaign",
"Status":"Sent",
"Links":[{"rel":"self","href":"link1"}]},
"Links":[{"rel":"self","href":"link2"},
{"rel":
"first","href":"https:xyz"},
{"rel":"last","href":"link3"},
{"rel":"next","href":"https://rest.emaildirect.com/v1/Campaigns/
All?PageSize=100&PageNumber=2"}]}
I have stream of shortened URLs as received from twitter feeds. I don't need entire page of that URL, but basic meta information like expanded original URL, page title, timestamps etc. I can get the entire page containing these meta as well using curl,wget, but any quicker way to only get the meta? Also, is there any java classes/methods to do this like curl.
HTTP Head requests may be what you are looking for, here is an example that uses curl (in its Python implementation though): Python: Convert those TinyURL (bit.ly, tinyurl, ow.ly) to full URLS
I am trying to scrape data from a website which uses javascript to load much of their content. Right now I am using jSoup to parse html pages, however since much of the content is loaded using javascript I haven't been able to parse the data I want.
How should I go about getting this javascript content? Should I first save the page then load and parse it using jSoup? If so, what should I use to load javascript content before I save? Is there an API which you would recommend that could output html?
Currently using java.
You might be interested in checking out pjscrape (disclaimer: this is my project). It's a command-line tool using PhantomJS to allow scraping using JavaScript and jQuery in a full browser context - among other things, you can define a "ready" function for the page and wait to scrape until the function (which might check for the existence of certain DOM elements, etc) returns true.
The other option, depending on the page, is to use a console like Firebug to figure out what data is being loaded (i.e. what URLs are being retrieved by the AJAX calls on the page), and scrape the data directly from those URLs.
If the data are generated with javascript then the data are in the downloaded page.
Better is to directly parse them on the fly as you do with plain HTML or Text parsing.
If you cannot isolate tokens with jSoup API just parse them using direct String options, as a plain text.
I tried using htmlUnit however I found it very slow.
I ended up using the curl command line function within java which worked for my purposes.
String command = "curl "+url;
Process p = Runtime.getRuntime().exec(command);
BufferedReader stdInput = new BufferedReader(new InputStreamReader(p.getInputStream()));
while ((s = stdInput.readLine()) != null) {
html = html+s+"\n";
}
return html;