We Keep Coding

Scrape information from Web Pages with Java?

I'm trying to extract data from a webpage, for example, lets say I wish to fetch information from chess.org.
I know the player's ID is 25022, which means I can request
http://www.chess.org.il/Players/Player.aspx?Id=25022
In that page I can see that this player's fide ID = 2821109.
From that, I can request this page:
http://ratings.fide.com/card.phtml?event=2821109
And from that I can see that stdRating=1602.
How can I get the "stdRating" output from a given "localID" input in Java?
(localID, fideID and stdRating are aid parameters that I use to clarify the question)

You could try the univocity-html-parser, which is very easy to use and avoids a lot of spaghetti code.
To get the standard rating for example you can use this code:
public static void main(String... args) {
UrlReaderProvider url = new UrlReaderProvider("http://ratings.fide.com/card.phtml?event={EVENT}");
url.getRequest().setUrlParameter("EVENT", 2821109);
HtmlElement doc = HtmlParser.parseTree(url);
String rating = doc.query()
.match("small").withText("std.")
.match("br").getFollowingText()
.getValue();
System.out.println(rating);
}
Which produces the value 1602.
But getting data by querying individual nodes and trying to stitch all pieces together is not exactly easy.
I expanded the code to illustrate how you can use the parser to get more information into records. Here I created records for the player and her rank details which are available in the table of the second page. It took me less than 1h to get this done:
public static void main(String... args) {
UrlReaderProvider url = new UrlReaderProvider("http://www.chess.org.il/Players/Player.aspx?Id={PLAYER_ID}");
url.getRequest().setUrlParameter("PLAYER_ID", 25022);
HtmlEntityList entities = new HtmlEntityList();
HtmlEntitySettings player = entities.configureEntity("player");
player.addField("id").match("b").withExactText("מספר שחקן").getFollowingText().transform(s -> s.replaceAll(": ", ""));
player.addField("name").match("h1").followedImmediatelyBy("b").withExactText("מספר שחקן").getText();
player.addField("date_of_birth").match("b").withExactText("תאריך לידה:").getFollowingText();
player.addField("fide_id").matchFirst("a").attribute("href", "http://ratings.fide.com/card.phtml?event=*").getText();
HtmlLinkFollower playerCard = player.addField("fide_card_url").matchFirst("a").attribute("href", "http://ratings.fide.com/card.phtml?event=*").getAttribute("href").followLink();
playerCard.addField("rating_std").match("small").withText("std.").match("br").getFollowingText();
playerCard.addField("rating_rapid").match("small").withExactText("rapid").match("br").getFollowingText();
playerCard.addField("rating_blitz").match("small").withExactText("blitz").match("br").getFollowingText();
playerCard.setNesting(Nesting.REPLACE_JOIN);
HtmlEntitySettings ratings = playerCard.addEntity("ratings");
configureRatingsBetween(ratings, "World Rank", "National Rank ISR", "world");
configureRatingsBetween(ratings, "National Rank ISR", "Continent Rank Europe", "country");
configureRatingsBetween(ratings, "Continent Rank Europe", "Rating Chart", "continent");
Results<HtmlParserResult> results = new HtmlParser(entities).parse(url);
HtmlParserResult playerData = results.get("player");
String[] playerFields = playerData.getHeaders();
for(HtmlRecord playerRecord : playerData.iterateRecords()){
for(int i = 0; i < playerFields.length; i++){
System.out.print(playerFields[i] + ": " + playerRecord.getString(playerFields[i]) +"; ");
}
System.out.println();
HtmlParserResult ratingData = playerRecord.getLinkedEntityData().get("ratings");
for(HtmlRecord ratingRecord : ratingData.iterateRecords()){
System.out.print(" * " + ratingRecord.getString("rank_type") + ": ");
System.out.println(ratingRecord.fillFieldMap(new LinkedHashMap<>(), "all_players", "active_players", "female", "u16", "female_u16"));
}
}
}
private static void configureRatingsBetween(HtmlEntitySettings ratings, String startingHeader, String endingHeader, String rankType) {
Group group = ratings.newGroup()
.startAt("table").match("b").withExactText(startingHeader)
.endAt("b").withExactText(endingHeader);
group.addField("rank_type", rankType);
group.addField("all_players").match("tr").withText("World (all", "National (all", "Rank (all").match("td", 2).getText();
group.addField("active_players").match("tr").followedImmediatelyBy("tr").withText("Female (active players):").match("td", 2).getText();
group.addField("female").match("tr").withText("Female (active players):").match("td", 2).getText();
group.addField("u16").match("tr").withText("U-16 Rank (active players):").match("td", 2).getText();
group.addField("female_u16").match("tr").withText("Female U-16 Rank (active players):").match("td", 2).getText();
}
The output will be:
id: 25022; name: יעל כהן; date_of_birth: 02/02/2003; fide_id: 2821109; rating_std: 1602; rating_rapid: 1422; rating_blitz: 1526;
* world: {all_players=195907, active_players=94013, female=5490, u16=3824, female_u16=586}
* country: {all_players=1595, active_players=1024, female=44, u16=51, female_u16=3}
* continent: {all_players=139963, active_players=71160, female=3757, u16=2582, female_u16=372}
Hope it helps
Disclosure: I'm the author of this library. It's commercial closed source but it can save you a lot of development time.

As #Alex R pointed out, you'll need a Web Scraping library for this.
The one he recommended, JSoup, is quite robust and is pretty commonly used for this task in Java, at least in my experience.
You'd first need to construct a document that fetches your page, eg:
int localID = 25022; //your player's ID.
Document doc = Jsoup.connect("http://www.chess.org.il/Players/Player.aspx?Id=" + localID).get();
From this Document Object, you can fetch a lot of information, for example the FIDE ID you requested, unfortunately the web page you linked inst very simple to scrape, and you'll need to basically go through every link on the page to find the relevant link, for example:
Elements fidelinks = doc.select("a[href*=fide.com]");
This Elements object should give you a list of all links that link to anything containing the text fide.com, but you probably only want the first one, eg:
Element fideurl = doc.selectFirst("a[href=*=fide.com]");
From that point on, I don't want to write all the code for you, but hopefully this answer serves as a good starting point!
You can get the ID alone by calling the text() method on your Element object, but You can also get the link itself by just calling Element.attr('href')
The css selector you can use to get the other value is
div#main-col table.contentpaneopen tbody tr td table tbody tr td table tbody tr:nth-of-type(4) td table tbody tr td:first-of-type, which will get you the std score specifically, at least with standard css, so this should work with jsoup as well.

Open graph fetching meta data

I am using open graph library to fetch the metadata from url.
I am getting the title and description from url link which follow og tag rules. How to get metadata from url link which don't follow og tag.
my simple code :
OpenGraph data = new OpenGraph(url, true);
response.setDescription(data.getContent("description"));
response.setMetaDataImage(data.getContent("image"));
response.setTitle(data.getContent("title"));
response.setMetaDataUrl(data.getContent("url"));
Data fetch is null.

I think you're talking about this library. If so, the boolean in the constructor serves the purpose:
public OpenGraph(String url, boolean ignoreSpecErrors) {
...
}
The way I use this library to fetch, for example, images is as follows:
OpenGraph og = new OpenGraph(url, true);
MetaElement[] imageElements = og.getProperties("image");
Perhaps you are just using the wrong getter? If the page has og tags, this snippet should work!

I had similar issues with opengraph-java (ie. getting null response).
I tried the example in the docs, but the response was null
OpenGraph movie = new OpenGraph("http://www.rottentomatoes.com/m/back_to_the_future/", true);
System.out.println("movie = " + movie)); // movie = null
Trying the false option for ignoreSpecErrors throws an exception java.lang.Exception: Does not conform to Open Graph protocol
So I made a library called ogmapper that's a little more flexible.
DefaultOgMapper ogMapper = new JsoupOgMapperFactory().build();
OgTags ogTags = ogMapper.process(new URL("http://www.rottentomatoes.com/m/back_to_the_future/"));
System.out.println("title = " + ogTags.getTitle()); // title = Back to the Future (1985)
Hopefully this is helpful!

Html Slurping in Groovy

I am trying to parse HTML that comes to me as a giant String. When I get to Line 13, NodeChild page = it.parent()
I am able to find the key that I am looking for, but the data comes to me like This Is Value One In My KeyThis is Value Two in my KeyThis is Value Three In My Key and so on. I see a recurring trend where the seperator between the two is always UppercaseUppercase (withoutSpaces).
I would like to put it into an ArrayList one way or another. Is there a method that I am missing from the docs that is able to automatically do this? Is there a better way to parse this together?
class htmlParsingStuff{
private def slurper = new XmlSlurper(new Parser())
private void slurpItUp(String rawHTMLString){
ArrayList urlList = []
def htmlParser = slurper.parseText(rawHTMLString)
htmlParser.depthFirst().findAll() {
//Loop through all of the HTML Tags to get to the key that I am looking for
//EDIT: I see that I am able to iterate through the parent object, I just need a way to figure out how to get into that object
boolean trigger = it.text() == 'someKey'
if (trigger){
//I found the key that I am looking for
NodeChild page = it.parent()
page = page.replace('someKey', '')
LazyMap row = ["page": page, "type": "Some Type"]
urlList.add(row)
}
}
}
}

I can't provide you with working code since I don't know your specific html.
But: don't use XmlSlurper for parsing HTML, HTML is not well formed and therefor XmlSlurper is not the right tool for the job.
For HTML use a library like JSoup. You will find it much easier to use especially if you have some JQuery knowledge. Since you didn't post your HTML snippet I made up my own example:
#Grab(group='org.jsoup', module='jsoup', version='1.10.1')
import org.jsoup.Jsoup
def html = """
<html>
<body>
<table>
<tr><td>Key 1</td></tr>
<tr><td>Key 2</td></tr>
<tr><td>Key 3</td></tr>
<tr><td>Key 4</td></tr>
<tr><td>Key 5</td></tr>
</table>
</body>
</html>"""
def doc = Jsoup.parse(html)
def elements = doc.select('td')
def result = elements.collect {it.text()}
// contains ['Key 1', 'Key 2', 'Key 3', 'Key 4', 'Key 5']
To manipulate the document you would use
def doc = Jsoup.parse(html)
def elements = doc.select('td')
elements.each { oldElement ->
def newElement = new Element(Tag.valueOf('td'), '')
newElement.text('Another key')
oldElement.replaceWith(newElement)
}
println doc.outerHtml()

I want to pull Facebook posts from a public page to a Java application

I am creating an app in Java that will take all the information from a public website and load it in the app for people to read using jsoup. I was trying the same kind of function with Facebook but it wasn't working the same way. Does anyone have a good idea about how I should go about this?
Thanks,
Calland
public String[] scrapeEvents(String... args) throws Exception {
Document doc = Jsoup.connect("http://www.facebook.com/cedarstreettimes?fref=ts").get();
Elements elements = doc.select("div._wk");
String s = elements.toString();
return s;
}
edit: I found this link of information,but I'm a little confused on how to manipulate it to get me only the content of what the specific user posts on their wall. http://developers.facebook.com/docs/getting-started/graphapi/

I had a look at the source of that page -- the thing that is tripping up the parse is that all the real content is wrapped in comments, like this:
<code class="hidden_elem" id="u_0_42"><!-- <div class="fbTimelineSection ...> --></code>
There is JS on the page that lifts that data into the real DOM, but as jsoup doesn't execute JS it stays as comments. So before extracting the content, we need to emulate that JS and "un-hide" those elements. Here's an example to get you started:
String url = "https://www.facebook.com/cedarstreettimes?fref=ts";
String ua = "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_8_2) AppleWebKit/537.33 (KHTML, like Gecko) Chrome/27.0.1438.7 Safari/537.33";
Document doc = Jsoup.connect(url).userAgent(ua).timeout(10*1000).get();
// move the hidden commented out html into the DOM proper:
Elements hiddenElements = doc.select("code.hidden_elem");
for (Element hidden: hiddenElements) {
for (Node child: hidden.childNodesCopy()) {
if (child instanceof Comment) {
hidden.append(((Comment) child).getData()); // comment data parsed as html
}
}
}
Elements articles = doc.select("div[role=article]");
for (Element article: articles) {
if (article.select("span.userContent").size() > 0) {
String text = article.select("span.userContent").text();
String imgUrl = article.select("div.photo img").attr("abs:src");
System.out.println(String.format("%s\n%s\n\n", text,imgUrl));
}
}
That example pulls out the article text and any photo that is associated with it.
(It's possibly better to use the FB API that this method; I wanted to show how you can emulate little bits of JS to make a scrape work properly.)

How to get the embed HTML code for a video hosted in youtube programmatically

How to get the embed HTML code for a video hosted in youtube programmatically. What Java API is available

Use the YouTube Data API (there's pre-built GData client libraries, or you can do the HTTP/XML stuff yourself).
One of the <media:content/> entries will contain a URL for the embeddable SWF, if the video is embeddable.

Assuming you have the URL of the video, it's fairly simple to generate one. You need the end of the URL (the part after the /watch?v=, let's call it ID). To generate the iframe embed html, just place it in the appropriate place (in the src attribute, don't include the brackets):
<iframe title="YouTube video player" class="youtube-player" type="text/html" width="640"
height="390" src="http://www.youtube.com/embed/{ID}" frameborder="0"
allowFullScreen></iframe>
There are a couple of ways to get the v parameter from the URL. A regular expression would work.

Though the accepted answer works, if you want to do this programmatically you need the correct aspect ratio in order to generate optimal iframe dimensions for your video. I wrote the following php function that can generate a link for you on the fly. It uses the bash utility youtube-dl to get information about the video from any youtube link, so you'll need to make sure that's installed (apt-get install youtube-dl should work on Ubuntu or other debian flavors)
function getYoutubeEmbed($link, $size = [], $options = [], $privacy = false) {
$options += [
'rel' => true, // Show suggested videos when the video finishes.
'controls' => true, // Show player controls.
'showinfo' => true, // Show video title and player actions.
];
$json = json_decode(exec('youtube-dl -j --no-warnings ' . $link . ' 2>/dev/null'));
if ($json && !empty($id = $json->id) && !empty($width = $json->width) && !empty($height = $json->height)) {
$args = [];
foreach ($options as $option => $value) {
if (!$value) {
$args[] = $option . '=0';
}
}
if ($size) {
if (!empty($size['width']) && !empty($size['height'])) {
$width = $size['width'];
$height = $size['height'];
} else if (!empty($size['width'])) {
$height = ceil(($height * $size['width']) / $width);
$width = $size['width'];
} else if (!empty($size['height'])) {
$width = ceil(($width * $size['height']) / $height);
$height = $size['height'];
}
}
$url = ($privacy ? 'www.youtube-nocookie.com/embed/' : 'www.youtube.com/embed/') . $id . ($args ? '?' . implode('&',$args) : '');
$iframe = '<iframe width="' . $width . '" height="' . $height . '" src="//' . $url . '" frameborder="0" allow="autoplay; encrypted-media" allowfullscreen></iframe>';
return $iframe;
} else {
return false;
}
}
The function is fairly self explanatory but here's the breakdown:
At the minimum you need to supply a link for the first argument.
The second argument is an array of width, height or both. If you only specify one it will treat keep the default aspect ratio and calculate the other dimension for you (this is how I'd typically use it).
The third argument is an optional array of arguments which are documented in the function itself.
The fourt is an optional boolean argument for 'privacy' which is explained as:
Enable privacy-enhanced mode. When you turn on privacy-enhanced mode, YouTube won't store information about visitors on your website unless they play the video.
Usage example:
$link = 'https://www.youtube.com/watch?v=adAqQct3vRI';
echo getYoutubeEmbed($link, ['width' => 560], ['rel' => false]);
Output:
<iframe width="560" height="315" src="//www.youtube.com/embed/605gdJGdaPE?rel=0" frameborder="0" allow="autoplay; encrypted-media" allowfullscreen></iframe>