We Keep Coding

Scrape information from Web Pages with Java?

I'm trying to extract data from a webpage, for example, lets say I wish to fetch information from chess.org.
I know the player's ID is 25022, which means I can request
http://www.chess.org.il/Players/Player.aspx?Id=25022
In that page I can see that this player's fide ID = 2821109.
From that, I can request this page:
http://ratings.fide.com/card.phtml?event=2821109
And from that I can see that stdRating=1602.
How can I get the "stdRating" output from a given "localID" input in Java?
(localID, fideID and stdRating are aid parameters that I use to clarify the question)

You could try the univocity-html-parser, which is very easy to use and avoids a lot of spaghetti code.
To get the standard rating for example you can use this code:
public static void main(String... args) {
UrlReaderProvider url = new UrlReaderProvider("http://ratings.fide.com/card.phtml?event={EVENT}");
url.getRequest().setUrlParameter("EVENT", 2821109);
HtmlElement doc = HtmlParser.parseTree(url);
String rating = doc.query()
.match("small").withText("std.")
.match("br").getFollowingText()
.getValue();
System.out.println(rating);
}
Which produces the value 1602.
But getting data by querying individual nodes and trying to stitch all pieces together is not exactly easy.
I expanded the code to illustrate how you can use the parser to get more information into records. Here I created records for the player and her rank details which are available in the table of the second page. It took me less than 1h to get this done:
public static void main(String... args) {
UrlReaderProvider url = new UrlReaderProvider("http://www.chess.org.il/Players/Player.aspx?Id={PLAYER_ID}");
url.getRequest().setUrlParameter("PLAYER_ID", 25022);
HtmlEntityList entities = new HtmlEntityList();
HtmlEntitySettings player = entities.configureEntity("player");
player.addField("id").match("b").withExactText("מספר שחקן").getFollowingText().transform(s -> s.replaceAll(": ", ""));
player.addField("name").match("h1").followedImmediatelyBy("b").withExactText("מספר שחקן").getText();
player.addField("date_of_birth").match("b").withExactText("תאריך לידה:").getFollowingText();
player.addField("fide_id").matchFirst("a").attribute("href", "http://ratings.fide.com/card.phtml?event=*").getText();
HtmlLinkFollower playerCard = player.addField("fide_card_url").matchFirst("a").attribute("href", "http://ratings.fide.com/card.phtml?event=*").getAttribute("href").followLink();
playerCard.addField("rating_std").match("small").withText("std.").match("br").getFollowingText();
playerCard.addField("rating_rapid").match("small").withExactText("rapid").match("br").getFollowingText();
playerCard.addField("rating_blitz").match("small").withExactText("blitz").match("br").getFollowingText();
playerCard.setNesting(Nesting.REPLACE_JOIN);
HtmlEntitySettings ratings = playerCard.addEntity("ratings");
configureRatingsBetween(ratings, "World Rank", "National Rank ISR", "world");
configureRatingsBetween(ratings, "National Rank ISR", "Continent Rank Europe", "country");
configureRatingsBetween(ratings, "Continent Rank Europe", "Rating Chart", "continent");
Results<HtmlParserResult> results = new HtmlParser(entities).parse(url);
HtmlParserResult playerData = results.get("player");
String[] playerFields = playerData.getHeaders();
for(HtmlRecord playerRecord : playerData.iterateRecords()){
for(int i = 0; i < playerFields.length; i++){
System.out.print(playerFields[i] + ": " + playerRecord.getString(playerFields[i]) +"; ");
}
System.out.println();
HtmlParserResult ratingData = playerRecord.getLinkedEntityData().get("ratings");
for(HtmlRecord ratingRecord : ratingData.iterateRecords()){
System.out.print(" * " + ratingRecord.getString("rank_type") + ": ");
System.out.println(ratingRecord.fillFieldMap(new LinkedHashMap<>(), "all_players", "active_players", "female", "u16", "female_u16"));
}
}
}
private static void configureRatingsBetween(HtmlEntitySettings ratings, String startingHeader, String endingHeader, String rankType) {
Group group = ratings.newGroup()
.startAt("table").match("b").withExactText(startingHeader)
.endAt("b").withExactText(endingHeader);
group.addField("rank_type", rankType);
group.addField("all_players").match("tr").withText("World (all", "National (all", "Rank (all").match("td", 2).getText();
group.addField("active_players").match("tr").followedImmediatelyBy("tr").withText("Female (active players):").match("td", 2).getText();
group.addField("female").match("tr").withText("Female (active players):").match("td", 2).getText();
group.addField("u16").match("tr").withText("U-16 Rank (active players):").match("td", 2).getText();
group.addField("female_u16").match("tr").withText("Female U-16 Rank (active players):").match("td", 2).getText();
}
The output will be:
id: 25022; name: יעל כהן; date_of_birth: 02/02/2003; fide_id: 2821109; rating_std: 1602; rating_rapid: 1422; rating_blitz: 1526;
* world: {all_players=195907, active_players=94013, female=5490, u16=3824, female_u16=586}
* country: {all_players=1595, active_players=1024, female=44, u16=51, female_u16=3}
* continent: {all_players=139963, active_players=71160, female=3757, u16=2582, female_u16=372}
Hope it helps
Disclosure: I'm the author of this library. It's commercial closed source but it can save you a lot of development time.

As #Alex R pointed out, you'll need a Web Scraping library for this.
The one he recommended, JSoup, is quite robust and is pretty commonly used for this task in Java, at least in my experience.
You'd first need to construct a document that fetches your page, eg:
int localID = 25022; //your player's ID.
Document doc = Jsoup.connect("http://www.chess.org.il/Players/Player.aspx?Id=" + localID).get();
From this Document Object, you can fetch a lot of information, for example the FIDE ID you requested, unfortunately the web page you linked inst very simple to scrape, and you'll need to basically go through every link on the page to find the relevant link, for example:
Elements fidelinks = doc.select("a[href*=fide.com]");
This Elements object should give you a list of all links that link to anything containing the text fide.com, but you probably only want the first one, eg:
Element fideurl = doc.selectFirst("a[href=*=fide.com]");
From that point on, I don't want to write all the code for you, but hopefully this answer serves as a good starting point!
You can get the ID alone by calling the text() method on your Element object, but You can also get the link itself by just calling Element.attr('href')
The css selector you can use to get the other value is
div#main-col table.contentpaneopen tbody tr td table tbody tr td table tbody tr:nth-of-type(4) td table tbody tr td:first-of-type, which will get you the std score specifically, at least with standard css, so this should work with jsoup as well.

Bing Search API error using odata4j

I am trying to run Bing search API. I used odata4j and tried the code provided here:
How to use Bing search api in Java
ODataConsumer c = ODataConsumers
.newBuilder("https://api.datamarket.azure.com/Bing/Search")
.setClientBehaviors(OClientBehaviors.basicAuth("accountKey", "{your account key here}"))
.build();
OQueryRequest<OEntity> oRequest = c.getEntities("Web")
.custom("Query", "stackoverflow bing api");
Enumerable<OEntity> entities = oRequest.execute();
After I registered in the bing service, I obtained the key and placed it inside the double quotation in the above code. I got the following error:
Exception in thread "main" java.lang.RuntimeException: Expected status OK, found Bad Request. Server response:
Parameter: Query is not of type String
at org.odata4j.jersey.consumer.ODataJerseyClient.doRequest(ODataJerseyClient.java:165)
at org.odata4j.consumer.AbstractODataClient.getEntities(AbstractODataClient.java:69)
at org.odata4j.consumer.ConsumerQueryEntitiesRequest.doRequest(ConsumerQueryEntitiesRequest.java:59)
at org.odata4j.consumer.ConsumerQueryEntitiesRequest.getEntries(ConsumerQueryEntitiesRequest.java:50)
at org.odata4j.consumer.ConsumerQueryEntitiesRequest.execute(ConsumerQueryEntitiesRequest.java:40)
at BingAPI.main(BingAPI.java:20)
Caused by: org.odata4j.exceptions.UnsupportedMediaTypeException: Unknown content type text/plain;charset=utf-8
at org.odata4j.format.FormatParserFactory.getParser(FormatParserFactory.java:78)
at org.odata4j.jersey.consumer.ODataJerseyClient.doRequest(ODataJerseyClient.java:161)
... 5 more
I could not figure out the problem.

All what you need is to set your query like that %27stackoverflow bing api%27
Here is my source code:
ODataConsumer consumer = ODataConsumers
.newBuilder("https://api.datamarket.azure.com/Bing/Search/v1/")
.setClientBehaviors(
OClientBehaviors.basicAuth("accountKey",
"{My Account ID}"))
.build();
System.out.println(consumer.getServiceRootUri() + consumer.toString());
OQueryRequest<OEntity> oQueryRequest = consumer.getEntities("Web")
.custom("Query", "%27stackoverflow%27");
System.out.println("oRequest : " + oQueryRequest);
Enumerable<OEntity> entities = oQueryRequest.execute();
System.out.println(entities.elementAt(0));

You can further try different queries with different filters by keep adding name-value pairs by using .custom("Type of parameter","parameter") of the oQueryrequest object.Say you want to search for indian food but you want only small square images.
ODataConsumer consumer = ODataConsumers
.newBuilder("https://api.datamarket.azure.com/Bing/Search/v1/")
.setClientBehaviors(
OClientBehaviors.basicAuth("accountKey",
"YOUR ACCOUNT KEY"))
.build();
System.out.println(consumer.getServiceRootUri() + consumer.toString());
OQueryRequest<OEntity> oQueryRequest = consumer.getEntities("Image")
.custom("Query", "%27indian food%27");
oQueryRequest.custom("Adult", "%27Moderate%27");
oQueryRequest.custom("ImageFilters", "%27Size:Small+Aspect:Square%27");
System.out.println("oRequest : " + oQueryRequest);
Enumerable<OEntity> entities = oQueryRequest.execute();
int count = 0;
Iterator<OEntity> iter = entities.iterator();
System.out.println(iter.next());

Is there a limit on the number of comments to be extracted from Youtube?

I am trying to extract comments on some YouTubeVideos using the youtube-api with Java. Everything is going fine except the fact that I am not able to extract all the comments if the video has a large number of comments (it stops at somewhere in between 950 and 999). I am following a simple method of paging through the CommentFeed of the VideoEntry, getting comments on each page and then storing each comment in an ArrayList before writing them in an XML file. Here is my code for retrieving the comments
int commentCount = 0;
CommentFeed commentFeed = service.getFeed(new URL(commentUrl), CommentFeed.class);
do {
//Gets each comment in the current feed and prints to the console
for(CommentEntry comment : commentFeed.getEntries()) {
commentCount++;
System.out.println("Comment " + commentCount + " plain text content: " + comment.getPlainTextContent());
}
//Checks if there is a next page of comment feeds
if (commentFeed.getNextLink() != null) {
commentFeed = service.getFeed(new URL(commentFeed.getNextLink().getHref()), CommentFeed.class);
}
else {
commentFeed = null;
}
}
while (commentFeed != null);
My question is: Is there some limit on the number of comments that I could extract or am I doing something wrong?

use/refer this
String commentUrl = videoEntry.getComments().getFeedLink().getHref();
CommentFeed commentFeed = service.getFeed(new URL(commentUrl), CommentFeed.class);
for(CommentEntry comment : commentFeed.getEntries()) {
System.out.println(comment.getPlainTextContent());
}
source
Max number of results per iteration is 50 (it seems) as mentioned here
and you can use start-index to retrieve multiple result sets as mentioned here

Google Search API and as well as Youtube Comments search limits max. 1000 results and u cant extract more than 1000 results

How to get h2 Tag of a table using Jsoup

I need some help scraping a webpage with Jsoup. I want to pars player profiles from the hcfactions webpage and gather their kills and deaths. The problem I'm running into is that each profile page is dynamically created and will only have said tables if the player has kills or deaths. So in order to tell which table I'm parsing I need to get the header text that's set after the call.
example web page: http://www.hcfactions.net/index.php?action=playerinfo&player=Djmaddox.
Below is a html segment from the web page I'm scraping:
<table class='table-bordered'><h2 style='text-align:center'>Deaths</h2>
<tr><td>Date</td><td>Reason</td><td>Details</td></tr><tr><td>Dec 11 5:27pm CST</td>.....
I have this code that pulls the tables and counts entries but it wont pull the h2 tags with it for me to select.
public void getPlayerDetails(String name) {
String data = "";
Avatar temp = _db.getPlayer(name);
playerUrl = "http://www.hcfactions.net/index.php?action=playersearch&player=" + name;
try {
// data = Jsoup.connect(url)
// .url(url).get().html();
playerDoc = Jsoup.connect(playerUrl).get();
} catch (IOException ex) {
Logger.getLogger(JParser.class.getName()).log(Level.SEVERE, null, ex);
}
if (playerDoc.select("table").size() == 1) {
return;
} else if (playerDoc.select("table").size() >= 2) {
for (int x = 1; x < playerDoc.select("table").size(); x++) {
System.out.println("deaths");
Element table = playerDoc.select("table").get(x);
Iterator<Element> ite = table.select("tr").iterator();
int count = 0;
while (ite.hasNext()) {
data = ite.next().text();
count++;
}
if (count > 0) {
temp.setDeaths(count - 1);
}
}
}
}

The tag <h2> is on an invalid position. That's why JSoup cannot find it I think. You have to extract it yourself with regular expressions. You can get the content of the <h2> with the following code:
String tableToString = "<table class='table-bordered'><h2 style='text-align:center'>Deaths</h2>" + "<tr>" + "<td>Date</td>" + "<td>Reason</td>" + "<td>Details</td>" + "</tr>" + "</table>";
String regex = "<h2.*>(.*)?</h2>";
Pattern pattern = Pattern.compile(regex);
Matcher matcher = pattern.matcher(tableToString);
if (matcher.find()) {
System.out.println(matcher.group(1));
}
You can init tableToString with table.toString() from your code.

As ka3ak says, the <h2> is mispositioned. But you don't have to abandon your parser as resort to regex for that. Assuming JSoup is a decent HTML parser (never used it myself) the <h2> element should be the element immediately preceding the <table> element. Get your 'select' statement to look for it there.

Elements headers=playerDoc.select("div.span10.offset1 h2");
IMHO Your selections seams to be little bit overcomplicated, but maybe it has to be like that. Anyway snippet above will get you every H2 tags present in proper container.
Later on you ca select required tables like that Elements tables=playerDoc.select("div.span10.offset1 table"); and apply proper data digging onto them. Headers will be in corresponding order to tables ofc. I think, that my job is done here :)

how to run the same step again without datasource in soapui

I don't have soap ui pro. I am testing the web service. The actual implementation is i need pass one error code on the request, and the corresponding error description should be displayed on the response. I need to add this assertion. Every time the description in the response varies.
Here is the thing i want exactly...
Every time i need to run the same request but the error code (which is input) only should be changed on each time and the description varies on the response. How to validate this? Is there any way to do this without data source.
Regards,
Chandra

This is the way i have created.. is there any way to improve the code to do better way;
import java.io.File;
File file = new File('c:/customers.csv');
InputStream inputFile = new FileInputStream(file);
String[] lines = inputFile.text.split('\n');
List<String[]> rows = lines.collect {it.split(',')}
log.info('There are ' + rows.size() + ' customers to be inserted');
for(int i = 0; i < rows.size(); i++) {
String[] row = rows.get(i);
String errorcode = row[0];
// log.info(errorcode)
String errorDescription = row[1];
//log.info(errorDescription)
testRunner.testCase.testSuite.project.setPropertyValue('errorcode', errorcode);
testRunner.testCase.testSuite.project.setPropertyValue('errorDescription', errorDescription);
testRunner.runTestStepByName("createCard-1");
log.info(errorcode +"Finsihed")
}