Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
We don’t allow questions seeking recommendations for books, tools, software libraries, and more. You can edit the question so it can be answered with facts and citations.
Closed 3 years ago.
Improve this question
Does anyone know if and how it is possible to search Google programmatically - especially if there is a Java API for it?
Some facts:
Google offers a public search webservice API which returns JSON: http://ajax.googleapis.com/ajax/services/search/web. Documentation here
Java offers java.net.URL and java.net.URLConnection to fire and handle HTTP requests.
JSON can in Java be converted to a fullworthy Javabean object using an arbitrary Java JSON API. One of the best is Google Gson.
Now do the math:
public static void main(String[] args) throws Exception {
String google = "http://ajax.googleapis.com/ajax/services/search/web?v=1.0&q=";
String search = "stackoverflow";
String charset = "UTF-8";
URL url = new URL(google + URLEncoder.encode(search, charset));
Reader reader = new InputStreamReader(url.openStream(), charset);
GoogleResults results = new Gson().fromJson(reader, GoogleResults.class);
// Show title and URL of 1st result.
System.out.println(results.getResponseData().getResults().get(0).getTitle());
System.out.println(results.getResponseData().getResults().get(0).getUrl());
}
With this Javabean class representing the most important JSON data as returned by Google (it actually returns more data, but it's left up to you as an exercise to expand this Javabean code accordingly):
public class GoogleResults {
private ResponseData responseData;
public ResponseData getResponseData() { return responseData; }
public void setResponseData(ResponseData responseData) { this.responseData = responseData; }
public String toString() { return "ResponseData[" + responseData + "]"; }
static class ResponseData {
private List<Result> results;
public List<Result> getResults() { return results; }
public void setResults(List<Result> results) { this.results = results; }
public String toString() { return "Results[" + results + "]"; }
}
static class Result {
private String url;
private String title;
public String getUrl() { return url; }
public String getTitle() { return title; }
public void setUrl(String url) { this.url = url; }
public void setTitle(String title) { this.title = title; }
public String toString() { return "Result[url:" + url +",title:" + title + "]"; }
}
}
###See also:
How to fire and handle HTTP requests using java.net.URLConnection
How to convert JSON to Java
Update since November 2010 (2 months after the above answer), the public search webservice has become deprecated (and the last day on which the service was offered was September 29, 2014). Your best bet is now querying http://www.google.com/search directly along with a honest user agent and then parse the result using a HTML parser. If you omit the user agent, then you get a 403 back. If you're lying in the user agent and simulate a web browser (e.g. Chrome or Firefox), then you get a way much larger HTML response back which is a waste of bandwidth and performance.
Here's a kickoff example using Jsoup as HTML parser:
String google = "http://www.google.com/search?q=";
String search = "stackoverflow";
String charset = "UTF-8";
String userAgent = "ExampleBot 1.0 (+http://example.com/bot)"; // Change this to your company's name and bot homepage!
Elements links = Jsoup.connect(google + URLEncoder.encode(search, charset)).userAgent(userAgent).get().select(".g>.r>a");
for (Element link : links) {
String title = link.text();
String url = link.absUrl("href"); // Google returns URLs in format "http://www.google.com/url?q=<url>&sa=U&ei=<someKey>".
url = URLDecoder.decode(url.substring(url.indexOf('=') + 1, url.indexOf('&')), "UTF-8");
if (!url.startsWith("http")) {
continue; // Ads/news/etc.
}
System.out.println("Title: " + title);
System.out.println("URL: " + url);
}
To search google using API you should use Google Custom Search, scraping web page is not allowed
In java you can use CustomSearch API Client Library for Java
The maven dependency is:
<dependency>
<groupId>com.google.apis</groupId>
<artifactId>google-api-services-customsearch</artifactId>
<version>v1-rev57-1.23.0</version>
</dependency>
Example code searching using Google CustomSearch API Client Library
public static void main(String[] args) throws GeneralSecurityException, IOException {
String searchQuery = "test"; //The query to search
String cx = "002845322276752338984:vxqzfa86nqc"; //Your search engine
//Instance Customsearch
Customsearch cs = new Customsearch.Builder(GoogleNetHttpTransport.newTrustedTransport(), JacksonFactory.getDefaultInstance(), null)
.setApplicationName("MyApplication")
.setGoogleClientRequestInitializer(new CustomsearchRequestInitializer("your api key"))
.build();
//Set search parameter
Customsearch.Cse.List list = cs.cse().list(searchQuery).setCx(cx);
//Execute search
Search result = list.execute();
if (result.getItems()!=null){
for (Result ri : result.getItems()) {
//Get title, link, body etc. from search
System.out.println(ri.getTitle() + ", " + ri.getLink());
}
}
}
As you can see you will need to request an api key and setup an own search engine id, cx.
Note that you can search the whole web by selecting "Search entire web" on basic tab settings during setup of cx, but results will not be exactly the same as a normal browser google search.
Currently (date of answer) you get 100 api calls per day for free, then google like to share your profit.
In the Terms of Service of google we can read:
5.3 You agree not to access (or attempt to access) any of the Services by any means other than through the interface that is provided by Google, unless you have been specifically allowed to do so in a separate agreement with Google. You specifically agree not to access (or attempt to access) any of the Services through any automated means (including use of scripts or web crawlers) and shall ensure that you comply with the instructions set out in any robots.txt file present on the Services.
So I guess the answer is No. More over the SOAP API is no longer available
Google TOS have been relaxed a bit in April 2014. Now it states:
"Don’t misuse our Services. For example, don’t interfere with our Services or try to access them using a method other than the interface and the instructions that we provide."
So the passage about "automated means" and scripts is gone now. It evidently still is not the desired (by google) way of accessing their services, but I think it is now formally open to interpretation of what exactly an "interface" is and whether it makes any difference as of how exactly returned HTML is processed (rendered or parsed). Anyhow, I have written a Java convenience library and it is up to you to decide whether to use it or not:
https://github.com/afedulov/google-web-search
Indeed there is an API to search google programmatically. The API is called google custom search. For using this API, you will need an Google Developer API key and a cx key. A simple procedure for accessing google search from java program is explained in my blog.
Now dead, here is the Wayback Machine link.
As an alternative to BalusC answer as it has been deprecated and you have to use proxies, you can use this package. Code sample:
Map<String, String> parameter = new HashMap<>();
parameter.put("q", "Coffee");
parameter.put("location", "Portland");
GoogleSearchResults serp = new GoogleSearchResults(parameter);
JsonObject data = serp.getJson();
JsonArray results = (JsonArray) data.get("organic_results");
JsonObject first_result = results.get(0).getAsJsonObject();
System.out.println("first coffee: " + first_result.get("title").getAsString());
Library on GitHub
In light of those TOS alterations last year we built an API that gives access to Google's search. It was for our own use only but after some requests we decided to open it up. We're planning to add additional search engines in the future!
Should anyone be looking for an easy way to implement / acquire search results you are free to sign up and give the REST API a try: https://searchapi.io
It returns JSON results and should be easy enough to implement with the detailed docs.
It's a shame that Bing and Yahoo are miles ahead on Google in this regard. Their APIs aren't cheap, but at least available.
I would like retrieve a list of all objects in a versioned bucket along with their versions in a list similar to how the AWS S3 Console does it:
I am able to list objects in a non-versioned bucket using the advice from this SO question. However, it is not all that clear how to list them along with version IDs as S3ObjectSummary contains no versioning information. It would seem cumbersome to make separate calls to versioning methods and then try to put them together with the information contained within an S3ObjectSummary
This looks like the closest answer I could find so far:
http://docs.aws.amazon.com/AmazonS3/latest/dev/list-obj-version-enabled-bucket.html
The relevant code snippet is here:
ListVersionsRequest request = new ListVersionsRequest()
.withBucketName(bucketName)
.withMaxResults(2);
// you can specify .withPrefix to obtain version list for a specific object or objects with
// the specified key prefix.
VersionListing versionListing;
do {
versionListing = s3client.listVersions(request);
for (S3VersionSummary objectSummary :
versionListing.getVersionSummaries()) {
System.out.println(" - " + objectSummary.getKey() + " " +
"(size = " + objectSummary.getSize() + ")" +
"(versionID= " + objectSummary.getVersionId() + ")");
}
request.setKeyMarker(versionListing.getNextKeyMarker());
request.setVersionIdMarker(versionListing.getNextVersionIdMarker());
} while (versionListing.isTruncated());
How do I get the unreadCount value using the YouTube client library?
I think this is a new tag in the YouTube Data API, but I can't find the API method to get it.
I can only get some values through SubscriptionEntry:
SubscriptionFeed feed = service.getFeed(new URL(feedUrl), SubscriptionFeed.class);
for(SubscriptionEntry entry : feed.getEntries()) {
System.out.println("Title: " + entry.getTitle().getPlainText());
System.out.println("Feed Url: " + entry.getFeedUrl());
System.out.println("CountHint: " + entry.getCountHint());
}
What you are looking is "contentDetails.newItemCount"
You can het it via getContentDetails().getNewItemCount()
I am trying to extract comments on some YouTubeVideos using the youtube-api with Java. Everything is going fine except the fact that I am not able to extract all the comments if the video has a large number of comments (it stops at somewhere in between 950 and 999). I am following a simple method of paging through the CommentFeed of the VideoEntry, getting comments on each page and then storing each comment in an ArrayList before writing them in an XML file. Here is my code for retrieving the comments
int commentCount = 0;
CommentFeed commentFeed = service.getFeed(new URL(commentUrl), CommentFeed.class);
do {
//Gets each comment in the current feed and prints to the console
for(CommentEntry comment : commentFeed.getEntries()) {
commentCount++;
System.out.println("Comment " + commentCount + " plain text content: " + comment.getPlainTextContent());
}
//Checks if there is a next page of comment feeds
if (commentFeed.getNextLink() != null) {
commentFeed = service.getFeed(new URL(commentFeed.getNextLink().getHref()), CommentFeed.class);
}
else {
commentFeed = null;
}
}
while (commentFeed != null);
My question is: Is there some limit on the number of comments that I could extract or am I doing something wrong?
use/refer this
String commentUrl = videoEntry.getComments().getFeedLink().getHref();
CommentFeed commentFeed = service.getFeed(new URL(commentUrl), CommentFeed.class);
for(CommentEntry comment : commentFeed.getEntries()) {
System.out.println(comment.getPlainTextContent());
}
source
Max number of results per iteration is 50 (it seems) as mentioned here
and you can use start-index to retrieve multiple result sets as mentioned here
Google Search API and as well as Youtube Comments search limits max. 1000 results and u cant extract more than 1000 results
I've been running through many of the text to speech examples available for Android and I have an issue that I assume is really simple, but I cannot for the life of me work it out!
I simply want to be able to view the output of EXTRA_AVAILABLE_VOICES (for example) which according to this link is returned in an ArrayList. There are many examples of how to deal with such output programmatically, but for the benefit of my learning and understanding, I want to see the actual returned data for myself.
My project is set up exactly as the android developers example from here
// We now return the list of available and unavailable voices
// as well as the return code.
Intent returnData = new Intent();
returnData.putStringArrayListExtra(
TextToSpeech.Engine.EXTRA_AVAILABLE_VOICES, available);
returnData.putStringArrayListExtra(
TextToSpeech.Engine.EXTRA_UNAVAILABLE_VOICES, unavailable);
setResult(result, returnData);
finish();
}
Ideally I'd like to have the output displayed after the 'constant value' in a simple TextView from a string, but I can't achieve that, neither can I get it in a ListView despite my many efforts... Please can someone help me solve this!
Once I know how to view the returned data, I can then go on to follow the examples of how to deal with it correctly.
I've not included any the code I've already tried, as I can't find an example anywhere and it's been pure guess work (which I be embarrassed to show!)
Thanks in advance.
For anyone who is ever stuck with the same thing, I used the code below, edited from the sample found here:
ArrayList<String> available = data
.getStringArrayListExtra("availableVoices");
Log.v("languages count", String.valueOf(available.size()));
Iterator<String> iter = available.iterator();
while (iter.hasNext()) {
String lang = iter.next();
Locale locale = new Locale(lang);
Log.v(TAG, "language: " + lang);
Log.v(TAG, "language locale: " + locale.toString());
TextView LocaleResults = (TextView) getView().findViewById(
R.id.textViewConfig);
LocaleResults.append("\nAvailable Engine Language: " + lang);
}
ArrayList<String> unavailable = data
.getStringArrayListExtra("unavailableVoices");
Log.v("languages count", String.valueOf(unavailable.size()));
Iterator<String> iteru = unavailable.iterator();
while (iteru.hasNext()) {
String ulang = iteru.next();
Locale ulocale = new Locale(ulang);
Log.v(TAG, "ulanguage: " + ulang);
Log.v(TAG, "ulanguage locale: " + ulocale.toString());
TextView LocaleResults = (TextView) getView().findViewById(
R.id.textViewConfig);
LocaleResults.append("\nUnavailable Engine Language: " + ulang);
}