Im using the new webmaster tools api to get all my site's crawling errors (+ details). Unfort. it only gives me 1000 but i have like 10000. Is there a way to get all of them ?
This is the code i use:
package main;
import com.google.api.client.googleapis.auth.oauth2.GoogleAuthorizationCodeFlow;
import com.google.api.client.googleapis.auth.oauth2.GoogleCredential;
import com.google.api.client.googleapis.auth.oauth2.GoogleTokenResponse;
import com.google.api.client.http.HttpTransport;
import com.google.api.client.http.javanet.NetHttpTransport;
import com.google.api.client.json.JsonFactory;
import com.google.api.client.json.jackson2.JacksonFactory;
import com.google.api.services.webmasters.Webmasters;
import com.google.api.services.webmasters.Webmasters.Urlcrawlerrorssamples;
import com.google.api.services.webmasters.model.SitesListResponse;
import com.google.api.services.webmasters.model.UrlCrawlErrorsSample;
import com.google.api.services.webmasters.model.UrlCrawlErrorsSamplesListResponse;
import com.google.api.services.webmasters.model.WmxSite;
import java.io.BufferedReader;
import java.io.IOException;
import java.io.InputStreamReader;
import java.net.HttpURLConnection;
import java.net.URL;
import java.util.Arrays;
import java.util.ArrayList;
import java.util.Collection;
import java.util.List;
public class WebmastersCommandLine {
private static String CLIENT_ID = "...";
private static String CLIENT_SECRET = "...";
private static String REDIRECT_URI = "urn:ietf:wg:oauth:2.0:oob";
private static String OAUTH_SCOPE = "https://www.googleapis.com/auth/webmasters.readonly";
private static String PAGE_URL = "...";
public static void main(String[] args) throws IOException {
HttpTransport httpTransport = new NetHttpTransport();
JsonFactory jsonFactory = new JacksonFactory();
GoogleAuthorizationCodeFlow flow = new GoogleAuthorizationCodeFlow.Builder(
httpTransport, jsonFactory, CLIENT_ID, CLIENT_SECRET, Arrays.asList(OAUTH_SCOPE))
.setAccessType("online")
.setApprovalPrompt("auto").build();
String url = flow.newAuthorizationUrl().setRedirectUri(REDIRECT_URI).build();
System.out.println("open URL:");
System.out.println(" " + url);
System.out.println("code:");
BufferedReader br = new BufferedReader(new InputStreamReader(System.in));
String code = br.readLine();
GoogleTokenResponse response = flow.newTokenRequest(code).setRedirectUri(REDIRECT_URI).execute();
GoogleCredential credential = new GoogleCredential().setFromTokenResponse(response);
// Create a new authorized API client
Webmasters service = new Webmasters.Builder(httpTransport, jsonFactory, credential)
.setApplicationName("WebmastersCommandLine")
.build();
Webmasters.Urlcrawlerrorssamples.List req2 = service.urlcrawlerrorssamples().list(PAGE_URL, "notFound", "web");
try
{
UrlCrawlErrorsSamplesListResponse urlList = req2.execute();
System.out.println("start");
for(UrlCrawlErrorsSample sample : urlList.getUrlCrawlErrorSample())
{
Webmasters.Urlcrawlerrorssamples.Get req3 = service.urlcrawlerrorssamples().get(PAGE_URL, sample.getPageUrl(), "notFound", "web");
UrlCrawlErrorsSample details = req3.execute();
System.out.println(sample.getPageUrl() + "," + details.getUrlDetails().getLinkedFromUrls());
}
}
catch(IOException e)
{
System.out.println("An error occurred: " + e);
}
System.out.println("done");
}
}
This however only gives me a list of 1000 errors, but i need all 10000 of them. Does anybody know a way to do that ?
The Webmaster Tools API URL Crawl Errors Sample method returns a sample of 1000 crawl errors. It's not meant to return a complete list (you could compile that from your server logs). If you want more samples through the API, one thing you can do is to mark these errors as fixed and check back in a day. It will then generate a set of samples from the remaining crawl errors.
The order of the samples is the same as in the UI, so the more important ones will be the first ones you see. This means that there are diminishing returns as you move on, with later crawl errors being either similar to the previous ones, or at least seen as being less critical. The original blog post has more on the prioritization:
We determine this based on a multitude of factors, including whether
or not you included the URL in a Sitemap, how many places it’s linked
from (and if any of those are also on your site), and whether the URL
has gotten any traffic recently from search.
Related
I am using Google Bigquery V2 Java API. I am not able to find a way to get query results in JSON format.
In Bigquery Web UI we can see this JSON and Table form of results. see scrrenshot.
Is there any way to get the GetQueryResultsResponse as JSON, using Java API.
One option is to apply the TO_JSON_STRING function to the results of your query. For example,
#standardSQL
SELECT TO_JSON_STRING(t)
FROM (
SELECT x, y
FROM YourTable
WHERE z = 10
) AS t;
If you want all of the table's columns as JSON, you can use a simpler form:
#standardSQL
SELECT TO_JSON_STRING(t)
FROM YourTable AS t
WHERE z = 10;
I'm using a service account to access the BigQuery REST API to get the response in JSON format.
In order to use a service account, you will have to go to credentials (https://console.cloud.google.com/apis/credentials) and choose a project.
You will get a drop down like this:
Create a Service account for your project and download the secret file in the JSON format. Keep the JSON file in your file system and set the path to it. Check below image to set the file path:
So, now all you have to do in is use JAVA client api to consume the Big Query REST API.
Here's is a simple solution that I've been using for my project.
package com.example.bigquery;
import java.io.IOException;
import java.io.InputStream;
import java.io.InputStreamReader;
import java.util.Arrays;
import org.apache.log4j.Logger;
import com.google.api.client.googleapis.auth.oauth2.GoogleCredential;
import com.google.api.client.http.GenericUrl;
import com.google.api.client.http.HttpContent;
import com.google.api.client.http.HttpHeaders;
import com.google.api.client.http.HttpRequest;
import com.google.api.client.http.HttpRequestFactory;
import com.google.api.client.http.HttpResponse;
import com.google.api.client.http.HttpTransport;
import com.google.api.client.http.javanet.NetHttpTransport;
import com.google.api.client.http.json.JsonHttpContent;
import com.google.api.client.json.JsonFactory;
import com.google.api.client.json.jackson2.JacksonFactory;
import com.google.common.io.CharStreams;
public class BigQueryDemo {
private static final String QUERY_URL_FORMAT = "https://www.googleapis.com/bigquery/v2/projects/%s/queries" + "?access_token=%s";
private static final String QUERY = "query";
private static final String QUERY_HACKER_NEWS_COMMENTS = "SELECT * FROM [bigquery-public-data:hacker_news.comments] LIMIT 1000";
private static final Logger logger = Logger.getLogger(BigQueryDemo.class);
static GoogleCredential credential = null;
static final HttpTransport HTTP_TRANSPORT = new NetHttpTransport();
static final JsonFactory JSON_FACTORY = new JacksonFactory();
static {
// Authenticate requests using Google Application Default credentials.
try {
credential = GoogleCredential.getApplicationDefault();
credential = credential.createScoped(Arrays.asList("https://www.googleapis.com/auth/bigquery"));
credential.refreshToken();
} catch (IOException e) {
e.printStackTrace();
}
}
public static void implicit() {
String projectId = credential.getServiceAccountProjectId();
String accessToken = generateAccessToken();
// Set the content of the request.
Dataset dataset = new Dataset().addLabel(QUERY, QUERY_HACKER_NEWS_COMMENTS);
HttpContent content = new JsonHttpContent(JSON_FACTORY, dataset.getLabels());
// Send the request to the BigQuery API.
GenericUrl url = new GenericUrl(String.format(QUERY_URL_FORMAT, projectId, accessToken));
logger.debug("URL: " + url.toString());
String responseJson = getQueryResult(content, url);
logger.debug(responseJson);
}
private static String getQueryResult(HttpContent content, GenericUrl url) {
String responseContent = null;
HttpRequestFactory requestFactory = HTTP_TRANSPORT.createRequestFactory();
HttpRequest request = null;
try {
request = requestFactory.buildPostRequest(url, content);
request.setParser(JSON_FACTORY.createJsonObjectParser());
request.setHeaders(
new HttpHeaders().set("X-HTTP-Method-Override", "POST").setContentType("application/json"));
HttpResponse response = request.execute();
InputStream is = response.getContent();
responseContent = CharStreams.toString(new InputStreamReader(is));
} catch (IOException e) {
logger.error(e);
}
return responseContent;
}
private static String generateAccessToken() {
String accessToken = null;
if ((System.currentTimeMillis() > credential.getExpirationTimeMilliseconds())) {
accessToken = credential.getRefreshToken();
} else {
accessToken = credential.getAccessToken();
}
System.out.println(accessToken);
return accessToken;
}
}
Following is the Github link to the code: https://github.com/vslala/BigQueryRestSample
It is just a demo project to fetch JSON data from the BQ REST API. Do not use it in your project directly.
Let me know if you have any questions.
i'm trying to handle my Google Agenda by using the Google Calendar APi V3( Java ).
However, i'm quite new to this and to OAUTH2 .. then i've searched for examples and i found one here :
Google Calendar API V3 Java: Unable to use 'primary' for Calendars:get
Here is the code :
import java.io.IOException;
import java.util.Collections;
import java.util.Scanner;
import java.util.Set;
import com.google.api.client.auth.oauth2.AuthorizationCodeFlow;
import com.google.api.client.auth.oauth2.AuthorizationCodeRequestUrl;
import com.google.api.client.auth.oauth2.AuthorizationCodeTokenRequest;
import com.google.api.client.auth.oauth2.TokenResponse;
import com.google.api.client.extensions.auth.helpers.Credential;
import com.google.api.client.googleapis.auth.oauth2.GoogleAuthorizationCodeFlow;
import com.google.api.client.http.HttpRequestInitializer;
import com.google.api.client.http.HttpTransport;
import com.google.api.client.http.javanet.NetHttpTransport;
import com.google.api.client.json.JsonFactory;
import com.google.api.client.json.jackson2.JacksonFactory;
import com.google.api.services.calendar.Calendar;
import com.google.api.services.calendar.Calendar.CalendarList;
import com.google.api.services.calendar.CalendarScopes;
import com.google.api.services.calendar.model.CalendarListEntry;
public class App {
public static void main(String[] args) throws IOException{
//Two globals that will be used in each step.
HttpTransport httpTransport = new NetHttpTransport();
JsonFactory jsonFactory = new JacksonFactory();
//Create the authorization code flow manager
Set<String> scope = Collections.singleton(CalendarScopes.CALENDAR);
String clientId = "xxxxxx.apps.googleusercontent.com";
String clientSecret = "xxxxxxxxxxx";
//Use a factory pattern to create the code flow
AuthorizationCodeFlow.Builder codeFlowBuilder =
new GoogleAuthorizationCodeFlow.Builder(
httpTransport,
jsonFactory,
clientId,
clientSecret,
scope
);
AuthorizationCodeFlow codeFlow = codeFlowBuilder.build();
//set the code flow to use a dummy user
//in a servlet, this could be the session id
String userId = "ipeech";
//"redirect" to the authentication url
String redirectUri = "https://www.example.com/oauth2callback";
AuthorizationCodeRequestUrl authorizationUrl = codeFlow.newAuthorizationUrl();
authorizationUrl.setRedirectUri(redirectUri);
System.out.println("Go to the following address:");
System.out.println(authorizationUrl);
//use the code that is returned as a url parameter
//to request an authorization token
System.out.println("What is the 'code' url parameter?");
String code = new Scanner(System.in).nextLine();
AuthorizationCodeTokenRequest tokenRequest = codeFlow.newTokenRequest(code);
tokenRequest.setRedirectUri(redirectUri);
TokenResponse tokenResponse = tokenRequest.execute();
//Now, with the token and user id, we have credentials
com.google.api.client.auth.oauth2.Credential credential = codeFlow.createAndStoreCredential(tokenResponse, userId);
//Credentials may be used to initialize http requests
HttpRequestInitializer initializer = credential;
//and thus are used to initialize the calendar service
Calendar.Builder serviceBuilder = new Calendar.Builder(
httpTransport, jsonFactory, initializer);
serviceBuilder.setApplicationName("Example");
Calendar calendar = serviceBuilder.build();
//get some data
String calendarID = "xxxxxxxxxxx";
getCalendarListSummary(calendarID,calendar);
getAllCalendarListSummary(calendar);
//getCalendarSummary(calendarID,calendar);
}
public static void getCalendarListSummary(String calendarID, Calendar calendar) throws IOException{
CalendarListEntry calendarListEntry = calendar.calendarList().get(calendarID).execute();
System.out.println(calendarListEntry.getSummary());
}
public static void getAllCalendarListSummary (Calendar calendar) throws IOException{
Calendar.CalendarList.List listRequest = calendar.calendarList().list();
com.google.api.services.calendar.model.CalendarList feed = listRequest.execute();
for(CalendarListEntry entry:feed.getItems()){
System.out.println("ID: " + entry.getId());
System.out.println("Summary: " + entry.getSummary());
}
}
When i launch the programm, it asks me to give the authorization code ("What is the 'code' url parameter?") but i don't know where to find it .. Any ideas ?
In this example, there is a part that says "Go to the following address:" you have to copy that url, paste it in the browser and then you will receive the authorization code. Copy that code and paste it after "What is the 'code' url parameter?" and press "Enter" so the program can continue.
This is a basic example and that why the OAuth 2 flow is done that way.
Here is a complete example of a Google calendar java program. I would suggest to first understand how OAuth 2 works, how to create projects in the Developer console and how to create credentials for those projects. Then it would be easier to understand and use the complete example.
I am embarrassed that I'm simply failing with an example piece of code, but I'll blame it on the fact that it is late...
I have taken a copy and paste of: https://developers.google.com/gmail/api/quickstart/quickstart-java
and downloaded the client libraries: https://code.google.com/p/google-api-java-client/
and https://developers.google.com/api-client-library/java/apis/gmail/v1
When I run the sample, I get the following exception:
Exception in thread "main" java.lang.IllegalArgumentException
at com.google.api.client.repackaged.com.google.common.base.Preconditions.checkArgument(Preconditions.java:76)
at com.google.api.client.util.Preconditions.checkArgument(Preconditions.java:37)
at com.google.api.client.googleapis.auth.oauth2.GoogleClientSecrets.getDetails(GoogleClientSecrets.java:82)
at com.google.api.client.googleapis.auth.oauth2.GoogleAuthorizationCodeFlow$Builder.<init>(GoogleAuthorizationCodeFlow.java:195)
at com.emailreply.musterion.GmailApiQuickstart.main(GmailApiQuickstart.java:40)
Googling, I can't find a simple answer, so am assuming stupidity or a library missing/incorrect.
The libraries as I have them are:
/libs/commons-logging-1.1.1.jar
/libs/google-api-client-1.19.0.jar
/libs/google-api-client-android-1.19.0.jar
/libs/google-api-client-appengine-1.19.0.jar
/libs/google-api-client-gson-1.19.0.jar
/libs/google-api-client-jackson2-1.19.0.jar
/libs/google-api-client-java6-1.19.0.jar
/libs/google-api-client-servlet-1.19.0.jar
/libs/google-http-client-1.19.0.jar
/libs/google-http-client-android-1.19.0.jar
/libs/google-http-client-appengine-1.19.0.jar
/libs/google-http-client-gson-1.19.0.jar
/libs/google-http-client-jackson2-1.19.0.jar
/libs/google-http-client-jdo-1.19.0.jar
/libs/google-oauth-client-1.19.0.jar
/libs/google-oauth-client-appengine-1.19.0.jar
/libs/google-oauth-client-java6-1.19.0.jar
/libs/google-oauth-client-jetty-1.19.0.jar
/libs/google-oauth-client-servlet-1.19.0.jar
/libs/gson-2.1.jar
/libs/httpclient-4.0.1.jar
/libs/httpcore-4.0.1.jar
/libs/jackson-core-2.1.3.jar
/libs/jdo2-api-2.3-eb.jar
/libs/jetty-6.1.26.jar
/libs/jetty-util-6.1.26.jar
/libs/jsr305-1.3.9.jar
/libs/transaction-api-1.1.jar
google-api-services-gmail-v1-rev10-1.19.0.jar
The example mentioned above:
import java.io.BufferedReader;
import java.io.IOException;
import java.io.InputStreamReader;
import java.util.Arrays;
import java.util.List;
import com.google.api.client.googleapis.auth.oauth2.GoogleAuthorizationCodeFlow;
import com.google.api.client.googleapis.auth.oauth2.GoogleClientSecrets;
import com.google.api.client.googleapis.auth.oauth2.GoogleCredential;
import com.google.api.client.googleapis.auth.oauth2.GoogleOAuthConstants;
import com.google.api.client.googleapis.auth.oauth2.GoogleTokenResponse;
import com.google.api.client.http.HttpTransport;
import com.google.api.client.http.javanet.NetHttpTransport;
import com.google.api.client.json.JsonFactory;
import com.google.api.client.json.jackson2.JacksonFactory;
import com.google.api.services.gmail.Gmail;
import com.google.api.services.gmail.model.ListThreadsResponse;
import com.google.api.services.gmail.model.Thread;
public class GmailApiQuickstart {
// Check https://developers.google.ciom/gmail/api/auth/scopes for all available scopes
private static final String SCOPE = "https://www.googleapis.com/auth/gmail.readonly";
private static final String APP_NAME = "Gmail API Quickstart";
// Email address of the user, or "me" can be used to represent the currently authorized user.
private static final String USER = "me";
// Path to the client_secret.json file downloaded from the Developer Console
private static final String CLIENT_SECRET_PATH = "./client_secret.json";
public static void main (String [] args) throws IOException {
HttpTransport httpTransport = new NetHttpTransport();
JsonFactory jsonFactory = new JacksonFactory();
GoogleClientSecrets clientSecrets = GoogleClientSecrets.load(jsonFactory, new BufferedReader(new InputStreamReader(GmailApiQuickstart.class.getResourceAsStream(CLIENT_SECRET_PATH))));
// Allow user to authorize via url.
GoogleAuthorizationCodeFlow flow = new GoogleAuthorizationCodeFlow.Builder(
httpTransport, jsonFactory, clientSecrets, Arrays.asList(SCOPE))
.setAccessType("online")
.setApprovalPrompt("auto").build();
String url = flow.newAuthorizationUrl().setRedirectUri(GoogleOAuthConstants.OOB_REDIRECT_URI).build();
System.out.println("Please open the following URL in your browser then type the authorization code:\n" + url);
// Read code entered by user.
BufferedReader br = new BufferedReader(new InputStreamReader(System.in));
String code = br.readLine();
// Generate Credential using retrieved code.
GoogleTokenResponse response = flow.newTokenRequest(code)
.setRedirectUri(GoogleOAuthConstants.OOB_REDIRECT_URI).execute();
GoogleCredential credential = new GoogleCredential()
.setFromTokenResponse(response);
// Create a new authorized Gmail API client
Gmail service = new Gmail.Builder(httpTransport, jsonFactory, credential)
.setApplicationName(APP_NAME).build();
// Retrieve a page of Threads; max of 100 by default.
ListThreadsResponse threadsResponse = service.users().threads().list(USER).execute();
List<Thread> threads = threadsResponse.getThreads();
// Print ID of each Thread.
for (Thread thread : threads) {
System.out.println("Thread ID: " + thread.getId());
}
}
}
I replaced the reference to CLIENT_SECRET_PATH with:
new BufferedReader(new InputStreamReader(GmailApiQuickstart.class.getResourceAsStream(CLIENT_SECRET_PATH)))
for no other reason than to try something different. It does work and reads the file correctly.
Any ideas?
Right, after some more research (asking a colleague/genius), I found the problem. Basically the GoogleClientSecrets object was not being properly bound with the information from my client_secrets.json file. This meant that during authentication, objects were null resulting in the IllegalArgumentException.
So the original file which looked like this:
{
"private_key_id": "zzz",
"private_key": "-----BEGIN PRIVATE KEY-----\nxyz\n-----END PRIVATE KEY-----\n",
"client_email": "1234#developer.gserviceaccount.com",
"client_id": "1wdfghyjmp.apps.googleusercontent.com",
"type": "service_account"
}
was edited to look like this:
{
"web" : {
"private_key_id": "zzz",
"private_key": "-----BEGIN PRIVATE KEY-----\nxyz\n-----END PRIVATE KEY-----\n",
"client_email": "1234#developer.gserviceaccount.com",
"client_id": "1wdfghyjmp.apps.googleusercontent.com",
"type": "service_account"
}
}
This allowed me to progress through the code with authentication.
Hope this helps.
here is a working example of non-interactive auth for Google Non-interactive authorization with Google OAuth2
the problem is not in "web" tag in the client json, but rather in the fact that their example is for web authentication while their suggested way of generating credentials is for non-interactive "service account". they have multiple problems with their documentation.
I'm trying to upload file to my Google drive Account using Java. The file is uploaded but i'm getting an warning: WARNING: Application name is not set. Call Builder#setApplicationName.
Code is
import java.io.BufferedReader;
import java.io.IOException;
import java.io.InputStreamReader;
import java.util.Arrays;
import com.google.api.client.googleapis.auth.oauth2.GoogleAuthorizationCodeFlow;
import com.google.api.client.googleapis.auth.oauth2.GoogleCredential;
import com.google.api.client.googleapis.auth.oauth2.GoogleTokenResponse;
import com.google.api.client.http.FileContent;
import com.google.api.client.http.HttpTransport;
import com.google.api.client.http.javanet.NetHttpTransport;
import com.google.api.client.json.JsonFactory;
import com.google.api.client.json.jackson2.JacksonFactory;
import com.google.api.services.drive.Drive;
import com.google.api.services.drive.DriveScopes;
import com.google.api.services.drive.model.File;
public class UploadFileGoogleDrive {
private static String CLIENT_ID = "*****";
private static String CLIENT_SECRET = "******";
private static String REDIRECT_URI = "****";
public static void main(String[] args) throws IOException{
HttpTransport httpTransport = new NetHttpTransport();
JsonFactory jsonFactory = new JacksonFactory();
GoogleAuthorizationCodeFlow flow = new GoogleAuthorizationCodeFlow.Builder(
httpTransport, jsonFactory, CLIENT_ID, CLIENT_SECRET, Arrays.asList(DriveScopes.DRIVE))
.setAccessType("online")
.setApprovalPrompt("auto").build();
String url = flow.newAuthorizationUrl().setRedirectUri(REDIRECT_URI).build();
System.out.println("Please open the following URL in your browser then type the authorization code:");
System.out.println(" " + url);
BufferedReader br = new BufferedReader(new InputStreamReader(System.in));
String code = br.readLine();
GoogleTokenResponse response = flow.newTokenRequest(code).setRedirectUri(REDIRECT_URI).execute();
GoogleCredential credential = new GoogleCredential().setFromTokenResponse(response);
//Create a new authorized API client
Drive service = new Drive.Builder(httpTransport, jsonFactory, credential).build();
//Insert a file
File body = new File();
body.setTitle("My document");
body.setDescription("A test document");
body.setMimeType("text/plain");
java.io.File fileContent = new java.io.File("document.txt");
FileContent mediaContent = new FileContent("text/plain", fileContent);
File file = service.files().insert(body, mediaContent).execute();
System.out.println("File ID: " + file.getId());
}
}
The Output is
May 24, 2014 11:59:14 AM com.google.api.client.googleapis.services.AbstractGoogleClient
WARNING: Application name is not set. Call Builder#setApplicationName.
File ID: 1Ze77mqHtKWDU3eVljATlHQ0U
The Google APIs Client Library for Java strongly encourages you to set an application name when constructing your clients, for use when troubleshooting requests against our server logs. You can see an example on how to set the application name in the Google Drive Java command line sample.
I'm trying to access Google BigQuery using Service Account approach. My code is as follows:
private static final HttpTransport HTTP_TRANSPORT = new NetHttpTransport();
private static final JsonFactory JSON_FACTORY = new JacksonFactory();
GoogleCredential credentials = new GoogleCredential.Builder()
.setTransport(HTTP_TRANSPORT)
.setJsonFactory(JSON_FACTORY)
.setServiceAccountId("XXXXX#developer.gserviceaccount.com")
.setServiceAccountScopes(BigqueryScopes.BIGQUERY)
.setServiceAccountPrivateKeyFromP12File(
new File("PATH-TO-privatekey.p12"))
.build();
Bigquery bigquery = Bigquery.builder(HTTP_TRANSPORT, JSON_FACTORY).setHttpRequestInitializer(credentials)
.build();
com.google.api.services.bigquery.Bigquery.Datasets.List datasetRequest = bigquery.datasets().list(
"PROJECT_ID");
DatasetList datasetList = datasetRequest.execute();
if (datasetList.getDatasets() != null) {
java.util.List<Datasets> datasets = datasetList.getDatasets();
System.out.println("Available datasets\n----------------");
for (Datasets dataset : datasets) {
System.out.format("%s\n", dataset.getDatasetReference().getDatasetId());
}
}
But it throws the following exception:
Exception in thread "main" com.google.api.client.googleapis.json.GoogleJsonResponseException: 401 Unauthorized
{
"code" : 401,
"errors" : [ {
"domain" : "global",
"location" : "Authorization",
"locationType" : "header",
"message" : "Authorization required",
"reason" : "required"
} ],
"message" : "Authorization required"
}
at com.google.api.client.googleapis.json.GoogleJsonResponseException.from(GoogleJsonResponseException.java:159)
at com.google.api.client.googleapis.json.GoogleJsonResponseException.execute(GoogleJsonResponseException.java:187)
at com.google.api.client.googleapis.services.GoogleClient.executeUnparsed(GoogleClient.java:115)
at com.google.api.client.http.json.JsonHttpRequest.executeUnparsed(JsonHttpRequest.java:112)
at com.google.api.services.bigquery.Bigquery$Datasets$List.execute(Bigquery.java:979)
The exception is fired on this line:
DatasetList datasetList = datasetRequest.execute();
I'm getting the account ID from Google's API console from the second line on the section that looks like this:
Client ID: XXXXX.apps.googleusercontent.com
Email address: XXXXX#developer.gserviceaccount.com
What am I missing?
Eureka! Both Eric's and Michael's code works well.
The error posted in the question can be reproduced by setting the time on the client machine incorrectly. Fortunately, it can be solved by setting the time on the client machine correctly.
Note: For what it's worth, I synchronized the time on a Windows 7 box using the "Update now" button in the "Internet Time Settings" dialog. I figured that should be pretty idiot-proof... but I guess I beat the system. It corrected the seconds but left the machine off by exactly one minute. The BigQuery call failed after that. It succeeded after I manually changed the time.
Our error handling code in the Java library needs to be improved a bit!
It looks like the signed JWT for requesting an OAuth access token is failing. You can see this by enabling the logs that #MichaelManoochehri mentioned above.
There's only a few things that I think could be causing this failure:
Invalid signature (using the wrong key)
Invalid e-mail address for the service account (I think that's been ruled out)
Invalid date/time stamp used for generating the signed blob (an issue date, and an expiration date)
Invalid scope (I think that's been ruled out)
You should check that your date/time is properly set on your server with the proper timezone -- sync'd to NTP. You can use time.gov to see the official US atomic clock time.
EDIT: The answer I gave below is relevant to using Google App Engine Service Accounts - leaving here for reference.
Double check that you have added your service account address to your project's team page as an owner.
I'd recommend using the AppIdentityCredential class to handle service account auth. Here's a small snippet that demonstrates this, and I'll add additional documentation about this on the BigQuery API developer page.
Also, make sure that you are using the latest version of the Google Java API client (as of today, it's version "v2-rev5-1.5.0-beta" here).
import java.io.IOException;
import javax.servlet.http.HttpServlet;
import javax.servlet.http.HttpServletRequest;
import javax.servlet.http.HttpServletResponse;
import com.google.api.client.googleapis.extensions.appengine.auth.oauth2.AppIdentityCredential;
import com.google.api.client.http.HttpTransport;
import com.google.api.client.http.javanet.NetHttpTransport;
import com.google.api.client.http.json.JsonHttpRequest;
import com.google.api.client.http.json.JsonHttpRequestInitializer;
import com.google.api.client.json.JsonFactory;
import com.google.api.client.json.jackson.JacksonFactory;
import com.google.api.services.bigquery.Bigquery;
import com.google.api.services.bigquery.BigqueryRequest;
#SuppressWarnings("serial")
public class Bigquery_service_accounts_demoServlet<TRANSPORT> extends HttpServlet {
// ENTER YOUR PROJECT ID HERE
private static final String PROJECT_ID = "";
private static final HttpTransport TRANSPORT = new NetHttpTransport();
private static final JsonFactory JSON_FACTORY = new JacksonFactory();
private static final String BIGQUERY_SCOPE = "https://www.googleapis.com/auth/bigquery";
AppIdentityCredential credential = new AppIdentityCredential(BIGQUERY_SCOPE);
Bigquery bigquery = Bigquery.builder(TRANSPORT,JSON_FACTORY)
.setHttpRequestInitializer(credential)
.setJsonHttpRequestInitializer(new JsonHttpRequestInitializer() {
public void initialize(JsonHttpRequest request) {
BigqueryRequest bigqueryRequest = (BigqueryRequest) request;
bigqueryRequest.setPrettyPrint(true);
}
}).build();
public void doGet(HttpServletRequest req, HttpServletResponse resp)
throws IOException {
resp.setContentType("text/plain");
resp.getWriter().println(bigquery.datasets()
.list(PROJECT_ID)
.execute().toString());
}
}
Here is a complete snippet for reference:
import com.google.api.client.http.HttpTransport;
import com.google.api.client.http.javanet.NetHttpTransport;
import com.google.api.client.json.JsonFactory;
import com.google.api.client.json.jackson.JacksonFactory;
import com.google.api.client.googleapis.auth.oauth2.GoogleCredential;
import com.google.api.services.bigquery.Bigquery;
import com.google.api.services.bigquery.Bigquery.Datasets;
import com.google.api.services.bigquery.model.DatasetList;
import java.io.File;
import java.io.IOException;
import java.security.GeneralSecurityException;
public class BigQueryJavaServiceAccount {
private static final String SCOPE = "https://www.googleapis.com/auth/bigquery";
private static final HttpTransport TRANSPORT = new NetHttpTransport();
private static final JsonFactory JSON_FACTORY = new JacksonFactory();
public static void main(String[] args) throws IOException, GeneralSecurityException {
GoogleCredential credential = new GoogleCredential.Builder().setTransport(TRANSPORT)
.setJsonFactory(JSON_FACTORY)
.setServiceAccountId("XXXXXXX#developer.gserviceaccount.com")
.setServiceAccountScopes(SCOPE)
.setServiceAccountPrivateKeyFromP12File(new File("my_file.p12"))
.build();
Bigquery bigquery = Bigquery.builder(TRANSPORT, JSON_FACTORY)
.setApplicationName("Google-BigQuery-App/1.0")
.setHttpRequestInitializer(credential).build();
Datasets.List datasetRequest = bigquery.datasets().list("publicdata");
DatasetList datasetList = datasetRequest.execute();
System.out.format("%s\n", datasetList.toPrettyString());
}