We Keep Coding

Read/Write Google sheet via Server without using OAuth

Tried going through the internet and google docs they provide OAuth way only. Is there a way to read/write to google sheets with API Key and not OAuth.

After some research, Credential object from google-oath-client module can help. Download the .p12 file from the google account. Code for reading a google sheet without OAUth prompt below. This can also be used to write or append sheets with some modification :
package com.mycomp;
import com.google.api.client.auth.oauth2.Credential;
import com.google.api.client.googleapis.auth.oauth2.GoogleCredential;
import com.google.api.client.googleapis.javanet.GoogleNetHttpTransport;
import com.google.api.client.http.HttpTransport;
import com.google.api.client.http.javanet.NetHttpTransport;
import com.google.api.client.json.JsonFactory;
import com.google.api.client.json.gson.GsonFactory;
import com.google.api.services.sheets.v4.Sheets;
import com.google.api.services.sheets.v4.SheetsScopes;
import com.google.api.services.sheets.v4.model.ValueRange;
import com.nm.vernacular.services.SpreadSheetsService;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.beans.factory.annotation.Qualifier;
import org.springframework.stereotype.Service;
import java.io.File;
import java.io.IOException;
import java.net.URISyntaxException;
import java.net.URL;
import java.security.GeneralSecurityException;
import java.util.Collections;
import java.util.LinkedList;
import java.util.List;
/**
* Created by ankushgupta & modified for SO.
*/
public class GoogleSheetsReader {
private static final JsonFactory JSON_FACTORY = GsonFactory.getDefaultInstance();
private static final String KEY_FILE_LOCATION = "<Name of p12 file>.p12";
private static final String SERVICE_ACCOUNT_EMAIL = "<email of google service account>";
private static final String APPLICATION_NAME = "Google Sheets API";
private static final Logger LOGGER = LoggerFactory.getLogger(GoogleSheetsReader.class);
/**
* Global instance of the scopes required by this quickstart.
* If modifying these scopes, delete your previously saved credentials/ folder.
*/
private static final List<String> SCOPES = Collections.singletonList(SheetsScopes.SPREADSHEETS);
/**
* Creates an authorized Credential object.
* #return An authorized Credential object.
* #throws IOException If there is no client_secret.
*/
private Credential getCredentials() throws URISyntaxException, IOException, GeneralSecurityException {
//Reading Key File
URL fileURL = GoogleSheetsReader.class.getClassLoader().getResource(KEY_FILE_LOCATION);
// Initializes an authorized analytics service object.
if(fileURL==null) {
fileURL = (new File("/resources/"+ KEY_FILE_LOCATION)).toURI().toURL();
}
// Construct a GoogleCredential object with the service account email
// and p12 file downloaded from the developer console.
HttpTransport httpTransport = GoogleNetHttpTransport.newTrustedTransport();
return new GoogleCredential.Builder()
.setTransport(httpTransport)
.setJsonFactory(JSON_FACTORY)
.setServiceAccountId(SERVICE_ACCOUNT_EMAIL)
.setServiceAccountPrivateKeyFromP12File(new File(fileURL.toURI()))
.setServiceAccountScopes(SCOPES)
.build();
}
#Override
public List<Object[]> readSheet(String nameAndRange, String key, int[] returnRange) throws GeneralSecurityException, IOException {
final NetHttpTransport HTTP_TRANSPORT = GoogleNetHttpTransport.newTrustedTransport();
final String spreadsheetId = key;
final String range = nameAndRange;
try {
Sheets service = new Sheets.Builder(HTTP_TRANSPORT, JSON_FACTORY, getCredentials())
.setApplicationName(APPLICATION_NAME)
.build();
ValueRange response = service.spreadsheets().values()
.get(spreadsheetId, range)
.execute();
List<List<Object>> values = response.getValues();
int a = returnRange.length;
List<Object[]> result = new LinkedList<>();
if (values == null || values.isEmpty()) {
return Collections.emptyList();
} else {
for (List row : values) {
if(row.size() >= a) {
Object[] objArr = new Object[a];
for(int i=0;i<a;i++) {
objArr[i] = row.get(returnRange[i]);
}
result.add(objArr);
}
}
}
return result;
} catch(Exception ex) {
LOGGER.error("Exception while reading google sheet", ex);
} finally {
}
return null;
}
public static void main(String[] args) {
GoogleSheetsReader reader = new GoogleSheetsReader();
reader.readSheet("<Sheet Name>!A2:B", "<sheets key from URL>", new int[]{0, 1});
}
}

Based from this documentation, when your application requests public data, the request doesn't need to be authorized, but does need to be accompanied by an identifier, such as an API key.
Every request your application sends to the Google Sheets API needs to identify your application to Google. There are two ways to identify your application: using an OAuth 2.0 token (which also authorizes the request) and/or using the application's API key. Here's how to determine which of those options to use:
If the request requires authorization (such as a request for an individual's private data), then the application must provide an OAuth 2.0 token with the request. The application may also provide the API key, but it doesn't have to.
If the request doesn't require authorization (such as a request for public data), then the application must provide either the API key or an OAuth 2.0 token, or both—whatever option is most convenient for you.
However, there are some scopes which require OAuth authorization. Check this link: Access Google spreadsheet API without Oauth token.

Using API key, you can read from google sheets, but only if the sheet is shared with public.
However to write to google sheets, you must you OAuth. See this link.

Webmaster Tools Api, get more then 1000 crawling errors

Im using the new webmaster tools api to get all my site's crawling errors (+ details). Unfort. it only gives me 1000 but i have like 10000. Is there a way to get all of them ?
This is the code i use:
package main;
import com.google.api.client.googleapis.auth.oauth2.GoogleAuthorizationCodeFlow;
import com.google.api.client.googleapis.auth.oauth2.GoogleCredential;
import com.google.api.client.googleapis.auth.oauth2.GoogleTokenResponse;
import com.google.api.client.http.HttpTransport;
import com.google.api.client.http.javanet.NetHttpTransport;
import com.google.api.client.json.JsonFactory;
import com.google.api.client.json.jackson2.JacksonFactory;
import com.google.api.services.webmasters.Webmasters;
import com.google.api.services.webmasters.Webmasters.Urlcrawlerrorssamples;
import com.google.api.services.webmasters.model.SitesListResponse;
import com.google.api.services.webmasters.model.UrlCrawlErrorsSample;
import com.google.api.services.webmasters.model.UrlCrawlErrorsSamplesListResponse;
import com.google.api.services.webmasters.model.WmxSite;
import java.io.BufferedReader;
import java.io.IOException;
import java.io.InputStreamReader;
import java.net.HttpURLConnection;
import java.net.URL;
import java.util.Arrays;
import java.util.ArrayList;
import java.util.Collection;
import java.util.List;
public class WebmastersCommandLine {
private static String CLIENT_ID = "...";
private static String CLIENT_SECRET = "...";
private static String REDIRECT_URI = "urn:ietf:wg:oauth:2.0:oob";
private static String OAUTH_SCOPE = "https://www.googleapis.com/auth/webmasters.readonly";
private static String PAGE_URL = "...";
public static void main(String[] args) throws IOException {
HttpTransport httpTransport = new NetHttpTransport();
JsonFactory jsonFactory = new JacksonFactory();
GoogleAuthorizationCodeFlow flow = new GoogleAuthorizationCodeFlow.Builder(
httpTransport, jsonFactory, CLIENT_ID, CLIENT_SECRET, Arrays.asList(OAUTH_SCOPE))
.setAccessType("online")
.setApprovalPrompt("auto").build();
String url = flow.newAuthorizationUrl().setRedirectUri(REDIRECT_URI).build();
System.out.println("open URL:");
System.out.println(" " + url);
System.out.println("code:");
BufferedReader br = new BufferedReader(new InputStreamReader(System.in));
String code = br.readLine();
GoogleTokenResponse response = flow.newTokenRequest(code).setRedirectUri(REDIRECT_URI).execute();
GoogleCredential credential = new GoogleCredential().setFromTokenResponse(response);
// Create a new authorized API client
Webmasters service = new Webmasters.Builder(httpTransport, jsonFactory, credential)
.setApplicationName("WebmastersCommandLine")
.build();
Webmasters.Urlcrawlerrorssamples.List req2 = service.urlcrawlerrorssamples().list(PAGE_URL, "notFound", "web");
try
{
UrlCrawlErrorsSamplesListResponse urlList = req2.execute();
System.out.println("start");
for(UrlCrawlErrorsSample sample : urlList.getUrlCrawlErrorSample())
{
Webmasters.Urlcrawlerrorssamples.Get req3 = service.urlcrawlerrorssamples().get(PAGE_URL, sample.getPageUrl(), "notFound", "web");
UrlCrawlErrorsSample details = req3.execute();
System.out.println(sample.getPageUrl() + "," + details.getUrlDetails().getLinkedFromUrls());
}
}
catch(IOException e)
{
System.out.println("An error occurred: " + e);
}
System.out.println("done");
}
}
This however only gives me a list of 1000 errors, but i need all 10000 of them. Does anybody know a way to do that ?

The Webmaster Tools API URL Crawl Errors Sample method returns a sample of 1000 crawl errors. It's not meant to return a complete list (you could compile that from your server logs). If you want more samples through the API, one thing you can do is to mark these errors as fixed and check back in a day. It will then generate a set of samples from the remaining crawl errors.
The order of the samples is the same as in the UI, so the more important ones will be the first ones you see. This means that there are diminishing returns as you move on, with later crawl errors being either similar to the previous ones, or at least seen as being less critical. The original blog post has more on the prioritization:
We determine this based on a multitude of factors, including whether
or not you included the URL in a Sitemap, how many places it’s linked
from (and if any of those are also on your site), and whether the URL
has gotten any traffic recently from search.

Error when getting groups for a particular email ID with Java google apps directory API

I want to get groups for a particular email ID in Google Apps with the help of the Java Google Apps Directory API but I am getting an error.
Can someone help explain what the error is and what in my code might be causing it? The error and related code follows.
Error:
WARNING: Application name is not set. Call Builder#setApplicationName.
Exception in thread "main" com.google.api.client.auth.oauth2.TokenResponseException: 400 Bad Request
{
"error" : "invalid_grant"
}
import java.util.ArrayList;
import java.util.Collection;
import com.google.api.client.googleapis.auth.oauth2.GoogleCredential;
import com.google.api.client.http.HttpTransport;
import com.google.api.client.http.javanet.NetHttpTransport;
import com.google.api.client.json.jackson2.JacksonFactory;
import com.google.api.services.admin.directory.Directory;
import com.google.api.services.admin.directory.model.Group;
import com.google.api.services.bigquery.Bigquery;
public class Test {
private static Bigquery bigquery;
/**
* #param args
*/
public static void main(String[] args) throws Exception{
// TODO Auto-generated method stub
HttpTransport httpTransport = new NetHttpTransport();
JacksonFactory jsonFactory = new JacksonFactory();
Collection<String> SCOPES = new ArrayList<String>();
SCOPES.add("https://www.googleapis.com/auth/admin.directory.group");
SCOPES.add("https://www.googleapis.com/auth/admin.directory.user");
GoogleCredential credential = new GoogleCredential.Builder()
.setTransport(httpTransport)
.setJsonFactory(jsonFactory)
.setServiceAccountId("244204474315-l08ah6g350oofeosi7p8pqmotlrmgion.apps.googleusercontent.com")
.setServiceAccountUser("244204474315-l08ah6g350oofeosi7p8pqmotlrmgion#developer.gserviceaccount.com")
.setServiceAccountScopes(SCOPES)
.setServiceAccountPrivateKeyFromP12File(
new java.io.File("C:\\Users\\nbaser\\Desktop\\76ba0ac39b06e8419bbb670734f3b2affeec43b2-privatekey.p12"))
.build();
Directory directory = new Directory.Builder(httpTransport, jsonFactory, credential).build();
// Directory.Users.List list = directory.users().list();
// list.setDomain("yourDomain.com");
// Users users = list.execute();
//Directory.Users.Get user = directory.users().get("nishant.baser#ahold.com");
//user.get
Directory.Groups.Get group = directory.groups().get("ausa.googleams.group#ahold.com");
Group groups = group.execute();
// Directory.Members.List list = directory.members().list("ausa.googleams.group#ahold.com");
// Members members = list.execute();
// System.out.println(members);
}
}

Call Builder#setApplicationName in your new GoogleCredential.Builder().

Java sample OAuth service flow apps for accessing Big Query

Is there any sample Java apps that show how to access Big Query using an OAuth service flow. Most of the apps seem to show web and desktop flows.
Thanks

Sure - try something like this (below). More examples of this type of flow here: https://developers.google.com/bigquery/authorization#service-accounts-server
import com.google.api.client.http.HttpTransport;
import com.google.api.client.http.javanet.NetHttpTransport;
import com.google.api.client.json.JsonFactory;
import com.google.api.client.json.jackson.JacksonFactory;
import com.google.api.client.googleapis.auth.oauth2.GoogleCredential;
import com.google.api.services.bigquery.Bigquery;
import com.google.api.services.bigquery.Bigquery.Datasets;
import com.google.api.services.bigquery.model.DatasetList;
import java.io.File;
import java.io.IOException;
import java.security.GeneralSecurityException;
public class JavaCommandLineServiceAccounts {
private static final String SCOPE = "https://www.googleapis.com/auth/bigquery";
private static final HttpTransport TRANSPORT = new NetHttpTransport();
private static final JsonFactory JSON_FACTORY = new JacksonFactory();
private static Bigquery bigquery;
public static void main(String[] args) throws IOException, GeneralSecurityException {
GoogleCredential credential = new GoogleCredential.Builder().setTransport(TRANSPORT)
.setJsonFactory(JSON_FACTORY)
.setServiceAccountId("XXXXXXX#developer.gserviceaccount.com")
.setServiceAccountScopes(SCOPE)
.setServiceAccountPrivateKeyFromP12File(new File("my_file.p12"))
.build();
bigquery = new Bigquery.Builder(TRANSPORT, JSON_FACTORY, credential)
.setApplicationName("BigQuery-Service-Accounts/0.1")
.setHttpRequestInitializer(credential).build();
Datasets.List datasetRequest = bigquery.datasets().list("publicdata");
DatasetList datasetList = datasetRequest.execute();
System.out.format("%s\n", datasetList.toPrettyString());
}
}

BigQuery and OAuth2

I'm trying to access Google BigQuery using Service Account approach. My code is as follows:
private static final HttpTransport HTTP_TRANSPORT = new NetHttpTransport();
private static final JsonFactory JSON_FACTORY = new JacksonFactory();
GoogleCredential credentials = new GoogleCredential.Builder()
.setTransport(HTTP_TRANSPORT)
.setJsonFactory(JSON_FACTORY)
.setServiceAccountId("XXXXX#developer.gserviceaccount.com")
.setServiceAccountScopes(BigqueryScopes.BIGQUERY)
.setServiceAccountPrivateKeyFromP12File(
new File("PATH-TO-privatekey.p12"))
.build();
Bigquery bigquery = Bigquery.builder(HTTP_TRANSPORT, JSON_FACTORY).setHttpRequestInitializer(credentials)
.build();
com.google.api.services.bigquery.Bigquery.Datasets.List datasetRequest = bigquery.datasets().list(
"PROJECT_ID");
DatasetList datasetList = datasetRequest.execute();
if (datasetList.getDatasets() != null) {
java.util.List<Datasets> datasets = datasetList.getDatasets();
System.out.println("Available datasets\n----------------");
for (Datasets dataset : datasets) {
System.out.format("%s\n", dataset.getDatasetReference().getDatasetId());
}
}
But it throws the following exception:
Exception in thread "main" com.google.api.client.googleapis.json.GoogleJsonResponseException: 401 Unauthorized
{
"code" : 401,
"errors" : [ {
"domain" : "global",
"location" : "Authorization",
"locationType" : "header",
"message" : "Authorization required",
"reason" : "required"
} ],
"message" : "Authorization required"
}
at com.google.api.client.googleapis.json.GoogleJsonResponseException.from(GoogleJsonResponseException.java:159)
at com.google.api.client.googleapis.json.GoogleJsonResponseException.execute(GoogleJsonResponseException.java:187)
at com.google.api.client.googleapis.services.GoogleClient.executeUnparsed(GoogleClient.java:115)
at com.google.api.client.http.json.JsonHttpRequest.executeUnparsed(JsonHttpRequest.java:112)
at com.google.api.services.bigquery.Bigquery$Datasets$List.execute(Bigquery.java:979)
The exception is fired on this line:
DatasetList datasetList = datasetRequest.execute();
I'm getting the account ID from Google's API console from the second line on the section that looks like this:
Client ID: XXXXX.apps.googleusercontent.com
Email address: XXXXX#developer.gserviceaccount.com
What am I missing?

Eureka! Both Eric's and Michael's code works well.
The error posted in the question can be reproduced by setting the time on the client machine incorrectly. Fortunately, it can be solved by setting the time on the client machine correctly.
Note: For what it's worth, I synchronized the time on a Windows 7 box using the "Update now" button in the "Internet Time Settings" dialog. I figured that should be pretty idiot-proof... but I guess I beat the system. It corrected the seconds but left the machine off by exactly one minute. The BigQuery call failed after that. It succeeded after I manually changed the time.

Our error handling code in the Java library needs to be improved a bit!
It looks like the signed JWT for requesting an OAuth access token is failing. You can see this by enabling the logs that #MichaelManoochehri mentioned above.
There's only a few things that I think could be causing this failure:
Invalid signature (using the wrong key)
Invalid e-mail address for the service account (I think that's been ruled out)
Invalid date/time stamp used for generating the signed blob (an issue date, and an expiration date)
Invalid scope (I think that's been ruled out)
You should check that your date/time is properly set on your server with the proper timezone -- sync'd to NTP. You can use time.gov to see the official US atomic clock time.

EDIT: The answer I gave below is relevant to using Google App Engine Service Accounts - leaving here for reference.
Double check that you have added your service account address to your project's team page as an owner.
I'd recommend using the AppIdentityCredential class to handle service account auth. Here's a small snippet that demonstrates this, and I'll add additional documentation about this on the BigQuery API developer page.
Also, make sure that you are using the latest version of the Google Java API client (as of today, it's version "v2-rev5-1.5.0-beta" here).
import java.io.IOException;
import javax.servlet.http.HttpServlet;
import javax.servlet.http.HttpServletRequest;
import javax.servlet.http.HttpServletResponse;
import com.google.api.client.googleapis.extensions.appengine.auth.oauth2.AppIdentityCredential;
import com.google.api.client.http.HttpTransport;
import com.google.api.client.http.javanet.NetHttpTransport;
import com.google.api.client.http.json.JsonHttpRequest;
import com.google.api.client.http.json.JsonHttpRequestInitializer;
import com.google.api.client.json.JsonFactory;
import com.google.api.client.json.jackson.JacksonFactory;
import com.google.api.services.bigquery.Bigquery;
import com.google.api.services.bigquery.BigqueryRequest;
#SuppressWarnings("serial")
public class Bigquery_service_accounts_demoServlet<TRANSPORT> extends HttpServlet {
// ENTER YOUR PROJECT ID HERE
private static final String PROJECT_ID = "";
private static final HttpTransport TRANSPORT = new NetHttpTransport();
private static final JsonFactory JSON_FACTORY = new JacksonFactory();
private static final String BIGQUERY_SCOPE = "https://www.googleapis.com/auth/bigquery";
AppIdentityCredential credential = new AppIdentityCredential(BIGQUERY_SCOPE);
Bigquery bigquery = Bigquery.builder(TRANSPORT,JSON_FACTORY)
.setHttpRequestInitializer(credential)
.setJsonHttpRequestInitializer(new JsonHttpRequestInitializer() {
public void initialize(JsonHttpRequest request) {
BigqueryRequest bigqueryRequest = (BigqueryRequest) request;
bigqueryRequest.setPrettyPrint(true);
}
}).build();
public void doGet(HttpServletRequest req, HttpServletResponse resp)
throws IOException {
resp.setContentType("text/plain");
resp.getWriter().println(bigquery.datasets()
.list(PROJECT_ID)
.execute().toString());
}
}

Here is a complete snippet for reference:
import com.google.api.client.http.HttpTransport;
import com.google.api.client.http.javanet.NetHttpTransport;
import com.google.api.client.json.JsonFactory;
import com.google.api.client.json.jackson.JacksonFactory;
import com.google.api.client.googleapis.auth.oauth2.GoogleCredential;
import com.google.api.services.bigquery.Bigquery;
import com.google.api.services.bigquery.Bigquery.Datasets;
import com.google.api.services.bigquery.model.DatasetList;
import java.io.File;
import java.io.IOException;
import java.security.GeneralSecurityException;
public class BigQueryJavaServiceAccount {
private static final String SCOPE = "https://www.googleapis.com/auth/bigquery";
private static final HttpTransport TRANSPORT = new NetHttpTransport();
private static final JsonFactory JSON_FACTORY = new JacksonFactory();
public static void main(String[] args) throws IOException, GeneralSecurityException {
GoogleCredential credential = new GoogleCredential.Builder().setTransport(TRANSPORT)
.setJsonFactory(JSON_FACTORY)
.setServiceAccountId("XXXXXXX#developer.gserviceaccount.com")
.setServiceAccountScopes(SCOPE)
.setServiceAccountPrivateKeyFromP12File(new File("my_file.p12"))
.build();
Bigquery bigquery = Bigquery.builder(TRANSPORT, JSON_FACTORY)
.setApplicationName("Google-BigQuery-App/1.0")
.setHttpRequestInitializer(credential).build();
Datasets.List datasetRequest = bigquery.datasets().list("publicdata");
DatasetList datasetList = datasetRequest.execute();
System.out.format("%s\n", datasetList.toPrettyString());
}