We Keep Coding

How to log in with a Google Service account in the Google Cloud?

I created a service account and generated a json key. I want to execute a get request to get a project.
I using https://cloud.google.com/dns/docs/reference/v1/projects/get
public static Dns createDnsService() throws IOException, GeneralSecurityException {
HttpTransport httpTransport = GoogleNetHttpTransport.newTrustedTransport();
JsonFactory jsonFactory = JacksonFactory.getDefaultInstance();
String jsonPath = "testdns-320312-136950c1074b.json";
FileInputStream stream = new FileInputStream(jsonPath);
GoogleCredential credential = GoogleCredential.fromStream(stream);
if (credential.createScopedRequired()) {
credential =credential.createScoped(Arrays.asList("https://www.googleapis.com/auth/cloud-platform"));
}
return new Dns.Builder(httpTransport, jsonFactory, credential)
.setApplicationName("Google-DnsSample/0.1")
.build();
}
When executing the code, I get error:
Exception in thread "main" com.google.api.client.googleapis.json.GoogleJsonResponseException: 403 Forbidden
GET https://dns.googleapis.com/dns/v1/projects/testdns-320312
{
"code" : 403,
"errors" : [ {
"domain" : "global",
"message" : "Forbidden",
"reason" : "forbidden"
} ],
"message" : "Forbidden"
}

#John Hanley is right - you lack proper permissions.
If you didn't assign any role to a service account after you created it you can't use it to do anything.
You need a roles/dns.admin role to be able to administer all the DNS records in your project.
You can do this with the following steps:
In the Google Cloud Console, go to the IAM page.
Go to the IAM page
Select your project from the top pull-down menu.
Click Add.
In New members, enter the email address of a new member.
Select the desired role from the drop-down menu.
Click Save.
Verify that the member is listed with the role that you granted.
Granting DNS admin role to this account will be the easiest way of getting control over Cloud DNS section of your project. If you need more fine-grained / limited access then you can create your custom role and assign it to this account instead of the basic one.
Here's more info about granting roles in GCP that you may find useful.
Very similar question was also answered here.

Bad request (invalid value) when trying to retrieve in-app purchase info using the Java Google Play Developer API v3 with a valid purchase token

When using the Java Google Play Developer API (version 3), and requesting purchase information for a valid purchase token, I get the below exception. The API call returns a 400 Bad Request response with the following message.
{
"code" : 400,
"errors" : [ {
"domain" : "global",
"message" : "Invalid Value",
"reason" : "invalid"
} ],
"message" : "Invalid Value"
}
Unfortunately there is no further information. I do not know what value is invalid.
First, we have contacted Google support about this. Unfortunately we have been going back and forth with Google support for almost 2 months and they still haven't even indicated they correctly understand what our problem is. Every time they reply, they give us suggestions that have literally nothing to do with the Google Play Developer API, even after thoroughly explaining the problem multiple times. This is the most frustrating support experience I've had in 20 years. Google support is clearly not going to be of any help here.
Second, we have successfully linked our Google Play account and our back-end. All necessary configuration is complete and working. On our back-end we are getting notifications from Google Play whenever an Android in-app purchase is made, and these notifications include a purchase token. This took some substantial effort, but I am confident this is now all correct, our service account is configured correctly, and all permissions and such are good.
I am able to successfully call the API that returns a list of available in-app-purchase products. I get a full and correct listing of the products we have configured in Google Play for users to purchase in our app. In order to make this call, we must provide credentials, and I assume the server is validating these credentials when we make this call. Since I can successfully make this call, I am assuming the "Invalid Value" is not related to our credentials. We are using a Google-generated JSON file created for a specific service account (not using a p12 file).
I have searched StackOverflow, and the rest of the internet looking for information on this issue. I have found many pages indicating the same or similar problem, but using an older version of the Google Play Developer API, which apparently did things differently. The solutions offered do not seem to apply to the v3 API. I have been searching for many weeks and trying various solutions I've come across all with no success.
I have written a test app that I run on my local PC which works when I am requesting a listing of in-app products, but does not work when requesting the purchase details for a given purchase token. I have tried with many different purchase tokens. I read that the test tokens are not actually valid for use with this particular API, so I actually released to production a build of our app that I am able to make a real, actual purchase with. I have done this, been charged by Google, and our back-end has received real and valid purchase tokens from these purchases.
private static void testGooglePlayDevAPI_Purchase() throws Throwable {
final String productIdSku = "[the_product_sku]";
final String purchaseToken = "[a_valid_purchase_token]";
final String packageName = "[our_app_id]";
final File GOOGLE_PLAY_DEV_API_CREDENTIALS =
new File("path_to_file/google-api-credentials.json");
final FileInputStream credentials =
new FileInputStream(GOOGLE_PLAY_DEV_API_CREDENTIALS);
final JsonFactory JSON_FACTORY = JacksonFactory.getDefaultInstance();
final HttpTransport TRANSPORT =
GoogleNetHttpTransport.newTrustedTransport();
String scope = AndroidPublisherScopes.ANDROIDPUBLISHER;
GoogleCredential credential = GoogleCredential
.fromStream(credentials, TRANSPORT, JSON_FACTORY)
.createScoped(Arrays.asList(scope));
AndroidPublisher publisher = new AndroidPublisher
.Builder(TRANSPORT, JSON_FACTORY, credential)
.setApplicationName(packageName)
.build();
try {
AndroidPublisher.Purchases purchases = publisher.purchases();
AndroidPublisher.Purchases.Products products = purchases.products();
AndroidPublisher.Purchases.Products.Get get =
products.get(packageName, productIdSku, purchaseToken);
ProductPurchase product = get.execute();
System.out.println(String.format(
"Product purchase data found [developerPayload=%s]",
product.getDeveloperPayload()));
} catch (Throwable t) {
System.out.println(ExceptionSupport.getExceptionDetails(t));
}
}
I expect the above code to give me a response with valid product purchase data (a valid ProductPurchase instance that I can interrogate and process). Instead, all I ever get here is a 400 Bad Request response indicating that SOME VALUE (I don't know which) is invalid. The actual response is:
com.google.api.client.googleapis.json.GoogleJsonResponseException:
400 Bad Request
{
"code" : 400,
"errors" : [ {
"domain" : "global",
"message" : "Invalid Value",
"reason" : "invalid"
} ],
"message" : "Invalid Value"
}
UPDATE:
I've provided my own answer to this question below.

I'll provide my own answer here because we finally found a work-around. It turns out that something in the Java Google Play Developer API (v3) client library is not working correctly. I don't know what it is, but the documentation for that client library is terrible and Google provides NO examples of using it. However, it turns out that we can successfully use the HTTPS/JSON/OAuth2 API directly without going through the Java client. We have implemented our server-side solution using this approach and caching the OAuth2 tokens in memcache until they expire.
I double-checked and we were emailing back-and-forth with Google support for 2 months!!! They were of no assistance at all. I find this to be remarkable (in a bad way).

Firestore import/export per API

I'm searching for a way to call the firestore import/export functionality programmatically from java code.
What i found so far is that the nice firestore client library does not yet support the import/export calls. But the more low level rest/grpc api already supports them. Using the java library i tried the following:
Firestore firestoreApi = new Firestore
.Builder(UrlFetchTransport.getDefaultInstance(), new GsonFactory(), null)
.setApplicationName(SystemProperty.applicationId.get())
.build();
GoogleFirestoreAdminV1beta2ImportDocumentsRequest importRequest = new GoogleFirestoreAdminV1beta2ImportDocumentsRequest();
importRequest.setInputUriPrefix(String.format("gs://{}/{}/", BUCKET, image));
GoogleLongrunningOperation operation = firestoreApi
.projects()
.databases()
.importDocuments("projects/" + SystemProperty.applicationId.get() + "/databases/(default)", importRequest)
.execute();
Which sadly ends with missing permissions when run in app engine:
com.google.api.client.googleapis.json.GoogleJsonResponseException: 401
{
"code": 401,
"errors": [
{
"domain": "global",
"location": "Authorization",
"locationType": "header",
"message": "Login Required.",
"reason": "required"
}
],
"message": "Request is missing required authentication credential. Expected OAuth 2 access token, login cookie or other valid authentication credential. See https://developers.google.com/identity/sign-in/web/devconsole-project.",
"status": "UNAUTHENTICATED"
I cannot get the official way to login to work, because the firestore builder does not have a method to accept a instance of AppEngineCredentials.
I already checked the python client library which also seems not support these methods (yet).
Does anyone have a idea how i can either login with the old rest api or get a client library which supports these methods (some language which runs on app engine please :) )
Thanks for reading!
Carsten

You can adapt this Cloud Datastore example for Cloud Firestore. See how they get an access token here:
import com.google.appengine.api.appidentity.AppIdentityService;
import com.google.appengine.api.appidentity.AppIdentityServiceFactory;
// Get an access token to authorize export request
ArrayList<String> scopes = new ArrayList<String>();
scopes.add("https://www.googleapis.com/auth/datastore");
final AppIdentityService appIdentity = AppIdentityServiceFactory.getAppIdentityService();
final AppIdentityService.GetAccessTokenResult accessToken =
AppIdentityServiceFactory.getAppIdentityService().getAccessToken(scopes);
connection.addRequestProperty("Authorization", "Bearer " + accessToken.getAccessToken());

Analytics API from app engine with AppIdentityCredential insufficient permissions

I am attempting to hit Google analytics API from my java app engine module.
I have the following code to retrieve the credential.
public HttpRequestInitializer getCredentials() throws GeneralSecurityException, IOException {
HttpRequestInitializer httpRequestInitializer;
if (com.google.appengine.api.utils.SystemProperty.environment.value() == com.google.appengine.api.utils.SystemProperty.Environment.Value.Development) {
String certificate = "/token.p12";
File privateKey = null;
try {
privateKey = new File(getClass().getResource(certificate).toURI());
} catch (URISyntaxException e) {
log.log(Level.SEVERE, e.getMessage(), e);
}
httpRequestInitializer = new GoogleCredential.Builder()
.setTransport(new NetHttpTransport())
.setJsonFactory(new JacksonFactory())
.setServiceAccountId("XXXX#developer.gserviceaccount.com")
.setServiceAccountPrivateKeyFromP12File(privateKey)
.setServiceAccountScopes(Collections.singleton(AnalyticsScopes.ANALYTICS_READONLY))
.build();
} else {
httpRequestInitializer = new AppIdentityCredential(Arrays.asList(AnalyticsScopes.ANALYTICS_READONLY));
}
return httpRequestInitializer;
}
I have enabled the analytics api for the project in the cloud dev console. In Google analytics I have given the service account read access. Google analytics is using the same google account as I am using for google cloud.
Anyway, when using the app engine dev server this code works and I can hit the api (using the client library). However, when run from app engine itself and it uses AppIdentityCredential I am getting the following exception when attempting to hit the API.
com.google.api.client.googleapis.json.GoogleJsonResponseException: 403 OK
{
"code" : 403,
"errors" : [ {
"domain" : "global",
"message" : "User does not have any Google Analytics account.",
"reason" : "insufficientPermissions"
} ],
"message" : "User does not have any Google Analytics account."
}
I would assume that AppIdentityCredential authenticates as the user who owns the app engine project, but I guess it doesn't?
I would like to use AppIdentityCredential as I understand that this is recommended for production instances.
How can I use AppIdentityCredential and access the analytics api with it?

Uh ok I figured it out.
The app engine project is tied to the identity/account of PROJECTNAME#appspot.gserviceaccount.com . Giving this account access in analytics settings allows the app engine code to access it using AppIdentityCredential

Google Drive API through Google App Engine

I'm trying to use the Google Drive API through the App Identity interface provided with Google App Engine. This basically allows my web application to communicate with Google's APIs from server to server.
I don't need my users to login, I simply need to display my own Google Drive documents.
However, after I set all the appropriate values and scopes, and enable all the right Google Drive knobs on the console page, I still get this for a simple GET request to https://www.googleapis.com/drive/v2/files:
{ "error": { "errors": [ { "domain": "usageLimits", "reason": "dailyLimitExceededUnreg", "message": "Daily Limit for Unauthenticated Use Exceeded. Continued use requires signup.", "extendedHelp": "https://code.google.com/apis/console" } ], "code": 403, "message": "Daily Limit for Unauthenticated Use Exceeded. Continued use requires signup." }}
What's wrong? What am I missing? Here's the code that actually does the request - funny thing is that it works great if I use other APIs such as the URL shortener API:
var scopes = new java.util.ArrayList();
scopes.add("https://www.googleapis.com/auth/drive");
var appIdentity = AppIdentityServiceFactory.getAppIdentityService();
var accessToken = appIdentity.getAccessToken(scopes);
var url = new URL("https://www.googleapis.com/drive/v2/files");
var connection = url.openConnection();
connection.setDoOutput(true);
connection.setRequestMethod("GET");
connection.addRequestProperty("Content-Type", "application/json");
connection.addRequestProperty("Authorization", "OAuth " + accessToken.getAccessToken());
EDIT
If I simply change the API to use the urlshortner API for example, it works:
var url = new URL("https://www.googleapis.com/urlshortener/v1/url/history");
And output:
{ "kind": "urlshortener#urlHistory", "totalItems": 0, "itemsPerPage": 30}
So there must be something not working with Google Drive and App Identity?
EDIT 2
I've found some help from the correct answer here: https://stackoverflow.com/a/12526286/50394
But it's talking about setting Client API scopes on Google Apps, and I'm not using Google Apps, I'm simply using Google App Engine's domain foo.appspot.com

The 403 error you are getting means that there was no Authorization header in your GET. The logic is that without an Authorization header, you are anonymous (you are legion blah blah :-)). The Drive quota for anonymous use is zero, hence the message. URL shortener has a higher quota for anonymous so it works.
I suggest you change the URL to point to an http server of your own, and check what headers you are actually sending.

AFAICT you should be using Bearer in the Authorization header.
Probably what's happening is, Drive API doesn't recognize the service account (because of the wrong header?) and thus taking it as an anonymous request since no key parameter wasn't provided either (see common query params).
Try this:
connection.addRequestProperty("Authorization", "Bearer " + accessToken.getAccessToken());
Or you could try adding the token as access_token query param.

I think you should at least setup an API console entry with Drive API enabled at https://code.google.com/apis/console
Once you create this you'll get an ID you can use in your GoogleCredential object. From the GoogleCredential object you can get the access token which you can than add to your request.

What I read here (Google drive via service accounts) was that you use a slightly different style that uses an API KEY that you retrieve from the Developer Console.
The pertinent parts for me were to generate a "Key for Server Applications", then use this technique, which I hadn't read anywhere else!
HttpTransport httpTransport = new NetHttpTransport();
JsonFactory jsonFactory = new JacksonFactory();
AppIdentityCredential credential =
new AppIdentityCredential.Builder(DriveScopes.DRIVE).build();
// API_KEY is from the Google Console as a server API key
GoogleClientRequestInitializer keyInitializer =
new CommonGoogleClientRequestInitializer(API_KEY);
Drive service = new Drive.Builder(httpTransport, jsonFactory, null)
.setHttpRequestInitializer(credential)
.setGoogleClientRequestInitializer(keyInitializer)
.build();

This answer claims that:
Service Accounts are not supported by the Drive SDK due to its
security model.
If that's still true, one workaround is to perform a regular OAuth dance once with a regular Google Account, and persist the access and refresh token in the datastore.