We Keep Coding

getting "401 Unauthorized" while trying to authenticated with service account

So here is the problem: Ive recently made this post.
Solution I've mentionned worked for one token and one API but when I tried to handle two APIs with two token (gmail and Sheets API) it failed.
So what I'm trying to do now is make the two work so I told myself "Hey let's create a service account". Even if I don't really understand the differences between both methods. Service account seems to prevent from having a consent screen (Am I right?).
I've crawled the web for answers but all of them seems to fail.
I've refreshed token, used GoogleCredential instead of Credential, created new key etc... one thing though I didn't tried is to use Gsuite account I'm using a basic account.
So now I'm at the point where I've created a new p12 file and instantly I get the 401 error. I will share my code for a better understanding.
my mail class
public class mailService {
private static final String APPLICATION_NAME = "AHS";
private static final JsonFactory JSON_FACTORY = JacksonFactory.getDefaultInstance();
private static final String TOKENS_DIRECTORY_PATH = "tokens";
//I've added sheet scope as it is activated in my project
private static final Collection<String> SCOPES = Arrays.asList(GmailScopes.GMAIL_SEND, GmailScopes.GMAIL_LABELS, SheetsScopes.SPREADSHEETS);
private static Credential getCredentials(final NetHttpTransport HTTP_TRANSPORT) throws IOException, GeneralSecurityException {
File sa = new File("WEB-INF/mykeyfile.p12");
Credential credential = new GoogleCredential.Builder()
.setTransport(HTTP_TRANSPORT)
.setJsonFactory(JSON_FACTORY)
.setServiceAccountId(
"myapp#appspot.gserviceaccount.com")
.setServiceAccountScopes(SCOPES)
.setServiceAccountPrivateKeyFromP12File(sa)
.setServiceAccountUser("myemailadress#gmail.com")
.build();
//credential.refreshToken();
return (credential);
}
public static Gmail getService() throws GeneralSecurityException, IOException {
final NetHttpTransport HTTP_TRANSPORT = GoogleNetHttpTransport.newTrustedTransport();
/*Gmail service = new Gmail.Builder(HTTP_TRANSPORT, JSON_FACTORY, getCredentials(HTTP_TRANSPORT))
.setApplicationName(APPLICATION_NAME)
.build();*/
Gmail service = new Gmail.Builder(HTTP_TRANSPORT, JSON_FACTORY, null)
.setHttpRequestInitializer(getCredentials(HTTP_TRANSPORT)).setApplicationName(APPLICATION_NAME).build();
return (service);
}
...
So to give you a better understanding, I'm creating a web app using Angular and Google app engine. I wan't to use Gmail API to send mail from my account, also I'm using sheets API to read/write from/to a spreadsheet. Just to be clear I have a secret file for the google-sign-in (for the user of the web app) but there this is server side code and I don't wan't user to see a consent screen.
I'm also asking myself if I need to use gcloud in order to activate service account.
I'm running (for the moment) my server locally using Eclipse and google app engine plugin.
if you need other code or precisions for better understanding of the problem let me know

Gmail dosent support service accounts unless its a gsuite account and you set up domain wide deligation.
If you check the documentation you will only see information about using Oauth2 not server account this is because Google only documents things that are supported not those that aren't.
Sheets does support service accounts just remembered that you need to pre-authorization on the service a account. That is done via sharing the sheet with the service account like you would any other user using the service accounts email address.

Drive API:Insufficient permissions when changing owner

I have made a program in Java that generates a spreadsheet filled with statistics for VEX teams. My whole goal is to essentially have a program be able to generate a spreadsheet and change that sheet's ownership to a specific email. The Sheets functionality works completely, being able to create and modify spreadsheet values easily(using the Sheets API). The problem is when I try to change the ownership of the file using the Drive API, I get an "Insufficient Permission" message when running Drive.permissions().create.
When using the APIs explorer and testing it out myself, I was able to transfer ownership of spreadsheets, but I cannot do it within Java.
Here is how I create my GoogleCredential:
private void setCredential() throws GeneralSecurityException, IOException {
//Create new transport
HttpTransport httpTransport = GoogleNetHttpTransport.newTrustedTransport();
//Build authenticated credential
GoogleCredential credential = new GoogleCredential.Builder()
.setTransport(httpTransport)
.setJsonFactory(jsonFactory)
.setClientSecrets(Constants.GOOGLE_CLIENT_ID,
Constants.GOOGLE_CLIENT_SECRET)
.build();
credential.setAccessToken(this.accessToken)
.setRefreshToken(Constants.GOOGLE_REFRESH_TOKEN);
//Set class field
this.credential = credential;
}
I noticed that I do not have any actual scopes here(which is weird, since this credential still works when the Sheets API does its work), in which I have looked around on how to input a proper scope here but found nothing. I don't know if this is why my program errors, or if it is for another reason.
This is how I call the method to transfer ownership:
private void transferOwnership(Drive driveService) throws IOException {
//Print message
System.out.printf("Transferring ownership to %s", this.usrEmail);
//Build request body
Permission body = new Permission()
.setRole("owner")
.setType("user")
.setEmailAddress(this.usrEmail);
//Execute Drive request
Permission permission = driveService.permissions().create(this.spreadsheetId, body)
.setFileId(this.spreadsheetId)
.setEmailMessage("Test - Replace with something")
.setSendNotificationEmail(true)
.setSupportsTeamDrives(true)
.setTransferOwnership(true)
.setUseDomainAdminAccess(false)
.setFields("emailAddress")
.execute();
//Print message
System.out.printf("Ownership successfully transferred to %s", this.usrEmail);
}
These are my questions:
Am I building my credential wrong? If so, how should I properly build it?
Do I need to make two different credentials?(one for Sheets API, one for Drive API)
How does my credential work fine with the Sheets API even though a scope isn't defined, but it does not work with the Drive API
EDIT: The way I use the APIs explorer to change ownership
EDIT 2: This is how I get my access token. As for the refresh token, I use the Oauth Playground from google.
public void setAccessToken() throws IOException, GeneralSecurityException{
//Create a token response using refresh token and oauth credentials
this.token_response = new GoogleRefreshTokenRequest(GoogleNetHttpTransport.newTrustedTransport(), JacksonFactory.getDefaultInstance(),
Constants.GOOGLE_REFRESH_TOKEN, Constants.GOOGLE_CLIENT_ID, Constants.GOOGLE_CLIENT_SECRET)
.execute();
//Set the access token
this.accessToken = token_response.getAccessToken();
}

How do I search for a file/open/create a googlesheet in another users folder using a service account?

Here is the scenario. I have an account where I keep test result spreadsheets. I have a service account that I use to programatically (with java and google drive apiv3) to add/update those spreadsheets. I need a way to create a spreadsheet and to search for a spreadsheet in that users account and folder from the service account. I have the following code which gets a list of files, however it's the files inside the service accounts folder, not the user accounts folder. Any help?
private static boolean findSpreadsheetFile(String sSpreadsheetName)
throws GeneralSecurityException, IOException, URISyntaxException
{
String pageToken = null;
// Build a new authorized API client service.
Drive driveService = getDriveService();
// Print the names and IDs
do
{
FileList result = driveService.files().list()
.setPageToken(pageToken)
.setFields("nextPageToken, files(id, name)")
.execute();
for (com.google.api.services.drive.model.File file : result.getFiles())
{
System.out.printf("Found file: %s (%s)\n", file.getName(), file.getId());
}
pageToken = result.getNextPageToken();
}
while (pageToken != null);
return true;
}
public static Drive getDriveService() throws GeneralSecurityException, IOException, URISyntaxException
{
Credential credential = authorizeDrive();
return new Drive.Builder(HTTP_TRANSPORT, JSON_FACTORY, credential).setApplicationName(APPLICATION_NAME).build();
}
private static GoogleCredential authorizeDrive() throws GeneralSecurityException, IOException, URISyntaxException
{
JacksonFactory JSON_FACTORY = JacksonFactory.getDefaultInstance();
HttpTransport httpTransport = GoogleNetHttpTransport.newTrustedTransport();
URL fileUrl = ParseTestResultsEmails.class.getResource(sP12Filename);
GoogleCredential credential = new GoogleCredential.Builder().setTransport(httpTransport)
.setJsonFactory(JSON_FACTORY).setServiceAccountId(sServiceAccountID)
.setServiceAccountPrivateKeyFromP12File(new File(fileUrl.toURI()))
.setServiceAccountScopes(DRIVE_SCOPES).build();
return credential;
}

Based from this documentation, domain administrators can use service accounts with OAuth 2.0 to delegate authority this way.
In enterprise applications you may want to programmatically access users data without any manual authorization on their part. In G Suite domains, the domain administrator can grant to third party applications domain-wide access to its users' data — this is referred as domain-wide delegation of authority.
Warning: Service accounts should only be used for performing delegation where the effective identity is that of an individual user in a domain. Using the service account as a common owner to create many shared documents can have severe performance implications. Additionally, service accounts may not acquire additional storage quota, nor do they act as members of a domain.
You may see the Using OAuth 2.0 for Server to Server Applications documentation for more information.

Seems as if there is no way to do what I wanted in Google Drive using the API. What I ended up doing, was creating the sheet as the "Service Account", then transferring ownership to the user I wanted, then could share it to the groups that needed access. That worked great!

Status code 403 when trying to access DoubleClick Bid Manager Partner specific private buckets from google cloud java project

Hi all,
The current situation is as follows:
I currently have a google cloud project. The account that I log into the google cloud project with can also log into a DoubleClick bid Manager account. My aim is to use the DoubleClick Bid Manager api to retrieve certain buckets stored by DBM and save them in my separate google cloud project.
So far i can access the public buckets (gdbm-public) and pull and download the data, however when I try to access the partner specific buckets the same way, i.e. (gdbm-201032-201843) I get a status code 403.
Upon Reading the documentation here, I have discovered that I need to add a google group to the DBM partner information On DBM itself. However when i try to add a google group and save the changes i get an error saying the changes cannot be saved.
This is where i authenticate:
return new GoogleCredential.Builder().setTransport(HTTP_TRANSPORT)
.setJsonFactory(JSON_FACTORY)
.setServiceAccountId("<service_account_i_cant_show_here>")
.setServiceAccountScopes(Collections.singleton(StorageScopes.DEVSTORAGE_READ_ONLY))
.setServiceAccountPrivateKeyFromP12File(new File("secret-privatekey.p12"))
.build();
I then try to access the bucket like this:
String bucketName = "gdbm-201032-201843";
GoogleCredential credentials = getCredentials();
Storage storage = new Storage(HTTP_TRANSPORT, JSON_FACTORY, credentials);
Storage.Objects.List listObjects = storage.objects().list(bucketName);
Objects objects;
do {
objects = listObjects.execute();
for (StorageObject object : objects.getItems()) {
System.out.println(object);
}
listObjects.setPageToken(objects.getNextPageToken());
} while (null != objects.getNextPageToken());
More specifically, listObjects.execute() is where the 403 is thrown.
The areas I am trying to edit are Log Read Google Group and Log Management Google Group in the partner itself.
Any help greatly appreciated, thanks!

I think i have a solution, I used a different means of authentication as found here.
Using the linked class i entered in my google cloud project client credentials and the user that i log into both the google cloud project, and DBM.
I also changed the scope to "https://www.googleapis.com/auth/cloud-platform" however i am not 100% sure this had an effect on the outcome.
Code example:
// Scopes for the OAuth token
private static final Set<String> SCOPES =
ImmutableSet.of("https://www.googleapis.com/auth/cloud-platform");
public static Credential getUserCredential() throws Exception {
GoogleClientSecrets clientSecrets = GoogleClientSecrets.load(
JSON_FACTORY,
new InputStreamReader(SecurityUtilities.class.
getResourceAsStream("/client_secret.json")));
dataStoreFactory = new FileDataStoreFactory(DATA_STORE_DIR);
// set up authorization code flow.
GoogleAuthorizationCodeFlow flow = new GoogleAuthorizationCodeFlow.Builder(
HTTP_TRANSPORT, JSON_FACTORY, clientSecrets, SCOPES)
.setDataStoreFactory(dataStoreFactory)
.build();
// authorize and get credentials.
return new AuthorizationCodeInstalledApp(flow, new LocalServerReceiver())
.authorize("<Personal user account>");
So instead of using a service account i used a personal account.

OAuth Google API for Java unable to impersonate user

I would like to impersonate a user and add files to the users Google Drive on their behalf from a server process. I've setup a service account and can successfully access the Drive as the service account adding and listing files, etc. using the following code:
/** Global instance of the HTTP transport. */
private static final HttpTransport HTTP_TRANSPORT = new NetHttpTransport();
/** Global instance of the JSON factory. */
private static final JsonFactory JSON_FACTORY = new JacksonFactory();
public static void main(String[] args) {
try {
GoogleCredential credential =
new GoogleCredential.Builder().setTransport(HTTP_TRANSPORT)
.setJsonFactory(JSON_FACTORY)
.setServiceAccountId("XXXXX#developer.gserviceaccount.com")
.setServiceAccountScopes(DriveScopes.DRIVE)
.setServiceAccountPrivateKeyFromP12File(new File("c:/junk/key.p12"))
.build();
Drive drive = new Drive.Builder(HTTP_TRANSPORT, JSON_FACTORY, credential).build();
drive.files().list().execute();
} catch (Exception e) {
e.printStackTrace();
}
This works, however only returns files that are associated to what I assume is associated with the service accounts drive (?).
According to the JavaDoc, GoogleCredential can also be used to impersonate a user by adding the service account users email address as follows:
GoogleCredential credential =
new GoogleCredential.Builder().setTransport(HTTP_TRANSPORT)
.setJsonFactory(JSON_FACTORY)
.setServiceAccountId("XXXXX#developer.gserviceaccount.com")
.setServiceAccountScopes(DriveScopes.DRIVE)
.setServiceAccountPrivateKeyFromP12File(new File("c:/junk/key.p12"))
.setServiceAccountUser("usera#domain.com") //<-- impersonate user a
.build();
However, when executing this code, the following exception is thrown:
com.google.api.client.auth.oauth2.TokenResponseException: 400 Bad Request
{
"error" : "access_denied"
}
at com.google.api.client.auth.oauth2.TokenResponseException.from(TokenResponseException.java:103)
at com.google.api.client.auth.oauth2.TokenRequest.executeUnparsed(TokenRequest.java:303)
at com.google.api.client.auth.oauth2.TokenRequest.execute(TokenRequest.java:323)
at com.google.api.client.googleapis.auth.oauth2.GoogleCredential.executeRefreshToken(GoogleCredential.java:340)
at com.google.api.client.auth.oauth2.Credential.refreshToken(Credential.java:508)
at com.google.api.client.auth.oauth2.Credential.intercept(Credential.java:260)
at com.google.api.client.http.HttpRequest.execute(HttpRequest.java:796)
at com.google.api.client.googleapis.json.GoogleJsonResponseException.execute(GoogleJsonResponseException.java:198)
at com.google.api.client.googleapis.services.GoogleClient.executeUnparsed(GoogleClient.java:237)
at com.google.api.client.http.json.JsonHttpRequest.executeUnparsed(JsonHttpRequest.java:207)
at com.google.api.services.drive.Drive$Files$List.execute(Drive.java:1071)
Am I missing a step or configuration setting?
Thanks,
David

I found a similar question as mine: Can a Google Apps Admin manage users files with Drive SDK? to mine which has helped me figure out the answer.
The cPanel documentation is a little misleading as it refers to enabling the consumer key and then adding the domain to the Manage API client access screen. This appears to be valid for the gdata api and not the new Google Drive api. By adding the client id as suggested in the other question and granting access to the Drive scope I'm now able to impersonate a user.

Get your admin to add scopes to xxxxx.apps.googleusercontent.com via admin panel:
I added the following to work on spreadsheets:
https://www.googleapis.com/auth/drive
https://docs.google.com/feeds
https://spreadsheets.google.com/feeds