We Keep Coding

getting "401 Unauthorized" while trying to authenticated with service account

So here is the problem: Ive recently made this post.
Solution I've mentionned worked for one token and one API but when I tried to handle two APIs with two token (gmail and Sheets API) it failed.
So what I'm trying to do now is make the two work so I told myself "Hey let's create a service account". Even if I don't really understand the differences between both methods. Service account seems to prevent from having a consent screen (Am I right?).
I've crawled the web for answers but all of them seems to fail.
I've refreshed token, used GoogleCredential instead of Credential, created new key etc... one thing though I didn't tried is to use Gsuite account I'm using a basic account.
So now I'm at the point where I've created a new p12 file and instantly I get the 401 error. I will share my code for a better understanding.
my mail class
public class mailService {
private static final String APPLICATION_NAME = "AHS";
private static final JsonFactory JSON_FACTORY = JacksonFactory.getDefaultInstance();
private static final String TOKENS_DIRECTORY_PATH = "tokens";
//I've added sheet scope as it is activated in my project
private static final Collection<String> SCOPES = Arrays.asList(GmailScopes.GMAIL_SEND, GmailScopes.GMAIL_LABELS, SheetsScopes.SPREADSHEETS);
private static Credential getCredentials(final NetHttpTransport HTTP_TRANSPORT) throws IOException, GeneralSecurityException {
File sa = new File("WEB-INF/mykeyfile.p12");
Credential credential = new GoogleCredential.Builder()
.setTransport(HTTP_TRANSPORT)
.setJsonFactory(JSON_FACTORY)
.setServiceAccountId(
"myapp#appspot.gserviceaccount.com")
.setServiceAccountScopes(SCOPES)
.setServiceAccountPrivateKeyFromP12File(sa)
.setServiceAccountUser("myemailadress#gmail.com")
.build();
//credential.refreshToken();
return (credential);
}
public static Gmail getService() throws GeneralSecurityException, IOException {
final NetHttpTransport HTTP_TRANSPORT = GoogleNetHttpTransport.newTrustedTransport();
/*Gmail service = new Gmail.Builder(HTTP_TRANSPORT, JSON_FACTORY, getCredentials(HTTP_TRANSPORT))
.setApplicationName(APPLICATION_NAME)
.build();*/
Gmail service = new Gmail.Builder(HTTP_TRANSPORT, JSON_FACTORY, null)
.setHttpRequestInitializer(getCredentials(HTTP_TRANSPORT)).setApplicationName(APPLICATION_NAME).build();
return (service);
}
...
So to give you a better understanding, I'm creating a web app using Angular and Google app engine. I wan't to use Gmail API to send mail from my account, also I'm using sheets API to read/write from/to a spreadsheet. Just to be clear I have a secret file for the google-sign-in (for the user of the web app) but there this is server side code and I don't wan't user to see a consent screen.
I'm also asking myself if I need to use gcloud in order to activate service account.
I'm running (for the moment) my server locally using Eclipse and google app engine plugin.
if you need other code or precisions for better understanding of the problem let me know

Gmail dosent support service accounts unless its a gsuite account and you set up domain wide deligation.
If you check the documentation you will only see information about using Oauth2 not server account this is because Google only documents things that are supported not those that aren't.
Sheets does support service accounts just remembered that you need to pre-authorization on the service a account. That is done via sharing the sheet with the service account like you would any other user using the service accounts email address.

IllegalArgumentException When Accessing Google Sheets API

I'm attempting to read/write to Google Sheet (owned by me) via Google Sheets API v4. To get Credential object I'm using the following code:
private static Credential getCredentials(final NetHttpTransport HTTP_TRANSPORT) throws IOException {
// Load client secrets.
InputStream in = ResourceX.class.getResourceAsStream(CLIENT_SECRET_DIR);
GoogleClientSecrets clientSecrets = GoogleClientSecrets.load(JSON_FACTORY, new InputStreamReader(in));
System.out.println(clientSecrets.toPrettyString());
// When I print this ^^ it loads contents of secret key exactly as expected
// Build flow and trigger user authorization request.
Builder builder = new GoogleAuthorizationCodeFlow.Builder(HTTP_TRANSPORT, JSON_FACTORY, clientSecrets, SCOPES); // error occurs on this line
builder.setDataStoreFactory(new FileDataStoreFactory(new java.io.File(CREDENTIALS_FOLDER)));
GoogleAuthorizationCodeFlow flow = builder.setAccessType("offline").build();
return new AuthorizationCodeInstalledApp(flow, new LocalServerReceiver()).authorize("user");
}
I understand that IllegalArgumentException means that somehow the argument clientSecrets is illegal to be passed as parameter however when I print out the contents of json file (see print statement in code above) the contents seems to match the expected contents (I cannot print the contents on SO for security reasons but nothing was changed in the file since I downloaded it from Google).
I found a similar question here but there are no answers addressing the core question.
What possibly have I missed that may cause this exception?
Stacktrace:
Exception in thread "main" java.lang.IllegalArgumentException
at com.google.api.client.repackaged.com.google.common.base.Preconditions.checkArgument(Preconditions.java:111)
at com.google.api.client.util.Preconditions.checkArgument(Preconditions.java:37)
at com.google.api.client.googleapis.auth.oauth2.GoogleClientSecrets.getDetails(GoogleClientSecrets.java:82)
at com.google.api.client.googleapis.auth.oauth2.GoogleAuthorizationCodeFlow$Builder.<init>(GoogleAuthorizationCodeFlow.java:195)
at global_projects.GoogleSheetsAPI.getCredentials(GoogleSheetsAPI.java:100)
at global_projects.GoogleSheetsAPI.updateSheetByOauth2(GoogleSheetsAPI.java:116)
at amz.Main.main(Main.java:78)
Update:
I think I found the problem. clientSecrets expects a json file from OAuth 2.0 client ID not a Service account key/jsonfile (as described in this link). However, since I am accessing my own spreadsheet is there a way to access it using a service account key/json file or must I use OAuth which would require me to provide consent to edit my own spreadsheet?

There are 2 ways of accessing your Google sheets file using the Google Client API.
Create service account credentials for which you must give edit access in your particular sheet to the email which Google assigned for the account.
OR
Create Oauth2 client id credentials to use with generating a credential object. This method will prompt you, by way of return URL, to provide authorisation in browser. Once you approve, you can query for a refresh token that enables you to refresh your access token without needing the browser flow approval again.
You are using method (1) and you're getting the IllegalArgumenException error because you have not given access to the email assigned to the service account.

How do I search for a file/open/create a googlesheet in another users folder using a service account?

Here is the scenario. I have an account where I keep test result spreadsheets. I have a service account that I use to programatically (with java and google drive apiv3) to add/update those spreadsheets. I need a way to create a spreadsheet and to search for a spreadsheet in that users account and folder from the service account. I have the following code which gets a list of files, however it's the files inside the service accounts folder, not the user accounts folder. Any help?
private static boolean findSpreadsheetFile(String sSpreadsheetName)
throws GeneralSecurityException, IOException, URISyntaxException
{
String pageToken = null;
// Build a new authorized API client service.
Drive driveService = getDriveService();
// Print the names and IDs
do
{
FileList result = driveService.files().list()
.setPageToken(pageToken)
.setFields("nextPageToken, files(id, name)")
.execute();
for (com.google.api.services.drive.model.File file : result.getFiles())
{
System.out.printf("Found file: %s (%s)\n", file.getName(), file.getId());
}
pageToken = result.getNextPageToken();
}
while (pageToken != null);
return true;
}
public static Drive getDriveService() throws GeneralSecurityException, IOException, URISyntaxException
{
Credential credential = authorizeDrive();
return new Drive.Builder(HTTP_TRANSPORT, JSON_FACTORY, credential).setApplicationName(APPLICATION_NAME).build();
}
private static GoogleCredential authorizeDrive() throws GeneralSecurityException, IOException, URISyntaxException
{
JacksonFactory JSON_FACTORY = JacksonFactory.getDefaultInstance();
HttpTransport httpTransport = GoogleNetHttpTransport.newTrustedTransport();
URL fileUrl = ParseTestResultsEmails.class.getResource(sP12Filename);
GoogleCredential credential = new GoogleCredential.Builder().setTransport(httpTransport)
.setJsonFactory(JSON_FACTORY).setServiceAccountId(sServiceAccountID)
.setServiceAccountPrivateKeyFromP12File(new File(fileUrl.toURI()))
.setServiceAccountScopes(DRIVE_SCOPES).build();
return credential;
}

Based from this documentation, domain administrators can use service accounts with OAuth 2.0 to delegate authority this way.
In enterprise applications you may want to programmatically access users data without any manual authorization on their part. In G Suite domains, the domain administrator can grant to third party applications domain-wide access to its users' data — this is referred as domain-wide delegation of authority.
Warning: Service accounts should only be used for performing delegation where the effective identity is that of an individual user in a domain. Using the service account as a common owner to create many shared documents can have severe performance implications. Additionally, service accounts may not acquire additional storage quota, nor do they act as members of a domain.
You may see the Using OAuth 2.0 for Server to Server Applications documentation for more information.

Seems as if there is no way to do what I wanted in Google Drive using the API. What I ended up doing, was creating the sheet as the "Service Account", then transferring ownership to the user I wanted, then could share it to the groups that needed access. That worked great!

Status code 403 when trying to access DoubleClick Bid Manager Partner specific private buckets from google cloud java project

Hi all,
The current situation is as follows:
I currently have a google cloud project. The account that I log into the google cloud project with can also log into a DoubleClick bid Manager account. My aim is to use the DoubleClick Bid Manager api to retrieve certain buckets stored by DBM and save them in my separate google cloud project.
So far i can access the public buckets (gdbm-public) and pull and download the data, however when I try to access the partner specific buckets the same way, i.e. (gdbm-201032-201843) I get a status code 403.
Upon Reading the documentation here, I have discovered that I need to add a google group to the DBM partner information On DBM itself. However when i try to add a google group and save the changes i get an error saying the changes cannot be saved.
This is where i authenticate:
return new GoogleCredential.Builder().setTransport(HTTP_TRANSPORT)
.setJsonFactory(JSON_FACTORY)
.setServiceAccountId("<service_account_i_cant_show_here>")
.setServiceAccountScopes(Collections.singleton(StorageScopes.DEVSTORAGE_READ_ONLY))
.setServiceAccountPrivateKeyFromP12File(new File("secret-privatekey.p12"))
.build();
I then try to access the bucket like this:
String bucketName = "gdbm-201032-201843";
GoogleCredential credentials = getCredentials();
Storage storage = new Storage(HTTP_TRANSPORT, JSON_FACTORY, credentials);
Storage.Objects.List listObjects = storage.objects().list(bucketName);
Objects objects;
do {
objects = listObjects.execute();
for (StorageObject object : objects.getItems()) {
System.out.println(object);
}
listObjects.setPageToken(objects.getNextPageToken());
} while (null != objects.getNextPageToken());
More specifically, listObjects.execute() is where the 403 is thrown.
The areas I am trying to edit are Log Read Google Group and Log Management Google Group in the partner itself.
Any help greatly appreciated, thanks!

I think i have a solution, I used a different means of authentication as found here.
Using the linked class i entered in my google cloud project client credentials and the user that i log into both the google cloud project, and DBM.
I also changed the scope to "https://www.googleapis.com/auth/cloud-platform" however i am not 100% sure this had an effect on the outcome.
Code example:
// Scopes for the OAuth token
private static final Set<String> SCOPES =
ImmutableSet.of("https://www.googleapis.com/auth/cloud-platform");
public static Credential getUserCredential() throws Exception {
GoogleClientSecrets clientSecrets = GoogleClientSecrets.load(
JSON_FACTORY,
new InputStreamReader(SecurityUtilities.class.
getResourceAsStream("/client_secret.json")));
dataStoreFactory = new FileDataStoreFactory(DATA_STORE_DIR);
// set up authorization code flow.
GoogleAuthorizationCodeFlow flow = new GoogleAuthorizationCodeFlow.Builder(
HTTP_TRANSPORT, JSON_FACTORY, clientSecrets, SCOPES)
.setDataStoreFactory(dataStoreFactory)
.build();
// authorize and get credentials.
return new AuthorizationCodeInstalledApp(flow, new LocalServerReceiver())
.authorize("<Personal user account>");
So instead of using a service account i used a personal account.

(Android/Java) Google Sheets v4 isUserRecoverableError status: NEED_PERMISSION

I am currently trying to simply delete a row off my Google Spreadsheet using the Google Sheets v4 API.
Here is the code I am using:
private void deleteRow()
{
List<Request> requests = new ArrayList<>();
DeleteDimensionRequest deleteDimensionRequest = new DeleteDimensionRequest();
DimensionRange dimensionRange = new DimensionRange();
dimensionRange.setStartIndex(14);
dimensionRange.setEndIndex(15);
deleteDimensionRequest.setRange(dimensionRange);
requests.add(new Request()
.setDeleteDimension(deleteDimensionRequest)
);
BatchUpdateSpreadsheetRequest batchUpdateRequest = new BatchUpdateSpreadsheetRequest()
.setRequests(requests);
try
{
mService.spreadsheets().batchUpdate("Spreadsheet_ID", batchUpdateRequest).execute();
}
catch(IOException e)
{
e.printStackTrace();
}
}
The error this function gives me is:
08-14 15:47:10.818 26956-27285/com.xxx.xxxxx.xxxxxxxx
W/GoogleAuthUtil: isUserRecoverableError status: NEED_PERMISSION
In my other class file, I've already indicated the scopes of permissions including drive and spreadsheets.
Here is the picture of the error:
In the java quickstart...
public static Credential authorize() throws IOException {
// Load client secrets.
InputStream in =
SheetsQuickstart.class.getResourceAsStream("/client_secret.json");
GoogleClientSecrets clientSecrets =
GoogleClientSecrets.load(JSON_FACTORY, new InputStreamReader(in));
// Build flow and trigger user authorization request.
GoogleAuthorizationCodeFlow flow =
new GoogleAuthorizationCodeFlow.Builder(
HTTP_TRANSPORT, JSON_FACTORY, clientSecrets, SCOPES)
.setDataStoreFactory(DATA_STORE_FACTORY)
.setAccessType("offline")
.build();
Credential credential = new AuthorizationCodeInstalledApp(
flow, new LocalServerReceiver()).authorize("user");
System.out.println(
"Credentials saved to " + DATA_STORE_DIR.getAbsolutePath());
return credential;
}
Is this the sort of oath credential in addition to the one provided by the android quickstart that I need to include?

Based from the given JSON response, encountered error is due to insufficient authentication scope. You can try to check required OAuth 2.0 scope information for the Google Sheets API as given in Authorizing requests with OAuth 2.0.
Please note that requests to the Google Sheets API for non-public user data must be authorized by an authenticated user. Likewise, if an application needs to create spreadsheets, or otherwise manipulate their metadata, then the application must also request a Google Drive API scope.
Additionally, the solution given in this SO post - 403 Forbidden error when accessing Google Drive API downloadURL regarding error code 403 might also help.

The problem is likely that you are loaded stored credentials with the insufficient scopes, and you need to delete your stored credentials and authorize access again.
You code appears to be adapted from the Java quickstart, which requests a readonly scope. Trying to use that scope for a write operation would fail with this error.

Since there is not one single question relating to managing sheets with an android application, here is the way to request an additional OATH 2.0 from scratch, which took me a few hours to understand:
Strategy to get the right additional authorization...
https://developers.google.com/identity/protocols/OAuth2InstalledApp#libraries
Diagram:
All of these requests can be handled with some imports such as HttpUrlConnection... within AsyncTask.
The list of scopes...
https://developers.google.com/sheets/guides/authorizing