I am currently using Gmail API to send emails on user's behalf. The Mails are sent one by one and the average size of recipients is 500.
I frequently see {
"code" : 500,
"errors" : [ {
"domain" : "global",
"message" : "Backend Error",
"reason" : "backendError"
} ],
"message" : "Backend Error"
}
as well as some occurrences of
{
"code" : 429,
"errors" : [ {
"domain" : "usageLimits",
"message" : "Rate Limit Exceeded",
"reason" : "rateLimitExceeded"
} ],
"message" : "Rate Limit Exceeded"
}
Google has suggested implementing Exponential backoff strategy to resolve these errors. I have implemented below solution, but it doesn't seem to work and is not helping with these errors.Here is my implementation;
public GoogleCredential createCredentialWithRefreshToken(String accessToken, String refreshToken)
{
GoogleCredential credential = new GoogleCredential.Builder().setTransport(new NetHttpTransport())
.setJsonFactory(new JacksonFactory())
.setClientSecrets(Constants.GOOGLE_CLIENT_ID, Constants.GOOGLE_CLIENT_SECRET)
.setRequestInitializer(setHttpTimeout())
.build();
credential.setAccessToken(accessToken).setRefreshToken(refreshToken);
return credential;
}
public HttpRequestInitializer setHttpTimeout() {
return new HttpRequestInitializer() {
#Override
public void initialize(HttpRequest httpRequest) throws IOException {
httpRequest.setUnsuccessfulResponseHandler(new HttpBackOffUnsuccessfulResponseHandler(backOff()));
httpRequest.setConnectTimeout(3 * 60000); // 3 minutes connect timeout
httpRequest.setReadTimeout(3 * 60000); // 3 minutes read timeout
}
private final ExponentialBackOff.Builder BACK_OFF = new ExponentialBackOff.Builder().setInitialIntervalMillis(500);
private BackOff backOff() {
return BACK_OFF.build();
}
};
}
public static Gmail getGmailServiceForGoogleAccount(GoogleAccount googleAcct){
Gmail gmailService = null;
GoogleCredential credential = new Utils().createCredentialWithRefreshToken(googleAcct.getAccess_token(),googleAcct.getRefresh_token());
gmailService = new Gmail.Builder(new NetHttpTransport(),
new JacksonFactory(), credential)
.setApplicationName("test")
.build();
return gmailService;
}
What is wrong with this implementation? Am i implementing the custom HttpRequestInitializer correctly.
Where could i set the log statements to find out if a request is being retried as per Exponential policy?
Please suggest
I see this is an old question, but will leave my answer here in case anyone finds it useful.
The problem with the code is that it is calling .setRequestInitializer() on the GoogleCredential.Builder, which sets the initializer for token requests and not the service API requests.
See the documentation here
Sets the HTTP request initializer for refresh token requests to the token server or null for none.
Instead the initializer should be configured on the Google service client and you can chain it with the Credential response handler to preserve its functionality too.
Something like this should work for the provided example:
public static HttpRequestInitializer requestInitializer(Credential credential) {
return new HttpRequestInitializer() {
#Override
public void initialize(HttpRequest httpRequest) throws IOException {
httpRequest.setConnectTimeout(3 * 60000); // 3 minutes connect timeout
httpRequest.setReadTimeout(3 * 60000); // 3 minutes read timeout
// chain response handler with the handler from the credential
// that handles retries for authentication errors
HttpUnsuccessfulResponseHandler responseHandler =
new HttpBackOffUnsuccessfulResponseHandler(backOff());
httpRequest.setUnsuccessfulResponseHandler((req, res, retry) ->
credential.handleResponse(req, res, retry)
|| responseHandler.handleResponse(req, res, retry));
}
private final ExponentialBackOff.Builder BACK_OFF = new ExponentialBackOff.Builder().setInitialIntervalMillis(500);
private BackOff backOff() {
return BACK_OFF.build();
}
};
}
public static Gmail getGmailServiceForGoogleAccount(GoogleAccount googleAcct){
GoogleCredential credential = new Utils().createCredentialWithRefreshToken(googleAcct.getAccess_token(),googleAcct.getRefresh_token());
return new Gmail.Builder(new NetHttpTransport(), new JacksonFactory(), requestInitializer(credential))
.setApplicationName("test")
.build();
}
Check the Exponential Backoff for Java implementation:
ExponentialBackOff backoff = new ExponentialBackOff.Builder()
.setInitialIntervalMillis(500)
.setMaxElapsedTimeMillis(900000)
.setMaxIntervalMillis(6000)
.setMultiplier(1.5)
.setRandomizationFactor(0.5)
.build();
request.setUnsuccessfulResponseHandler(new HttpBackOffUnsuccessfulResponseHandler(backoff));
Check this SO post for additional reference.
Related
I'm creating a channel that receive changes on users that are on my application. The main problem is that after 2-3 webhooks, I receive an error that says that user has exceeded the quota limits.
That has no sense, because I only received 2 post message (I saw it on ngrok).
I've went on google console on drive API and quota. Each time I receive a webhook the amount of queries is increased by 500. So, when a user make two changes and I receive two webhooks, the number of queries exceed the 1000 allowed by google and I receive that error.
That's the code where I enable the channel:
#GET
#Path("/enable")
public void enable(#Context HttpServletRequest request, #Context HttpServletResponse response) throws IOException {
Credential credential = initFlow().loadCredential("user");
Drive service = new Drive.Builder(HTTP_TRANSPORT, JSON_FACTORY, credential).setApplicationName(APPLICATION_NAME).build();
Channel channel = new Channel();
channel.setId(UUID.randomUUID().toString());
channel.setType("web_hook");
channel.setAddress("https://389825dc.ngrok.io/GDriveRest/app/gdrive/webhook");
StartPageToken page = service.changes().getStartPageToken().execute();
GDrive.savedPageToken = page.getStartPageToken();
service.changes().watch(savedPageToken, channel).execute();
}
And the following one is the webhook:
#POST
#Path("/webhook")
public void webhook(#Context HttpServletRequest request, #Context HttpServletResponse response) throws IOException {
Credential credential = initFlow().loadCredential("user");
Drive service = new Drive.Builder(HTTP_TRANSPORT, JSON_FACTORY, credential).setApplicationName(APPLICATION_NAME).build();
String pageToken = savedPageToken;
while (pageToken != null) {
ChangeList changes = service.changes().list(pageToken).execute();
for (Change change : changes.getChanges()) {
Log.info("Change found for file: " + change.getFileId());
}
if (changes.getNewStartPageToken() != null) {
savedPageToken = changes.getNewStartPageToken();
}
pageToken = changes.getNewStartPageToken();
}
response.setStatus(200);
}
This is the error:
Caused by: com.google.api.client.googleapis.json.GoogleJsonResponseException: 403 Forbidden
{
"code" : 403,
"errors" : [ {
"domain" : "usageLimits",
"message" : "User Rate Limit Exceeded",
"reason" : "userRateLimitExceeded"
} ],
"message" : "User Rate Limit Exceeded"
}
Why this is happening?
There was an error in the code. I had to change the following:
PageToken = changes.getNewStartPageToken();
to
pageToken = changes.getNextPageToken();
The huge amount of queries was because I was requesting on every loop a new start page token, forcing an infinite loop.
The error you have encountered was discussed in this guide.
Suggested actions:
Raise the per-user quota in the Developer Console project.
If one user is making a lot of requests on behalf of many users of a G Suite domain, consider a Service Account with authority delegation
(setting the quotaUser parameter).
Use exponential backoff.
You can also try visiting this SO post for related issue.
I am trying to access Google Cloud Storage buckets objects with Java JSON API.
We are uses service account authentication. We also check our permission of service account ID as "Editor".
When we tried to list out bucket Object we got following error.
403 Forbidden
{
"code" : 403,
"errors" : [ {
"domain" : "global",
"message" : "Forbidden",
"reason" : "forbidden"
} ],
"message" : "Forbidden"
}
Authentication Code
HttpTransport transport = GoogleNetHttpTransport.newTrustedTransport();
JsonFactory jsonFactory = new JacksonFactory();
GoogleCredential credential = null;
InputStream credentialsStream = null;
try {
credentialsStream = new FileInputStream("<Location of Service Account JSON File>");
credential =GoogleCredential.fromStream(credentialsStream, transport, jsonFactory);
} catch (IOException e) {
} finally {
if (credentialsStream != null) {
credentialsStream.close();
}
}
if (credential.createScopedRequired()) {
Collection<String> scopes = StorageScopes.all();
credential = credential.createScoped(scopes);
}
return new Storage.Builder(transport, jsonFactory, credential)
.setApplicationName("DemoApplication")
.build();
Bucket Object Code
Storage client = StorageFactory.getService();
Storage.Objects.List listRequest = client.objects().list(bucketName);
List<StorageObject> results = new ArrayList<StorageObject>();
Objects objects;
// Iterate through each page of results, and add them to our results list.
do {
objects = listRequest.execute();
// Add the items in this page of results to the list we'll return.
results.addAll(objects.getItems());
// Get the next page, in the next iteration of this loop.
listRequest.setPageToken(objects.getNextPageToken());
} while (null != objects.getNextPageToken());
Please help me out for this issue.
I am using Retrofit 2 and before Firebase Auth I used to store my token in SharedPreferences and in my HttpInterceptor
#Named("rest_api")
#Singleton
#Provides
Interceptor provideRESTInterceptor(final UserManager userManager) {
return new Interceptor() {
#Override
public Response intercept(final Chain chain) throws IOException {
final Request original = chain.request();
Request request = original.newBuilder()
.header("Accept", "application/json")
.header("Authorization", "XXXX " + sharedPreference.getToken())
.method(original.method(), original.body())
.build();
Response response = chain.proceed(request);
return response;
}
};
}
How could i achieve something like this with
FirebaseUser.getToken(false/true) ? I don't know how to wait for the listener callback and than process the request with firebase token.
I am also thinking to check for token validity in here and if its about to expire getToken(true)
I am not sure to understand the whole problem, but to wait for the listener callback (in other words, making a synchronous method out of an asynchronous call) can be achieved like this :
private String getToken() {
final StringBuilder token = new StringBuilder() ;
final CountDownLatch countDownLatch = new CountDownLatch(1) ;
FirebaseAuth.getInstance().getCurrentUser().getToken(true).addOnCompleteListener(new OnCompleteListener<GetTokenResult>() {
#Override
public void onComplete(#NonNull Task<GetTokenResult> task) {
token.append(task.getResult().getToken());
countDownLatch.countDown();
}
});
try {
countDownLatch.await(30L, TimeUnit.SECONDS);
return token.toString() ;
} catch (InterruptedException ie) {
return null;
}
}
Notes :
StringBuilder is just used here as a String holder.
Bullet-proof implementation should of course check task.isSuccessful()
Timeout (here 30 sec) should be adapted to your situation
Hope this helps.
I'm working on a project that uses the Google Apps Reseller API (Found here).
I'm running into a 403 Forbidden Exception.
Code (most of it origins from the Google Codelab Example Here:
try {
try {
Reseller service = GoogleResellerApiUtil.getResellerService();
Customer customerRecord = service.customers().get("acme.com").execute(); //crashes here
// "acme.com" is also used in the example from Google
System.out.println(customerRecord.toString());
} catch (GoogleJsonResponseException e) {
e.printStackTrace();
}
And this is the class I use to connect to the API.
I've provided a p12 file and it uses the service account, when calling the API it is impersonating one of the super admins, so it should be allowed to make all the calls.
At the moment I'm only using the read-only scope.
public class GoogleResellerApiUtil {
/** HTTP_TRANSPORT */
private static final HttpTransport HTTP_TRANSPORT = new NetHttpTransport();
/** JSON Factory*/
private static final JsonFactory JSON_FACTORY = new JacksonFactory();
/** Service Account Email */
public static final String SERVICE_ACCOUNT_EMAIL = "****#appspot.gserviceaccount.com";
/** P12 File Location */
public static final String PRIVATE_KEY_FILE = "WEB-INF/key.p12";
/** Reseller Admin Account to impersonate */
public static final String RESELLER_ADMIN = "**.**#**.com";
/** Scopes */
public static final List<String> SCOPES = Arrays.asList(ResellerScopes.APPS_ORDER_READONLY);
/** Application name. */
private static final String APPLICATION_NAME = "**-subscription-portal";
/** Logger */
private final static Logger LOGGER =
Logger.getLogger(GoogleResellerApiUtil.class.getName());
public static GoogleCredential getCredentials() throws IOException {
GoogleCredential credentials = null;
try {
credentials = new GoogleCredential.Builder()
.setTransport(HTTP_TRANSPORT)
.setJsonFactory(JSON_FACTORY)
.setServiceAccountId(SERVICE_ACCOUNT_EMAIL)
.setServiceAccountScopes(SCOPES)
.setServiceAccountUser(RESELLER_ADMIN)
.setServiceAccountPrivateKeyFromP12File(new File(PRIVATE_KEY_FILE))
.build();
} catch (GeneralSecurityException e) {
e.printStackTrace();
}
System.out.println("credential has been build, returning credential "); //this gets printed, so I think the credentials are valid?
return credentials;
}
/**
* Build and return an authorized Reseller client service.
* #return an authorized Reseller client service
* #throws IOException
*/
public static Reseller getResellerService() throws Exception {
Credential credential = getCredentials();
return new Reseller.Builder(
HTTP_TRANSPORT, JSON_FACTORY, credential)
.setApplicationName(APPLICATION_NAME)
.build();
}
}
But I get the following error message when making the call:
com.google.api.client.googleapis.json.GoogleJsonResponseException: 403 OK
{
"code" : 403,
"errors" : [ {
"domain" : "global",
"message" : "Forbidden",
"reason" : "forbidden"
} ],
"message" : "Forbidden"
}
at com.google.api.client.googleapis.json.GoogleJsonResponseException.from(GoogleJsonResponseException.java:146)
etc. etc. etc.
It is noted in Reseller API: Manage Subscriptions that
Note: If the customerAuthToken is not valid or has expired, the API response returns a 403 "Forbidden" error.
To solve the issue, please make sure that requests must be authorized by an authenticated user who has access to the data. As also noted in Reseller API: Authorizing
Note: The user granting permission for the Reseller API must be a domain super administrator.
In addition to that, it was suggested in Token expiration that you write your code to anticipate the possibility that a granted token might no longer work. A token might stop working for one of these reasons:
The user has revoked access.
The token has not been used for six months.
The user changed passwords and the token contains Gmail scopes.
The user account has exceeded a certain number of token requests.
Hope that helps!
I try to get the userinfo after successfully authenticate with a gmail account (tok is a valid token):
GoogleCredential credential2 = new GoogleCredential.Builder()
.setTransport(TRANSPORT).setJsonFactory(JSON_FACTORY)
.setClientSecrets(CLIENT_ID, CLIENT_SECRET)
.setRequestInitializer((new HttpRequestInitializer() {
#Override
public void initialize(HttpRequest request)
throws IOException {
request.getHeaders().setAuthorization("Bearer ".concat(tok));
}
}))
.build();
Oauth2 userInfoService = new Oauth2.Builder(TRANSPORT,
JSON_FACTORY, credential2.getRequestInitializer())
.setApplicationName(APPLICATION_NAME).build();
Userinfo userInfo = userInfoService.userinfo().get().execute();
logger.warn("User email: {}", userInfo.getEmail());
logger.warn("User gender: {}", userInfo.getGender());
logger.warn("User complet name: {} - {}", userInfo.getFamilyName(), userInfo.getName());
But logs display 'null' for all fields, the json data returned contains only the id:
{
"id": "113695880661351193041"
}
What i'm supposed to do ? Add a special scope to do this? I tried it several times without success, just by adding scope=https://www.googleapis.com/auth/userinfo.profile as url parameter, maybe that's wrong ?
Hope someone can help or know how to add scopes to my request and get the correct response from this service.
This works for me :
Credential credential = OAuth2Utils.newFlow().loadCredential(userId);
Oauth2 service = new Oauth2.Builder(OAuth2Utils.HTTP_TRANSPORT, OAuth2Utils.JSON_FACTORY, credential).setApplicationName("appname").build();
UserInfo userInfo = service.userinfo().get().execute();
These are some of the properties that are returned :
userInfo.getBirthday()
userInfo.getFamilyName()
userInfo.getGender()
userInfo.getGivenName()
userInfo.getHd()
userInfo.getLink()
The utility class OAuth2Utils referred in the code :
/** Global instance of the HTTP transport. */
public final static HttpTransport HTTP_TRANSPORT = new UrlFetchTransport();
/** Global instance of the JSON factory. */
public final static com.google.api.client.json.JsonFactory JSON_FACTORY = new com.google.api.client.json.jackson2.JacksonFactory();
public static GoogleAuthorizationCodeFlow newFlow() throws IOException {
return new GoogleAuthorizationCodeFlow.Builder(HTTP_TRANSPORT, JSON_FACTORY, getClientCredential(),
Arrays.asList(SCOPES)).setCredentialStore(new OAuth2CredentialStore()).setAccessType("offline")
.setApprovalPrompt("force").build();
}