We Keep Coding

Upload a file to s3 bucket with REST API

Doing this with S3 SDK makes it simple. But want to go with S3 REST API(Read some advantages with this).
I have gone through with S3 API documentation and find difficult to code using it. I am totally new to this type of coding wherein it uses Request Parameters, Request Headers, Response Headers, Authorization, Error Codes, ACL etc. It also provided sample examples but could not find a way how to use those examples and do coding.
Can any one help where to start and end so that I can code for all CRUD operations on S3 using API. An example for uploading image file will helps me in better understanding.

I am putting some basic code snippets below, that you can easily integrate in your code.
Getting s3 client:
private AmazonS3 getS3Client() {
AmazonS3 s3Client = AmazonS3ClientBuilder.standard().withCredentials(credentials)
.withAccelerateModeEnabled(true).withRegion(Regions.US_EAST_1).build();
return s3Client;
}
Uploading file:
public void processOutput(FileServerDTO fileRequest) {
try {
AmazonS3 s3Client = getS3Client();
s3Client.putObject(fileRequest.getBucketName(), fileRequest.getKey(), fileRequest.getFileContent(), null);
} catch (Exception e) {
logger.error("Exception while uploading file" + e.getMessage());
throw e;
}
}
Downloading File:
public byte[] downloadFile(FileServerDTO fileRequest) {
AmazonS3 s3Client = getS3Client();
S3Object s3object = s3Client.getObject(new GetObjectRequest(fileRequest.getBucketName(), fileRequest.getKey()));
S3ObjectInputStream inputStream = s3object.getObjectContent();
try {
return FileCopyUtils.copyToByteArray(inputStream);
} catch (Exception e) {
logger.error("Exception while downloading file" + e.getMessage());
}
return null;
}
FileServerDTO contains basic attributes related to file info.
You can easily use these util methods in your service.

If you go through S3 services you will get better understanding of how S3 services works here are some example how to create upload delete files form S3 server using S3 servies:-
1) how to use Amazon’s S3 storage with the Java API
2) S3 Docs
There is brief explanation how it works.

Google API authentication in Java without using environment variable

I have set up a simple test app to interact with Google's Natural Language API. I created a service account, and downloaded the JSON credentials. I am running on a local development machine, so I set the GOOGLE_APPLICATION_CREDENTIALS environment variable to point to the JSON file. To be clear, this works: the app successfully makes some API calls and displays the results.
I would like to remove the dependence on the environment variable. How can I use the known location of the JSON file (or any other approach) in the application to create the LanguageServiceClient with those credentials?

You can register is like this:
DatastoreOptions options = DatastoreOptions.newBuilder()
.setProjectId(PROJECT_ID)
.setAuthCredentials(AuthCredentials.createForJson(
new FileInputStream(PATH_TO_JSON_KEY))).build();
Does that help?

You can always pass the full json file as String as follows:
CredentialsProvider credentialsProvider;
String credentials = "[YOUR JSON FILE CONTENT]";
try {
credentialsProvider
= FixedCredentialsProvider.create(
ServiceAccountCredentials.fromStream(new ByteArrayInputStream(credentials.getBytes())));
} catch (IOException ex) {
Logger.getLogger(GoogleNLPService.class.getName()).log(Level.SEVERE, null, ex);
}
LanguageServiceSettings.Builder languageServiceSettingsBuilder
= LanguageServiceSettings.newBuilder();
LanguageServiceSettings languageServiceSettings = languageServiceSettingsBuilder.setCredentialsProvider(credentialsProvider).build();
List<NamedEntity> entities = new ArrayList<>();
try (LanguageServiceClient language = LanguageServiceClient.create(languageServiceSettings)) {
...
}
Alternatively, you can place your json file in resources folder and then read it as:
credentialsProvider
= FixedCredentialsProvider.create(
ServiceAccountCredentials.fromStream(new FileInputStream("./src/main/resources/FILENAME.json")));
However, this relative path didn't work when I uploaded my app in Heroku. So, I have decided to use the String solution.

We use a service account + GoogleCredential.Builder -- (note that this example uses a credential file in p12 format); example follows:
private GoogleCredential authorize() throws IOException, GeneralSecurityException
{
return new GoogleCredential.Builder()
.setTransport(HTTP_TRANSPORT)
.setJsonFactory(JSON_FACTORY)
.setServiceAccountId(serviceAccount)
.setServiceAccountScopes(SCOPES)
.setServiceAccountUser(serviceAccountUser)
// variable p12File is a String w/ path to the .p12 file name
.setServiceAccountPrivateKeyFromP12File(new java.io.File(p12File))
.build();
}

Following the advice from tokyohans answer above, I can confirm this works for LanguageServiceClient:
// old-school Google Authentication
GoogleCredential credential = null;
credential = GoogleCredential.fromStream(new FileInputStream("google.json"));
Collection<String> scopes = Collections.singleton("https://www.googleapis.com/auth/cloud-language");
if (credential.createScopedRequired()) {
credential = credential.createScoped(scopes);
}
// copy over key values, note the additional "s", set some expiry
// com.google.auth.oauth2.GoogleCredentials
GoogleCredentials sac = ServiceAccountCredentials.newBuilder()
.setPrivateKey(credential.getServiceAccountPrivateKey())
.setPrivateKeyId(credential.getServiceAccountPrivateKeyId())
.setClientEmail(credential.getServiceAccountId())
.setScopes(scopes)
.setAccessToken(new AccessToken(credential.getAccessToken(), new LocalDate().plusYears(1).toDate()))
.build();
// Latest generation Google libs, GoogleCredentials extends Credentials
CredentialsProvider cp = FixedCredentialsProvider.create(sac);
LanguageServiceSettings settings = (LanguageServiceSettings) LanguageServiceSettings.newBuilder().setCredentialsProvider(cp).build();
return LanguageServiceClient.create(settings);

This looks like an older thread but sharing our findings for what it's worth.
This example is for the Google ImageAnnotatorClient, but I am pretty sure it is very similar for LanguageServiceClient.
Old-school google libraries (P12 file era) use GoogleCredential vs. the new GoogleCredentials. They look very similar. Digging into the Type hierarchy I found a FixedCredentialsProvider which seems to do the trick.
This worked for us, we got the Google Vision API up and running with an existing P12 file without environment variable. It does look like Google wants us to migrate away from this so not recommending this approach long-term.
// old-school Google Authentication
HttpTransport httpTransport = GoogleNetHttpTransport.newTrustedTransport();
JsonFactory jsonFactory = JacksonFactory.getDefaultInstance();
// Spring code
String pemFile = "yourPemFile.p12";
Resource r = new ClassPathResource(pemFile);
String serviceAccountEmail = "xxxx#xxxx.iam.gserviceaccount.com";
// com.google.api.client.googleapis.auth.oauth2.GoogleCredential.Builder
Builder credentialBuilder = new GoogleCredential.Builder()
.setTransport(httpTransport)
.setJsonFactory(jsonFactory)
.setServiceAccountId(serviceAccountEmail)
.setServiceAccountPrivateKeyFromP12File(r.getFile());
// Cloud API endpoints, make sure that the API is enabled
Collection<String> scopes = Collections.singleton("https://www.googleapis.com/auth/cloud-vision");
GoogleCredential credential = credentialBuilder
.setServiceAccountScopes(scopes).build();
// copy over key values, note the additional "s", set some expiry
// com.google.auth.oauth2.GoogleCredentials
GoogleCredentials sac = ServiceAccountCredentials.newBuilder()
.setPrivateKey(gc.getServiceAccountPrivateKey())
.setPrivateKeyId(gc.getServiceAccountPrivateKeyId())
.setClientEmail(gc.getServiceAccountId())
.setScopes(scopes)
.setAccessToken(new AccessToken(gc.getAccessToken(), new LocalDate().plusYears(1).toDate()))
.build();
// Latest generation Google libs, GoogleCredentials extends Credentials
CredentialsProvider cp = FixedCredentialsProvider.create(sac);
ImageAnnotatorSettings settings = ImageAnnotatorSettings.newBuilder().setCredentialsProvider(cp).build();
ImageAnnotatorClient googleApi = ImageAnnotatorClient.create(settings);

Java based Google App Engine and Google Drive API for Service Accounts

I have the following program on Eclipse in the Google App Engine Framework and it is working with Google Drive API. New files are created in Google Drive. I have a folder ID - that is being shared between the Service Account and my personal Gmail account. When I run the program on eclipse, I can see the files being generated on google drive.
However, when I deploy the same program on GAE, it fails to create the files on Google Drive. Any pointers that could help.
public class GDrive implements Callable<String> {
private static HttpTransport httpTransport;
private static String APPLICATION_NAME = "xxxxxxx";
private static String USER_EMAIL = "xxxxxxx#gmail.com";
private static JacksonFactory JSON_FACTORY = JacksonFactory.getDefaultInstance();
private static String SERVICE_ACCOUNT_EMAIL = "account-1#xxxxxxx.iam.gserviceaccount.com";
private static String CLIENT_ID = "xx06238724813717381290";
private static String KEY_PASSWORD = "notasecret";
private static String CLIENT_SECRET_P12_LOC = "file.p12";
MyConstants obMyConstants = new MyConstants();
public void insertLog(RootLog obRootLog) throws IOException, GeneralSecurityException {
String LogFolderId = obLBLSSS_Constants.LogFolderId;
//ID of the Parent Folder in Google Drive
httpTransport = GoogleNetHttpTransport.newTrustedTransport();
GoogleCredential credential = new GoogleCredential.Builder()
.setTransport(httpTransport)
.setJsonFactory(JSON_FACTORY)
.setServiceAccountId(SERVICE_ACCOUNT_EMAIL)
.setClientSecrets(CLIENT_ID, KEY_PASSWORD)
.setServiceAccountScopes(DriveScopes.all())
.setServiceAccountPrivateKeyFromP12File(
new java.io.File(CLIENT_SECRET_P12_LOC))
.build();
Drive driveService = new Drive.Builder(httpTransport, JSON_FACTORY,credential)
.setApplicationName(APPLICATION_NAME).build();
Gson gson = new GsonBuilder().create();
String eJsonOutput = gson.toJson(obRootLog);
Instant instant = Instant.now();
String filename = instant + "_" + obRootLog.requestURI;
// File's metadata.
File child = new File();
child.setTitle(filename);
child.setDescription("My File Description");
child.setMimeType("application/json");
child.setParents(
Arrays.asList(new ParentReference().setId(LogFolderId)));
// File's content.
java.io.File fileContent = new java.io.File(filename);
PrintWriter writer = new PrintWriter(fileContent, "UTF-8");
writer.println(eJsonOutput);
writer.close();
FileContent mediaContent = new FileContent("application/json", fileContent);
File filetemp = driveService.files().insert(child, mediaContent).execute();
}

The GAE file system is read-only, so you cannot write into a file like you attempt with
java.io.File fileContent = new java.io.File(filename);
PrintWriter writer = new PrintWriter(fileContent, "UTF-8");
writer.println(eJsonOutput);
writer.close();
Why don't you put your json data in a byte array and wrap
that with a ByteArrayInputStream. Like this:
InputStream bais = new ByteArrayInputStream(gson.toJson(obRootLog).getBytes());
Then use the input stream as a parameter for your
FileContent mediaContent = new InputStreamContent("application/json", bais);
call.
Additionally, you cannot use
httpTransport = GoogleNetHttpTransport.newTrustedTransport();
on App Engine. As descriped here you would use the UrlFetchTransport in AppEngine. It should look something like this:
httpTransport = new UrlFetchTransport.Builder().build();
By the way. You should add the error from your appengine logs since that makes it way easier to diagnose your code.

How do I access a Google Spreadsheet from App Engine using a Service Account?

I have an application on Google App Engine (hosted on appspot.com) which is going to be used by a few users to grab some stuff off the internet, save it in the Datastore and then write it to a Google Spreadsheet. I'm looking to use a Service Account to access the spreadsheet but I run into a 400 OK { "error" : "invalid_grant" } problem when I try to do anything, regardless of whether I try to create a file using the Drive API or access an existing file using Spreadsheets API.
I've turned on both Drive API and Drive SDK in the APIs Console and generated a P12 key. This is my method for building a GoogleCredential:
private GoogleCredential getGoogleCredential() throws IOException, GeneralSecurityException {
GoogleCredential credential = new GoogleCredential.Builder()
.setTransport(TRANSPORT)
.setJsonFactory(FACTORY)
.setServiceAccountId("1511XXX90247.apps.googleusercontent.com")
.setServiceAccountScopes(scopes)
.setServiceAccountPrivateKeyFromP12File(
new File("05a605b0fXXXd2a6be864be15d81a2bd629d3bd6-privatekey.p12"))
.setServiceAccountUser("XXX#gmail.com") // My personal e-mail address which I used to create the project
.build();
return credential;
}
The ServiceAccountID comes from the APIs Console. I've tried using 1511XXX90247#developer.gserviceaccount.com as my ServiceAccountUser as well as myappname#appspot.gserviceaccount.com. The scopes I'm using are as follows:
https://spreadsheets.google.com/feeds
https://docs.google.com/feeds
https://www.googleapis.com/auth/drive
https://www.googleapis.com/auth/drive.file
Here is the code that fails when I try to use Spreadsheets API:
SpreadsheetService service = new SpreadsheetService("name of my app");
service.setOAuth2Credentials(getGoogleCredential());
FeedURLFactory factory = FeedURLFactory.getDefault();
SpreadsheetQuery query = new SpreadsheetQuery(factory.getSpreadsheetsFeedUrl());
query.setTitleQuery("test");
SpreadsheetFeed feed = service.query(query, SpreadsheetFeed.class);
The code fails when the service tries execute the query.
Here's the Drive code I tried that also fails:
Drive drive = new Drive.Builder(TRANSPORT, FACTORY,
getGoogleCredential()).build();
com.google.api.services.drive.model.File file = new com.google.api.services.drive.model.File();
file.setTitle("test");
file.setMimeType("application/vnd.google-apps.spreadsheet");
drive.files().insert(file).execute(); // This is where the code fails
No matter what I try, I always seem to end up with the same ìnvalid grant` error. Here's a partial stacktrace:
com.google.gdata.util.AuthenticationException: Failed to refresh access token: 400 OK { "error" : "invalid_grant" }
at com.google.gdata.client.GoogleAuthTokenFactory$OAuth2Token.refreshToken(GoogleAuthTokenFactory.java:260)
at com.google.gdata.client.GoogleAuthTokenFactory.handleSessionExpiredException(GoogleAuthTokenFactory.java:702)
at com.google.gdata.client.GoogleService.handleSessionExpiredException(GoogleService.java:738) at com.google.gdata.client.GoogleService.getFeed(GoogleService.java:680)
at com.google.gdata.client.Service.query(Service.java:1237)
at com.google.gdata.client.Service.query(Service.java:1178)
at spam.gwt.scraper.server.spreadsheets.APIConnector.doGet(APIConnector.java:66)
I've tried to look for answers everywhere, including but not limited to the Google Developers Spreadsheets Guide, this Service Account-related Stack Overflow question and this Google App Engine-specific Stack Overflow question. What with the developments (eg. Docs -> Drive) it's hard to find up-to-date info, never mind info that pertains specifically to GAE.

I use the builtin Service Account of the App Engine application rather than creating my own; this way the private key is save (not stored locally, only on app engine).
For Drive:
protected Drive createDriveService() throws BookingSheetException {
HttpTransport httpTransport = new NetHttpTransport();
JsonFactory jsonFactory = new JacksonFactory();
GoogleClientRequestInitializer keyInitializer =
new CommonGoogleClientRequestInitializer(API_KEY);
AppIdentityCredential credential =
new AppIdentityCredential.Builder(Arrays.asList(DriveScopes.DRIVE)).build();
Drive service = new Drive.Builder(httpTransport, jsonFactory, null)
.setHttpRequestInitializer(credential)
.setGoogleClientRequestInitializer(keyInitializer)
.setApplicationName(appname).build();
return service;
}
For Spreadsheet:
private Credential creds;
/**
* Keep the expiration time of the access token to renew it before expiry
*/
private Calendar expirationTime = Calendar.getInstance();
protected void initCredentials() {
List<String> scopes = Arrays.asList("https://spreadsheets.google.com/feeds");
AppIdentityService appIdentity = AppIdentityServiceFactory.getAppIdentityService();
AppIdentityService.GetAccessTokenResult accessToken = appIdentity.getAccessToken(scopes);
expirationTime.setTime(accessToken.getExpirationTime());
expirationTime.add(Calendar.MINUTE,-REFRESH_MINUTES); //refresh some time before expiry
creds = new Credential(BearerToken.authorizationHeaderAccessMethod());
creds.setAccessToken(accessToken.getAccessToken());
}
public SpreadsheetUtil(String appname) {
myService = new SpreadsheetService(appname);
myService.setOAuth2Credentials(cred);
}
Make sure to creds.refreshToken() or reinitialize when they expire.

How to store a file as blob values using the file service in google cloud

With the help of using the Google document Writing files to a blob store
I can store the file in Google Cloud in below way,
//My servlet Class
private StorageService storage = new StorageService();
FileService fileService = FileServiceFactory.getFileService();
AppEngineFile file = fileService.createNewBlobFile(mime, fileName);
boolean lock = true;
FileWriteChannel writeChannel = fileService.openWriteChannel(file, lock);
byte[] b1 = new byte[BUFFER_SIZE];
int readBytes1;
storage.init(fileName, mime); // Calling my Java Class Here
while ((readBytes1 = is1.read(b1)) != -1) {
writeChannel.write(ByteBuffer.wrap(b1, 0, readBytes1));
storage.storeFile(b1, readBytes1);}
writeChannel.closeFinally();
I have a storage java class to store the data in Google cloud that is below here,
//My Storage Class
public class StorageService {
public static final String BUCKET_NAME = "MyBucket";
public void init(String fileName, String mime) throws Exception {
GSFileOptionsBuilder builder = new GSFileOptionsBuilder().setAcl("public_read").setBucket(BUCKET_NAME).setKey(fileName).setMimeType(mime);
AppEngineFile writableFile = fileService.createNewGSFile(builder.build());
boolean lock = true;
writeChannel = fileService.openWriteChannel(writableFile, lock);
bos = new BufferedOutputStream(Channels.newOutputStream(writeChannel));
public void storeFile(byte[] b, int readSize) throws Exception {
bos.write(b,0,readSize);
bos.flush();
}}
From the above reference i can store the file directly in Google cloud. But i need to store those files as blob values using the File service.
Can anyone suggest me an idea.
your help will be appreciated.

Have you add appengine service account into Google API Console Project ?
If you have not done, you should prepare Google API Console Project.
Confirm your appengine service account on "Application Settings" tab on GAE Admin Console
Create new Project on Google API Console. or Open API Console Project for your exists bucket.
Make enable "Google Cloud Storage" in "Services" tab.
Add appengine service account as team on API Console "Team" tab.