Store Blob in Heroku (or similar cloud services) - java

I want to deploy an app in Heroku to try their new Play! Framework support. For what I've read in the site (I gotta confess I did not try it yet) they don't provide any file system. This means that (probably) Blob fields used in Play to store files won't work properly.
Could somebody:
Confirm if you can use the Play Blob in Heroku?
Provide the "best" alternative to store files in Heroku? Is better to store them in the database (they use PostgreSQL) or somewhere else?


I put an example of how to do this with Amazon S3 on github:
https://github.com/jamesward/plays3upload
Basically you just need to send the file to S3 and save the key in the entity:
AWSCredentials awsCredentials = new BasicAWSCredentials(System.getenv("AWS_ACCESS_KEY"), System.getenv("AWS_SECRET_KEY"));
AmazonS3 s3Client = new AmazonS3Client(awsCredentials);
s3Client.createBucket(BUCKET_NAME);
String s3Key = UUID.randomUUID().toString();
s3Client.putObject(BUCKET_NAME, s3Key, attachment);
Document doc = new Document(comment, s3Key, attachment.getName());
doc.save();
listUploads();

Related

What is the best way to create multipart upload to a presigned url in aws s3 bucket?

I want to generate a presign url(with upload rights only) and share it with my second application, so the second application can upload multipart file to it. How can I achieve this with java + spring boot ?
P.S. all the solutions I googled work only for single upload but not for multipart.

You can use GeneratePresignedUrlRequest, try this ;
GeneratePresignedUrlRequest generatePresignedUrlRequest = new GeneratePresignedUrlRequest(S3_BUCKET_NAME, key);
generatePresignedUrlRequest.addRequestParameter("uploadId", uploadIdentifier);
generatePresignedUrlRequest.addRequestParameter("partNumber", Integer.toString(partNumber))
generatePresignedUrlRequest.putCustomRequestHeader(Headers.CONTENT_MD5, md5Hash);
generatePresignedUrlRequest.putCustomRequestHeader(Headers.CONTENT_LENGTH, Long.toString(contentLength));

S3 path configuration and SQS Extended Client Library

I want to save all messages that go in a particular SQS queue in the already created s3 bucket.
But I want to save those messages in certain directories for an easier search by date and time.
S3Client has software.amazon.awssdk.services.s3.model.PutObjectRequest
Where I can determine bucket, path where the object is saved and some headers
PutObjectRequest objectRequest =
PutObjectRequest.builder()
.bucket(bucketName)
.key(s3Path)
.metadata(keyAndMetadata.getMetadata())
.build();
After that s3Client.putObject(objectRequest, body) do the thing
Now, I want to configure s3 in a similar way using ExtendedClientConfiguration, but I can only see very simple input parameters
ExtendedClientConfiguration extendedClientConfiguration =
new ExtendedClientConfiguration()
.withPayloadSupportEnabled(s3Client, bucketName, false)
.withAlwaysThroughS3(true);
And after that, we create that extended Sqs client with no way to configure s3 more extensively
AmazonSQSExtendedClient amazonSQSExtendedClient = new AmazonSQSExtendedClient(sqsClient, extendedClientConfiguration);
I know that I could probably separately save all messages that go to SQS to s3, but I'd better configure all that on the client level. Does someone have any ideas?

I found out there's no way to configure s3 path on the client level. But back up to s3 wasn't created for that purpose, and saving to s3 probably should be handled differently. Deleting files from s3 as they disappear from SQS is the best option for using this library.

Use AWS Java SDK to read text file from S3

Im trying to read a text file from AWS S3 object store (and then send it via http to a client). I have AWS CLI command which copies the file locally, but how can I do that via the SDK? I want to read the contents as string and avoid saving as a file and then read it back.
In CLI, I create a profile with keys (one time only):
aws configure --profile cloudian
Which then asks for questions like AWS Access Key ID [None]: and such. And then I need to run this command to retrieve the file:
aws --profile=cloudian --endpoint-url=https://s3-abc.abcstore.abc.net s3 cp s3://abc-store/STORE1/abc2/ABC/test_08.txt test.txt

For Reading S3 Object using SDK :
String s3Key ="";
AmazonS3 s3Client = AmazonS3ClientBuilder.standard().withRegion(region).build();
s3Key = URLDecoder.decode("s3Key", "UTF-8");
String s3BuckerName="Your Bucket Name";
S3Object object = s3Client.getObject(new GetObjectRequest(s3BuckerName, s3Key));
S3ObjectInputStream inputStream = object.getObjectContent();
You can get the content with the above code.
And I dint get second part of your question, do you wanna send this data somewhere?

How to get an excel file from AWS S3 bucket into a MultipartFile in Java

I've been trying to extract an .xlsx file from a AWS bucket I created and store it as a multipartfile variable. I've tried many different approaches, but at best I get weird characters. I'm not finding much documentation on how to do this.
Thanks!

// you may need to initialize this differently to get the correct authorization
final AmazonS3Client s3Client = AmazonS3ClientBuilder.defaultClient();
final S3Object object = s3Client.getObject("myBucket", "fileToDownload.xlsx");
// with Java 7 NIO
final Path filePath = Paths.get("localFile.xlsx");
Files.copy(object.getObjectContent(), filePath);
final File localFile = filePath.toFile();
// or Apache Commons IO
final File localFile = new File("localFile.xlsx");
FileUtils.copyToFile(object.getObjectContent(), localFile);
I'm not 100% sure what you mean by "MultipartFile" - that's usually in the context of a file that's been sent to your HTTP web service via a multipart POST or PUT. The file you're getting from S3 is technically part of the response to an HTTP GET request, but the Amazon Java Library abstracts this away for you, and just gives you the results as an InputStream.

Is is secure to transfer data between S3 and my data center using AWS S3 SDK?

I want to use AWS SDK to upload/download files from a private S3 bucket using Java to our data center.
I am planning to use following code
AmazonS3 s3Client = new AmazonS3Client(credentials);
s3Client.putObject(new PutObjectRequest(bucket, key, fileToUpload));
S3Object s3object = s3Client.getObject(new GetObjectRequest(bucket, key));
Will the data be traveling between S3 and our Datacenter in plain text or will it use some form of secure transport like SSL?

The default for communication is SSL. While HTTP is available, you would have to specifically use it. For example, from this page, you would have to do something like:
AmazonS3 s3Client = new AmazonS3Client(credentials);
s3Client.setEndpoint("http://s3-us-west-1.amazonaws.com");
with the endpoints taken from this page.
Since you're using the defaults, your communication is via SSL/HTTPS.

Categories