We Keep Coding

AWS S3 Using transferManager to make multipart upload by sending parts

I have recently learned about TransferManager class in AWS S3.
This behind the scenes creates multi-part upload, however it seems like I need to put whole file inside for it to work.
I receive my file in parts, so I need to create multipart upload manually. Is something like that possible using TransferManager? For example instead of
Upload upload = tm.upload(bucketName, keyName, new File(filePath));
to use for example something like
Upload upload = tm.upload(bucketName, keyName, partOfFile1);
Upload upload = tm.upload(bucketName, keyName, partOfFile2);
Upload upload = tm.upload(bucketName, keyName, partOfFile3);
Or am I stuck with AmazonS3 class when I need to upload file by parts manually?
Thanks for help!

Upload multipart file to AWS without saving it locally

I wrote a Rest API that accepts MultipartFile. I want to upload the files that come in to Amazon S3. The problem is that I don't know a way other than first saving it to the local system before uploading it to S3. Is there any way to do so?
Right now, there is an issue in saving the file locally and I'm looking for a work around: Multipart transferTo looks for a wrong file address when using createTempFile

Yes, you can do this.Use putObject which consume InputStream as param.
Here is sample code.
public void saveFile(MultipartFile multipartFile) throws AmazonServiceException, SdkClientException, IOException {
ObjectMetadata data = new ObjectMetadata();
data.setContentType(multipartFile.getContentType());
data.setContentLength(multipartFile.getSize());
BasicAWSCredentials creds = new BasicAWSCredentials("accessKey", "secretKey");
AmazonS3 s3client = AmazonS3ClientBuilder.standard().withRegion(Regions.US_EAST_2).withCredentials(new AWSStaticCredentialsProvider(creds)).build();
PutObjectResult objectResult = s3client.putObject("myBucket", multipartFile.getOriginalFilename(), multipartFile.getInputStream(), data);
System.out.println(objectResult.getContentMd5()); //you can verify MD5
}
You can find javadoc here

How to get an excel file from AWS S3 bucket into a MultipartFile in Java

I've been trying to extract an .xlsx file from a AWS bucket I created and store it as a multipartfile variable. I've tried many different approaches, but at best I get weird characters. I'm not finding much documentation on how to do this.
Thanks!

// you may need to initialize this differently to get the correct authorization
final AmazonS3Client s3Client = AmazonS3ClientBuilder.defaultClient();
final S3Object object = s3Client.getObject("myBucket", "fileToDownload.xlsx");
// with Java 7 NIO
final Path filePath = Paths.get("localFile.xlsx");
Files.copy(object.getObjectContent(), filePath);
final File localFile = filePath.toFile();
// or Apache Commons IO
final File localFile = new File("localFile.xlsx");
FileUtils.copyToFile(object.getObjectContent(), localFile);
I'm not 100% sure what you mean by "MultipartFile" - that's usually in the context of a file that's been sent to your HTTP web service via a multipart POST or PUT. The file you're getting from S3 is technically part of the response to an HTTP GET request, but the Amazon Java Library abstracts this away for you, and just gives you the results as an InputStream.

How to choose upload method in GCS Storage

I read that GCS Storage REST api supports 3 upload methods:
simple HTTP uploaded
chunked upload
resumed upload
I see that google-api-services-storage-v1 uses resumed upload approach,
but I am curious how to change this, because resume upload wastes
2 HTTP requests 1 for metadata and the second for data.
Request body of the first request is just {"name": "xxx"}.
InputStreamContent contentStream = new InputStreamContent(
APPLICATION_OCTET_STREAM, stream);
StorageObject objectMetadata = new StorageObject()
.setName(id.getValue());
Storage.Objects.Insert insertRequest = storage.objects().insert(
bucketName, objectMetadata, contentStream);
StorageObject object = insertRequest.execute();

I believe that particular library exclusively uses resumable uploads. Resumable uploads are very useful for large transfers, as they can recover from error and continue the upload. This is indeed sub-optimal in some cases, such as if you wanted to upload a very large number of very small objects one at a time.
If you want to do perform simpler uploads, you might want to consider another library, such as gcloud-java, which can perform direct uploads like so:
Storage storage = StorageOptions.defaultInstance().service();
Bucket bucket = storage.get(bucketName);
bucket.create(objectName, /*byte[] or InputStream*/, contentType);
That'll use only one request, although for larger uploads I recommend sticking with resumable uploads.

Put file to Amazon S3 using multipart upload

I'm trying to upload a file with the Amazon Java SDK, via multipart upload. The idea is to pass an upload-id to an applet, which puts the file parts into a readonly-bucket. Going this way, I avoid to store AWS credentials in the applet.
In my tests, I generate an upload-id with boto (python) and store a file into the bucket. That works well.
My Applet gets a "403 Access denied" from the S3, and I have no idea why.
Here's my code (which is partially taken from http://docs.amazonwebservices.com/AmazonS3/latest/dev/llJavaUploadFile.html):
AmazonS3 s3Client = new AmazonS3Client();
List<PartETag> partETags = new ArrayList<PartETag>();
long contentLength = file.length();
long partSize = Config.getInstance().getInt("part_size");
String bucketName = Config.getInstance().getString("bucket");
String keyName = "mykey";
String uploadId = getParameter("upload_id");
try {
long filePosition = 0;
for (int i = 1; filePosition < contentLength; i++) {
partSize = Math.min(partSize, (contentLength - filePosition));
// Create request to upload a part.
UploadPartRequest uploadRequest = new UploadPartRequest()
.withBucketName(bucket).withKey(keyName)
.withUploadId(uploadId).withPartNumber(i)
.withFileOffset(filePosition)
.withFile(file)
.withPartSize(partSize);
// Upload part and add response to our list.
partETags.add(s3Client.uploadPart(uploadRequest).getPartETag());
filePosition += partSize;
}
System.out.println("Completing upload");
CompleteMultipartUploadRequest compRequest = new
CompleteMultipartUploadRequest(bucket,
keyName,
uploadId,
partETags);
s3Client.completeMultipartUpload(compRequest);
} catch (Exception e) {
s3Client.abortMultipartUpload(new AbortMultipartUploadRequest(
bucketName, keyName, uploadId));
}
In the applet debug log, I find this, then:
INFO: Sending Request: PUT https://mybucket.s3.amazonaws.com /mykey Parameters: (uploadId: V4hwobOLQ1rYof54zRW0pfk2EfhN7B0fpMJTOpHOcmaUl8k_ejSo_znPI540.lpO.ZO.bGjh.3cx8a12ZMODfA--, partNumber: 1, ) Headers: (Content-Length: 4288546, Content-Type: application/x-www-form-urlencoded; charset=utf-8, )
24.01.2012 16:48:42 com.amazonaws.http.AmazonHttpClient handleErrorResponse
INFO: Received error response: Status Code: 403, AWS Service: null, AWS Request ID: DECF32CCFEE9EBF0, AWS Error Code: AccessDenied, AWS Error Message: Access Denied, S3 Extended Request ID: xtL1ixsGM2/vsxJ+cZRHpkPZ23SMfP8hZZjQCQnp8oWGwdS2/aGfYgomihyqaDCQ
Do you find any obvious failures in the code?
Thanks,
Stefan

While your use case is sound and this is an obvious attempt indeed, I don't think the Multipart Upload API has been designed to allow this and you are actually violating a security barrier:
The upload ID is merely an identifier to assist the Multipart Upload API in assembling the parts together (i.e. more like a temporary object key) not a dedicated security mechanism (see below). Consequently you still require proper access credentials in place, but since you are calling AmazonS3Client(), which Constructs a new Amazon S3 client that will make anonymous requests to Amazon S3, your request yields a 403 Access denied accordingly.
What you are trying to achieve is possible via Uploading Objects Using Pre-Signed URLs, albeit only without the multipart functionality, unfortunately:
A pre-signed URL gives you access to the object identified in the URL,
provided that the creator of the pre-signed URL has permissions to
access that object. That is, if you receive a pre-signed URL to upload
an object, you can upload the object only if the creator of the
pre-signed URL has the necessary permissions to upload that object.
[...] The pre-signed URLs
are useful if you want your user/customer to be able upload a specific
object [...], but you don't require them to have AWS security
credentials or permissions. When you create a pre-signed URL, you must
provide your security credentials, specify a bucket name an object
key, an HTTP method (PUT of uploading objects) and an expiration date
and time. [...]
The lenghty quote illustrates, why a system like this likely needs a more complex security design than 'just' handing out an upload ID (as similar as both might appear at first sight).
Obviously one would like to be able to use both features together, but this doesn't appear to be available yet.