I have recently learned about TransferManager class in AWS S3.
This behind the scenes creates multi-part upload, however it seems like I need to put whole file inside for it to work.
I receive my file in parts, so I need to create multipart upload manually. Is something like that possible using TransferManager? For example instead of
Upload upload = tm.upload(bucketName, keyName, new File(filePath));
to use for example something like
Upload upload = tm.upload(bucketName, keyName, partOfFile1);
Upload upload = tm.upload(bucketName, keyName, partOfFile2);
Upload upload = tm.upload(bucketName, keyName, partOfFile3);
Or am I stuck with AmazonS3 class when I need to upload file by parts manually?
Thanks for help!
Related
I want to generate a presign url(with upload rights only) and share it with my second application, so the second application can upload multipart file to it. How can I achieve this with java + spring boot ?
P.S. all the solutions I googled work only for single upload but not for multipart.
You can use GeneratePresignedUrlRequest, try this ;
GeneratePresignedUrlRequest generatePresignedUrlRequest = new GeneratePresignedUrlRequest(S3_BUCKET_NAME, key);
generatePresignedUrlRequest.addRequestParameter("uploadId", uploadIdentifier);
generatePresignedUrlRequest.addRequestParameter("partNumber", Integer.toString(partNumber))
generatePresignedUrlRequest.putCustomRequestHeader(Headers.CONTENT_MD5, md5Hash);
generatePresignedUrlRequest.putCustomRequestHeader(Headers.CONTENT_LENGTH, Long.toString(contentLength));
I wrote a Rest API that accepts MultipartFile. I want to upload the files that come in to Amazon S3. The problem is that I don't know a way other than first saving it to the local system before uploading it to S3. Is there any way to do so?
Right now, there is an issue in saving the file locally and I'm looking for a work around: Multipart transferTo looks for a wrong file address when using createTempFile
Yes, you can do this.Use putObject which consume InputStream as param.
Here is sample code.
public void saveFile(MultipartFile multipartFile) throws AmazonServiceException, SdkClientException, IOException {
ObjectMetadata data = new ObjectMetadata();
data.setContentType(multipartFile.getContentType());
data.setContentLength(multipartFile.getSize());
BasicAWSCredentials creds = new BasicAWSCredentials("accessKey", "secretKey");
AmazonS3 s3client = AmazonS3ClientBuilder.standard().withRegion(Regions.US_EAST_2).withCredentials(new AWSStaticCredentialsProvider(creds)).build();
PutObjectResult objectResult = s3client.putObject("myBucket", multipartFile.getOriginalFilename(), multipartFile.getInputStream(), data);
System.out.println(objectResult.getContentMd5()); //you can verify MD5
}
You can find javadoc here
I've been trying to extract an .xlsx file from a AWS bucket I created and store it as a multipartfile variable. I've tried many different approaches, but at best I get weird characters. I'm not finding much documentation on how to do this.
Thanks!
// you may need to initialize this differently to get the correct authorization
final AmazonS3Client s3Client = AmazonS3ClientBuilder.defaultClient();
final S3Object object = s3Client.getObject("myBucket", "fileToDownload.xlsx");
// with Java 7 NIO
final Path filePath = Paths.get("localFile.xlsx");
Files.copy(object.getObjectContent(), filePath);
final File localFile = filePath.toFile();
// or Apache Commons IO
final File localFile = new File("localFile.xlsx");
FileUtils.copyToFile(object.getObjectContent(), localFile);
I'm not 100% sure what you mean by "MultipartFile" - that's usually in the context of a file that's been sent to your HTTP web service via a multipart POST or PUT. The file you're getting from S3 is technically part of the response to an HTTP GET request, but the Amazon Java Library abstracts this away for you, and just gives you the results as an InputStream.
I read that GCS Storage REST api supports 3 upload methods:
simple HTTP uploaded
chunked upload
resumed upload
I see that google-api-services-storage-v1 uses resumed upload approach,
but I am curious how to change this, because resume upload wastes
2 HTTP requests 1 for metadata and the second for data.
Request body of the first request is just {"name": "xxx"}.
InputStreamContent contentStream = new InputStreamContent(
APPLICATION_OCTET_STREAM, stream);
StorageObject objectMetadata = new StorageObject()
.setName(id.getValue());
Storage.Objects.Insert insertRequest = storage.objects().insert(
bucketName, objectMetadata, contentStream);
StorageObject object = insertRequest.execute();
I believe that particular library exclusively uses resumable uploads. Resumable uploads are very useful for large transfers, as they can recover from error and continue the upload. This is indeed sub-optimal in some cases, such as if you wanted to upload a very large number of very small objects one at a time.
If you want to do perform simpler uploads, you might want to consider another library, such as gcloud-java, which can perform direct uploads like so:
Storage storage = StorageOptions.defaultInstance().service();
Bucket bucket = storage.get(bucketName);
bucket.create(objectName, /*byte[] or InputStream*/, contentType);
That'll use only one request, although for larger uploads I recommend sticking with resumable uploads.
I am looking to upload a PDF using the resumable upload mechanism. However, the web server is throwing a 403 exception which states: "Files must uploaded using the resumable upload mechanism."
This is particularly frustrating, since, the resumable upload mechanism is what I'm using. I am able to change the file to a .txt and it works efficiently.
String contentType = DocumentListEntry.MediaType.fromFileName(file3.getName()).getMimeType();
System.out.println("This is break zero.");
MediaFileSource mediaFile = new MediaFileSource(file3, contentType);
System.out.println("This is break one.");
ResumableGDataFileUploader uploader =
new ResumableGDataFileUploader.Builder(
client, new URL("https://docs.google.com/feeds/default/private/full"), mediaFile, null /*empty meatadata*/)
.title(mediaFile.getName())
.chunkSize(DEFAULT_CHUNK_SIZE).executor(executor)
.trackProgress(listener, PROGRESS_UPDATE_INTERVAL)
.build();
You should send the request to the resumable upload url, which is https://docs.google.com/feeds/upload/create-session/default/private/full.
Check the resumable upload documentation for more details: https://developers.google.com/gdata/docs/resumable_upload