I'm trying to copy files from folder to folder in the same bucket in s3 storage.
I need to copy files that are greater than 5GB and I saw in the docs that regular copy doesn't support this kind of copy see here
This link in the docs shows how to do it, but this code is for version 1.x and not 2.x.
I searched the new docs but I found only this and there is no code that shows how to multipart copy only regular copy.
It should be noted that another user asked about this but with no replies.
This code will help you to copy object with multipart in Java S3 SDK v2.
private final S3Client s3Client = S3Client.builder().build();
public void copyObjectWithMultiPart() {
String destBucketName = "destination-bucket";
String destObjectKey = "destination-object-key";
String sourceBucketName = "source-bucket";
String sourceObjectKey = "source-object-key";
// Initiate the multipart upload.
CreateMultipartUploadRequest createMultipartUploadRequest = CreateMultipartUploadRequest.builder()
.bucket(destBucketName)
.key(destObjectKey)
.build();
CreateMultipartUploadResponse multipartUploadResponse = s3Client.createMultipartUpload(createMultipartUploadRequest);
// Get the object size to track the end of the copy operation.
HeadObjectRequest headObjectRequest = HeadObjectRequest.builder()
.bucket(sourceBucketName)
.key(sourceObjectKey)
.build();
long objectSize = s3Client.headObject(headObjectRequest).contentLength();
// Copy the object using 5 MB parts.
long partSize = 5 * 1024 * 1024;
long bytePosition = 0;
int partNum = 1;
List<CompletedPart> etags = new ArrayList<>();
while (bytePosition < objectSize) {
// The last part might be smaller than partSize, so check to make sure
// that lastByte isn't beyond the end of the object.
long lastByte = Math.min(bytePosition + partSize - 1, objectSize - 1);
// Copy this part.
UploadPartCopyRequest uploadPartCopyRequest = UploadPartCopyRequest.builder()
.sourceBucket(sourceBucketName)
.sourceKey(sourceObjectKey)
.destinationBucket(destBucketName)
.destinationKey(destObjectKey)
.uploadId(multipartUploadResponse.uploadId())
.partNumber(partNum)
.copySourceRange(String.format("bytes=%d-%d", bytePosition, lastByte))
.build();
UploadPartCopyResponse uploadPartCopyResponse = s3Client.uploadPartCopy(uploadPartCopyRequest);
etags.add(
CompletedPart.builder()
.partNumber(partNum++)
.eTag(uploadPartCopyResponse.copyPartResult().eTag())
.build()
);
bytePosition += partSize;
}
// Complete the upload request to concatenate all uploaded parts and make the copied object available.
CompletedMultipartUpload completedMultipartUpload = CompletedMultipartUpload.builder()
.parts(etags)
.build();
CompleteMultipartUploadRequest completeMultipartUploadRequest =
CompleteMultipartUploadRequest.builder()
.bucket(destBucketName)
.key(destObjectKey)
.uploadId(multipartUploadResponse.uploadId())
.multipartUpload(completedMultipartUpload)
.build();
s3Client.completeMultipartUpload(completeMultipartUploadRequest);
}
Related
I am looking for some help/example to perform a resumeable upload to Google Drive using the new v3 REST API in Java.
I know there is a low level description here: Upload files | Google Drive API. But at the moment I am not willing to understand any of these low level requests, if there isn't another, simpler method ( like former MediaHttpUploader, which is deprecated now...)
What I currently do is:
File fileMetadata = new File();
fileMetadata.setName(name);
fileMetadata.setDescription(...);
fileMetadata.setParents(parents);
fileMetadata.setProperties(...);
FileContent mediaContent = new FileContent(..., file);
drive.files().create(fileMetadata, mediaContent).execute();
But for large files, this isn't good if the connection interrupts.
I've just created an implementation on that recently. It will create a new file on your DriveFolder and return its metadata when the task succeeds. While uploading, it will also update the listener with uploading info. I added comments to make it auto explanable:
public Task<File> createFile(java.io.File yourfile, MediaHttpUploaderProgressListener uploadListener) {
return Tasks.call(mExecutor, () -> {
//Generates an input stream with your file content to be uploaded
FileContent mediaContent = new FileContent("yourFileMimeType", yourfile);
//Creates an empty Drive file
File metadata = new File()
.setParents(parents)
.setMimeType(yourFileMimeType)
.setName(yourFileName);
//Builds up the upload request
Drive.Files.Create uploadFile = mDriveService.files().create(metadata, mediaContent);
//This will handle the resumable upload
MediaHttpUploader uploader = uploadBackup.getMediaHttpUploader();
//choose your chunk size and it will automatically divide parts
uploader.setChunkSize(MediaHttpUploader.MINIMUM_CHUNK_SIZE);
//according to Google, this enables gzip in future (optional)
uploader.setDisableGZipContent(false); versions
//important, this enables resumable upload
uploader.setDirectUploadEnabled(false);
//listener to be updated
uploader.setProgressListener(uploadListener);
return uploadFile.execute();
});
}
And make your Activity extends MediaHttpUploaderProgressListener so you have real time updates on the file progress:
#Override
public void progressChanged(MediaHttpUploader uploader) {
String sizeTemp = "Uploading"
+ ": "
+ Formatter.formatShortFileSize(this, uploader.getNumBytesUploaded())
+ "/"
+ Formatter.formatShortFileSize(this, totalFileSize);
runOnUiThread(() -> textView.setText(sizeTemp));
}
For calculating the progress percentage, you simply do:
double percentage = uploader.getNumBytesUploaded() / totalFileSize
Or use this one:
uploader.getProgress()
It gives you the percentage of bytes that have been uploaded, represented between 0.0 (0%) and 1.0 (100%). But be sure to have your content length specified, otherwise it will throw IllegalArgumentException.
We want to upload multiple files to Google Cloud Storage. Currently, we are uploading one by one using the Google Java API. The code is below:
public void uploadFile(File srcFile,String bucketName, String destPath) throws IOException {
BlobId blobId = BlobId.of(bucketName, srcFile.getName());
BlobInfo blobInfo = BlobInfo.newBuilder(blobId).build();
long startTime = System.currentTimeMillis();
// Blob blob = storage.create(blobInfo,new FileInputStream(srcFile));
try (WriteChannel writer = storage.writer(blobInfo)) {
try (FileInputStream in = new FileInputStream(srcFile)){
byte[] buffer = new byte[1024 * 1024 * 100] ;
writer.setChunkSize(buffer.length);
int readSize = 0;
while((readSize = in.read(buffer)) > 0) {
writer.write(ByteBuffer.wrap(buffer, 0, readSize));
}
long endTime = System.currentTimeMillis();
double writeTime = (double)(endTime - startTime) / 1000;
System.out.println("File write time : " + writeTime);
}
}
}
Our application wants to upload multiple files at a time. I tried to find a method to upload multiple files in the Java API, but could not find the any method to upload multiple files using a single call.
If I loop and upload using multiple calls, it is adding a huge network overhead and performance is very slow, which the application cannot afford.
My questions are:
Is it possible to upload multiple files via a single call, either using the Java API or REST API?
Could you please provide an example?
Neither the google-cloud-java library nor the underlying JSON API has an API call to upload multiple files, but you could accomplish what you're trying to do by spawning multiple threads and having each thread upload a file. The gsutil -m option does it this way.
I am successfully serving videos using the Play framework, but I'm experiencing an issue: each time a file is served, the Play framework creates a copy in C:\Users\user\AppData\Temp. I'm serving large files so this quickly creates a problem with disk space.
Is there any way to serve a file in Play without creating a copy? Or have Play automatically delete the temp file?
Code I'm using to serve is essentially:
public Result video() {
return ok(new File("whatever"));
}
Use Streaming
I use following method for video streaming. This code does not create temp copies of the media file.
Basically this code responds to the RANGE queries sent by the browser. If browser does not support RANGE queries I fallback to the method where I try to send the whole file using Ok.sendFile (internally play also tries to stream the file) (this might create temp files). but this happens very rarely when range queries is not supported by the browser.
GET /media controllers.MediaController.media
Put this code inside a Controller called MediaController
def media = Action { req =>
val file = new File("/Users/something/Downloads/somefile.mp4")
val rangeHeaderOpt = req.headers.get(RANGE)
rangeHeaderOpt.map { range =>
val strs = range.substring("bytes=".length).split("-")
if (strs.length == 1) {
val start = strs.head.toLong
val length = file.length() - 1L
partialContentHelper(file, start, length)
} else {
val start = strs.head.toLong
val length = strs.tail.head.toLong
partialContentHelper(file, start, length)
}
}.getOrElse {
Ok.sendFile(file)
}
}
def partialContentHelper(file: File, start: Long, length: Long) = {
val fis = new FileInputStream(file)
fis.skip(start)
val byteStringEnumerator = Enumerator.fromStream(fis).&>(Enumeratee.map(ByteString.fromArray(_)))
val mediaSource = Source.fromPublisher(Streams.enumeratorToPublisher(byteStringEnumerator))
PartialContent.sendEntity(HttpEntity.Streamed(mediaSource, None, None)).withHeaders(
CONTENT_TYPE -> MimeTypes.forExtension("mp4").get,
CONTENT_LENGTH -> ((length - start) + 1).toString,
CONTENT_RANGE -> s"bytes $start-$length/${file.length()}",
ACCEPT_RANGES -> "bytes",
CONNECTION -> "keep-alive"
)
}
I am trying to upload files to Amazon S3, nothing special. I have managed to do the actual upload, and the file uploads successfully. The only issue that's left is that I cannot change the name of the file in S3. It seems that by default, the name of the file is being set the same as the secret key. It could be that I am sending the secret key as a parameter where I should send the name of the file instead. However, I tried changing the parameters around and errors prop up.
Below please find the code I am using:
Bucket bucket = client.createBucket("testBucket", Region.EU_Ireland);
List<PartETag> partTags = new ArrayList<>();
InitiateMultipartUploadRequest request = new InitiateMultipartUploadRequest(
bucket.getName(), secretAmazonKey);
InitiateMultipartUploadResult result = client
.initiateMultipartUpload(request);
File file = new File(filePath);
long contentLength = file.length();
long partSize = 8 * 1024 * 1024;
try {
// Uploading the file, part by part.
long filePosition = 0;
for (int i = 1; filePosition < contentLength; i++) {
// Last part can be less than 8 MB therefore the partSize needs
// to be adjusted accordingly
partSize = Math.min(partSize, (contentLength - filePosition));
// Creating the request for a part upload
UploadPartRequest uploadRequest = new UploadPartRequest()
.withBucketName(bucket.getName()).withKey(secretAmazonKey)
.withUploadId(result.getUploadId()).withPartNumber(i)
.withFileOffset(filePosition).withFile(file)
.withPartSize(partSize);
// Upload part and add response to the result list.
partTags.add(client.uploadPart(uploadRequest).getPartETag());
filePosition += partSize;
}
}
catch (Exception e) {
client.abortMultipartUpload(new AbortMultipartUploadRequest(bucket
.getName(), secretAmazonKey, result.getUploadId()));
}
CompleteMultipartUploadRequest compRequest = new CompleteMultipartUploadRequest(
bucket.getName(), secretAmazonKey, result.getUploadId(), partTags);
client.completeMultipartUpload(compRequest);
Any help is much appreciated.
Thanks a lot :)
The key in your upload requests is actually your object (file) key (name), and not your AWS secret key. Whenever you instantiate your client instance, this is the time where you specify your AWS credentials.
Could you be more specific regarding the errors you are seeing when doing this?
Well, I used Amazon S3 for the first time recently and was able to upload a file as below:
public void saveMinutes(Minutes minutes, byte [] data)
{
AmazonS3 s3 = new AmazonS3Client(new BasicAWSCredentials(amazonS3AccessKey, amazonS3SecretAccessKey));
ObjectMetadata metaData = new ObjectMetadata();
metaData.setContentLength(data.length);
metaData.setContentType("application/pdf");
s3.putObject(new PutObjectRequest(amazonS3MinutesBucketName, minutes.getFileName(), new ByteArrayInputStream(data), metaData));
}
This question is unlikely to help any future visitors; it is only relevant to a small geographic area, a specific moment in time, or an extraordinarily narrow situation that is not generally applicable to the worldwide audience of the internet. For help making this question more broadly applicable, visit the help center.
Closed 10 years ago.
I am using third party server which return me following things.
1)url
2)acl
3)policy
4)awsAccesskeyID
5)Signature
6)key
I can upload file using following code
final File localFile = new File(localFilePath);
final Part[] parts = { new StringPart("acl", acl),
new StringPart("policy", policy),
new StringPart("AWSAccessKeyId", awsAccessKeyId),
new StringPart("signature", signature),
new StringPart("key", key, HTTP.UTF_8),
new FilePart("file", localFile) };
final MultipartRequestEntity mpRequestEntity = new MultipartRequestEntity(parts, filePost.getParams());
filePost.setRequestEntity(mpRequestEntity);
final HttpClient client = new HttpClient();
try
{
status = client.executeMethod(filePost);
}
But now I want to use AmazonS3Client using following code but its throwing exception that
10-31 16:21:36.070: INFO/com.amazonaws.request(13882): Received error
response: Status Code: 403, AWS Request ID: 51F7CB27E58F88FD, AWS
Error Code: SignatureDoesNotMatch, AWS Error Message: The request
signature we calculated does not match the signature you provided.
Check your key and signing method., S3 Extended Request ID:
YwNNsWOXg71vXY1VS0apHnHpHp4YVWRJ63xm8C7w36SYg1MNuIykw75YhQco5Lk7
final AmazonS3Client s3Client = new AmazonS3Client(new BasicAWSCredentials(awsAccessKeyId, key));
// Create a list of UploadPartResponse objects. You get one of these
// for each part upload.
final List<PartETag> partETags = new ArrayList<PartETag>();
// Step 1: Initialize.
final InitiateMultipartUploadRequest initRequest = new InitiateMultipartUploadRequest(targetURL, key);
final InitiateMultipartUploadResult initResponse = s3Client.initiateMultipartUpload(initRequest);
final File file = new File(localFilePath);
final long contentLength = file.length();
long partSize = 5242880; // Set part size to 5 MB.
try
{
// Step 2: Upload parts.
long filePosition = 0;
for ( int i = 1; filePosition < contentLength; i++ )
{
// Last part can be less than 5 MB. Adjust part size.
partSize = Math.min(partSize, (contentLength - filePosition));
// Create request to upload a part.
final UploadPartRequest uploadRequest = new UploadPartRequest().withBucketName(targetURL).withKey(key)
.withUploadId(initResponse.getUploadId()).withPartNumber(i).withFileOffset(filePosition)
.withFile(file).withPartSize(partSize);
// Upload part and add response to our list.
partETags.add(s3Client.uploadPart(uploadRequest).getPartETag());
filePosition += partSize;
}
// Step 3: complete.
final CompleteMultipartUploadRequest compRequest = new CompleteMultipartUploadRequest(targetURL, key,
initResponse.getUploadId(), partETags);
s3Client.completeMultipartUpload(compRequest);
}
catch ( final Exception e )
{
s3Client.abortMultipartUpload(new AbortMultipartUploadRequest(targetURL, key, initResponse.getUploadId()));
return false;
}
return true;
am I missing something here?
I found that server sending signature to upload file in one shot. In case of multipart upload multiple signature needed and that will needed at various steps..
There is no way to upload file in multiple part until server shares the key :(.
http://dextercoder.blogspot.in/2012/02/multipart-upload-to-amazon-s3-in-three.html