I am using Minio's JAVA SDK. I managed to copy objects within the same Minio Server. Is there a way to copy the objects from one Minio server to another?
I have tried using the below code:
InputStream inputStream = minioClientServer1.getObject(getBucket(), fileName);
minioClientServer2.putObject(getBucket(), fileName, inputStream, (long) inputStream.available(), null, null, contentType);
That is I got the object from one server and then uploaded to the next. The problem that I'm facing is that the contentType is unknown.
Is there a way to do this without hard coding the content type?
Or downloading the object to a file then uploading is a better way?
I'm not sure that this applies to you because you don't provide enough information for me to be sure we're talking about the same SDK. But from what you show, you should be able to call statObject in the same way that you call getObject and get an ObjectStat instance rather than an InputStream instance for a particular S3 object. Once you have an ObjectStat instance, you should be able to call the contentType method on it to get the content type of the S3 object.
This should work to do what you're asking:
ObjectStat objectStat = minioClientServer1.statObject(getBucket(), fileName);
InputStream inputStream = minioClientServer1.getObject(getBucket(), fileName);
minioClientServer2.putObject(getBucket(), fileName, inputStream, (long) inputStream.available(), null, null, objectStat.contentType());
Related
I have a web page with an upload feature which lets you upload a excel file, on hitting upload an Ajax call is fired. From there I get the FileItem input stream and using the method fileItem.getInputStream(), I have another class with a method which I need to pass the file to, which has a FileInputStream parameter. So my question is how do I convert the input stream to a FileInputStream?
A detailed solution would be appreciated as I am a junior developer, so I am still learning.
Many thanks.
From JavaDoc
A FileInputStream obtains input bytes from a file in a file system.
I would suggest two solutions:
The proper one is to change the API and to have InputStream as a parameter. I don't see a reason why you have FileInputStream in your API.
If you don't own the API and cannot change it I'm afraid you will need to save the InputStream to temp file and then create FileInputStream giving a path to this file (it's a suboptimal solution as you first write the file to disk - risking out of space - and then read it and streaming API is designed for reading / writing data on the fly)
If you are using org.apache.commons.fileupload.FileItem interface then your class is probably DefaultFileItem which is a subclass of DiskFileItem. So you can cast FileItem to DiskFileItem. then if you look at the source code of DiskFileItem you'll find that getInputStream() is actually returning a FileInputStream or a ByteArrayInputStream If you get a FileInputStream from DiskFileItem you can pass it directly to your other class. But if you get a ByteArrayInputStream you will have to write the contents to your own temporary file and then open another FileInputStream on this temp file. There is also another method DiskFileItem.getStoreLocation() which seem to return the server side File used for upload, but it may return null if the file is cached in memory.
In conclusion: you cannot be sure that there is going to be a server side file because the upload may be cached in memory. Therefore if you need a FileInputStream elsewhere you will have to create it yourself by creating a temp file. There is an example on how to pipe between two streams here.
//Pass file path/name directly to FileInputStream
FileInputStream input1 = new FileInputStream("input.txt");
//Save file path that has been passed in by the user, into a string variable.
String fileName = args[0];
//pass path to File object
File inputFile = new File(fileName);
//pass file object to FileOutputStream
FileOutputStream output = new FileOutputStream(inputFile);
I'm implementing an helper class to handle transfers from and to an AWS S3 storage from my web application.
In a first version of my class I was using directly a AmazonS3Client to handle upload and download, but now I discovered TransferManager and I'd like to refactor my code to use this.
The problem is that in my download method I return the stored file in form of byte[]. TransferManager instead has only methods that use File as download destination (for example download(GetObjectRequest getObjectRequest, File file)).
My previous code was like this:
GetObjectRequest getObjectRequest = new GetObjectRequest(bucket, key);
S3Object s3Object = amazonS3Client.getObject(getObjectRequest);
S3ObjectInputStream objectInputStream = s3Object.getObjectContent();
byte[] bytes = IOUtils.toByteArray(objectInputStream);
Is there a way to use TransferManager the same way or should I simply continue using an AmazonS3Client instance?
The TransferManager uses File objects to support things like file locking when downloading pieces in parallel. It's not possible to use an OutputStream directly. If your requirements are simple, like downloading small files from S3 one at a time, stick with getObject.
Otherwise, you can create a temporary file with File.createTempFile and read the contents into a byte array when the download is done.
I am a total newbie to amazon and java trying two things:
I am trying to create a folder in my Amazon S3 bucket that i have already created and have got the credentials for.
I am trying to upload a file to this bucket.
As per my understanding i can use putObjectRequest() method for acheiving both of my tasks.
PutObjectRequest(bucketName, keyName, file)
for uploading a file.
I am not sure if i should use this method
PutObjectRequest(String bucketName, String key, InputStream input,
ObjectMetadata metadata)
for just creating a folder. I am struggling with InputSteam and ObjectMetadata. I don't know what exactly is this for and how I can use it.
You do not need to create a folder in Amazon S3. In fact, folders do not exist!
Rather, the Key (filename) contains the full path and the object name.
For example, if a file called cat.jpg is in the animals folder, then the Key (filename) is: animals/cat.jpg
Simply Put an object with that Key and the folder is automatically created. (Actually, this isn't true because there are no folders, but it's a nice simple way to imagine the concept.)
As to which function to use... always use the simplest one that meets your needs. Therefore, just use PutObjectRequest(bucketName, keyName, file).
Yes, you can use PutObjectRequest(bucketName, keyName, file) to achive both task.
1, create S3 folder
With AWS S3 Java SDK , just add "/" at the end of the key name, it will create empty folder.
var folderKey = key + "/"; //end the key name with "/"
Sample code:
final InputStream im = new InputStream() {
#Override
public int read() throws IOException {
return -1;
}
};
final ObjectMetadata om = new ObjectMetadata();
om.setContentLength(0L);
PutObjectRequest putObjectRequest = new PutObjectRequest(bucketName, objectName, im, om);
s3.putObject(putObjectRequest);
2, Uploading file
Just similar, you can get input stream from your local file.
Alternatively you can use [minio client] java library
You can follow MakeBucket.java example to create a bucket & PutObject.java example to add an object.
Hope it help.
I'm trying to generate a PDF document from an uploaded ".docx" file using JODConverter.
The call to the method that generates the PDF is something like this :
File inputFile = new File("document.doc");
File outputFile = new File("document.pdf");
// connect to an OpenOffice.org instance running on port 8100
OpenOfficeConnection connection = new SocketOpenOfficeConnection(8100);
connection.connect();
// convert
DocumentConverter converter = new OpenOfficeDocumentConverter(connection);
converter.convert(inputFile, outputFile);
// close the connection
connection.disconnect();
I'm using apache commons FileUpload to handle uploading the docx file, from which I can get an InputStream object. I'm aware that Java.io.File is just an abstract reference to a file in the system.
I want to avoid the disk write (saving the InputStream to disk) and the disk read (reading the saved file in JODConverter).
Is there any way I can get a File object refering to an input stream? just any other way to avoid disk IO will also do!
EDIT: I don't care if this will end up using a lot of system memory. The application is going to be hosted on a LAN with very little to zero number of parallel users.
File-based conversions are faster than stream-based ones (provided by StreamOpenOfficeDocumentConverter) but they require the OpenOffice.org service to be running locally and have the correct permissions to the files.
Try the doc to avoid disk writting:
convert(java.io.InputStream inputStream, DocumentFormat inputFormat, java.io.OutputStream outputStream, DocumentFormat outputFormat)
There is no way to do it and make the code solid. For one, the .convert() method only takes two Files as arguments.
So, this would mean you'd have to extend File, which is possible in theory, but very fragile, as you are required to delve into the library code, which can change at any time and make your extended class non functional.
(well, there is a way to avoid disk writes if you use a RAM-backed filesystem and read/write from that filesystem, of course)
Chances are that commons fileupload has written the upload to the filesystem anyhow.
Check if your FileItem is an instance of DiskFileItem. If this is the case the write implementation of DiskFileItem willl try to move the file to the file object you pass. You are not causing any extra disk io then since the write already happened.
so i'm trying to clone objects in a folder on my S3 (Amazon S3) account. But i was wondering if there a way to do it without having to write the file to my local system first, then uploading that file back up to S3?
eventually i want it to be fully recursive cloning folders and objects in a given bucket, but for now i'm stuck on getting it to clone efficiently.
say the bucket path is images.example.com/products/prodSku
and in that prodSku folder i have a bunch of images i want to copy to a new folder
here's what i have so far.
(note: this is written in groovy, but if you know java, it's the same thing)
try{
def s3os = restService.listObjects(bucket_name, sourcePrefix, null)
def s3o
for(def i in s3os){
s3o = get(bucket_name, i.key)
// i want to be able to do something like this, just putting the input stream
// back into s3. but i can't. from what i know now, i have to write the
// dataInputStream into a file locally, then use that file to create a new S3Object
// which is placed as the second argument in the putObject method
restService.putObject(destinationBucketName, s3o.dataInputStream)
}
}catch(S3ServiceException e)
{
println e
}
Sorry the formatting is all messed up, first time posting a message.
but any help would be greatly appreciated!
Thanks!
Not sure about JetS3t API but, the AWS SDK for Java does provide a simple copyObject method
so i ended up figuring out how to do clone the asset in s3 using JetS3t. it was simpler than i expected. i'll post it up incase anyone ever googles this question.
all do is first get the s3 object you want to clone. after you have it, call setKey(filename) on the s3 object. "filename" is the path for where you want the object to be followed by the file name itself i.e. yours3bucketname/products/assets/picture.png
after your done with that, just call putObject(bucket_name, s3object), passing the s3object that you called setKey on as the second argument.
good luck! happy programming!