i am new to AWS, and my first module i try to learn is S3 for file storage.
Uploading works fine, the problem is with deleting.So when i upload a file i store the string version of the name of the file in AWS bucket mybucket and the whole URL in mysql database like this
-> https://mybucket.s3.eu-west-2.amazonaws.com/what.png
The problem with deleting is that even if i pass the whole URL in this case https://mybucket.s3.eu-west-2.amazonaws.com/what.png to the delete method, the method goes to each steps successfully, telling me that the file has been succesfully deleted but when i check the bucket, the file is still there.I have tried searching around here for a similar issue, but couldn't find something that could help me understand what the problem is.here is the code
#Service
public class AmazonS3ClientServiceImpl {
private String awsS3AudioBucket; //bucket name
private AmazonS3 amazonS3; // s3 object which uploads file
private static final Logger logger = LoggerFactory.getLogger(AmazonS3ClientServiceImpl.class);
#Autowired
public AmazonS3ClientServiceImpl(Region awsRegion, AWSCredentialsProvider awsCredentialsProvider, String awsS3AudioBucket) {
this.amazonS3 = AmazonS3ClientBuilder.standard()
.withCredentials(awsCredentialsProvider)
.withRegion(awsRegion.getName()).build();
this.awsS3AudioBucket = awsS3AudioBucket;
}
public String uploadFileToS3Bucket(MultipartFile multipartFile, boolean enablePublicReadAccess) {
String uploadedfile = ""; // the file path which is on s3
String fileName = multipartFile.getOriginalFilename();
try {
//creating the file in the server (temporarily)
File file = new File(fileName);
FileOutputStream fos = new FileOutputStream(file);
fos.write(multipartFile.getBytes());
fos.close();
PutObjectRequest putObjectRequest = new PutObjectRequest(this.awsS3AudioBucket, fileName, file);
if (enablePublicReadAccess) {
putObjectRequest.withCannedAcl(CannedAccessControlList.PublicRead);
}
this.amazonS3.putObject(putObjectRequest);
uploadedfile = String.valueOf(this.amazonS3.getUrl(awsS3AudioBucket, fileName));
System.out.println(this.amazonS3.getUrl(awsS3AudioBucket, fileName));
System.out.println(uploadedfile);
//removing the file created in the server
file.delete();
} catch (IOException | AmazonServiceException ex) {
logger.error("error [" + ex.getMessage() + "] occurred while uploading [" + fileName + "] ");
}
return uploadedfile;
}
public void deleteFileFromS3Bucket(String fileName) {
LOGGER.info("Deleting file with name= " + fileName);
final DeleteObjectRequest deleteObjectRequest = new DeleteObjectRequest(this.awsS3AudioBucket, fileName);
amazonS3.deleteObject(deleteObjectRequest);
LOGGER.info("File deleted successfully");
}
and when i call the deletemethod i use this
#GetMapping("/dashboard/showposts/delete/{id}")
public String deletePost(#PathVariable("id") Long id, Model model) {
System.out.println("GOT HERE");
//Retrieving Post image name
Post post = postService.findBydId(id);
String imageName = post.getImage();
System.out.println(imageName);
//Deleting image from S3 bucket
amazonClient.deleteFileFromS3Bucket(imageName);
//Deleting post from db
postService.detelePost(id);
String success = "Successfully deleted post with Id" + id;
model.addAttribute("success", success);
return "redirect:/admin/dashboard/showposts";
}
Any help would be greatly appreciated.
L.E For anyone having the same issue and searching for a quick answer.You have to pass only the string image name to the delete method not the whole URL.
You aren't checking the response returned from amazonS3.deleteObject() to see if it was actually successful or not. It is probably returning a failure status.
I'm guessing the root issue is that you are passing the full URL to the delete method, instead of just the path to the file within S3. For example with this URL: https://mybucket.s3.eu-west-2.amazonaws.com/what.png the S3 object path is simply what.png.
The simplest answer is to use the URL class. Something like:
URL url = null;
try {
url = new URL("https://mybucket.s3.eu-west-2.amazonaws.com/some/path/what.png");
} catch (MalformedURLException e) {
e.printStackTrace();
}
System.out.println( "file is \""+ url.getFile() + "\"" );
output would be "/some/path/what.png". You can remove the first "/" character to use for the key.
Aws S3 is eventual consistent. You might delete object and s3 list that object in browser . So it take few seconds or less to delete .
Please refer this link
Related
I am currently storing and downloading my Thymeleaf templates in S3.
I am using the following function to retrieve the Template from S3:
public String getTemplateFile(String name, File localFile) {
ObjectMetadata object = s3Client.getObject(new GetObjectRequest(connectionProperties.getBucket(), name), localFile);
boolean success = localFile.exists() && localFile.canRead();
return localFile.getPath();
}
After doing this the file is successfully downloaded in the desired location.
But when trying to access the file from the FlyingSaucer PDF generator the file doesn't exist, despite it is already downloaded in FILE_LOCATION_PATH. (I can open the file... the file is there but the function doesn't see it)
String xHtmlStringDocument =
convertHtmlToXhtml(templateEngine
.process(FILE_LOCATION_PATH,
initializeLetterHtmlTemplateContext(letter)));
When I run the program again and again I get the same result. But when I STOP the program and RUN it AGAIN then everything works because the file form the last execution is now recognized by the program.
This sounds to me like an asynchronous function issue.
Does anybody know how can I fix this?
Thanks in advance.
EDITED (following suggestion)
New function: Same result:
(And the file was created, the Download from S3 was successful)
java.io.FileNotFoundException: ClassLoader resource "static/templates/template.html" could not be resolved
public String getTemplateFileN(String name, File localFile) throws IOException {
S3Object fullObject = null;
InputStream in = null;
try {
fullObject = s3Client.getObject(new GetObjectRequest(connectionProperties.getBucket(), name));
System.out.println("Content-Type: " + fullObject.getObjectMetadata().getContentType());
System.out.println("Content: ");
displayTextInputStream(fullObject.getObjectContent());
in = fullObject.getObjectContent();
System.out.println(localFile.toPath());
Files.copy(in, localFile.toPath());
} //then later
finally {
// To ensure that the network connection doesn't remain open, close any open input streams.
if (fullObject != null) {
fullObject.close();
}
if (in != null) {
in.close();
}
}
return localFile.getPath();
}
Checking javadoc
https://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/services/s3/AmazonS3Client.html#getObject-com.amazonaws.services.s3.model.GetObjectRequest-java.io.File-
I see not method signature ObjectMetadata getObject(GetObjectRequest getObjectRequest,String file)
There is
ObjectMetadata getObject(GetObjectRequest getObjectRequest,
File destinationFile)
Where you provide File (not String) as second argument. Make sure the file is not opened for write before you try reading it!
Our requirement is to download files from FTP server which we used a download files in sequential manner one after another which is taking huge time.
Sequential code:
for(String mediaValue: generatedMediaTypes) {
// Logic here is getting the user selected files from Ui and keep it in string array str
try {
ddownloadMultipleNavDbs(str[i]),uniqueID);
//now we download all the files here with downloadfile logic with help of unique id we are a folder by that name and keep all this files in that folder
} catch (Exception ex) {
logger.error(ex.getMessage());
}
}
After downloading all files into uniqueID folder now I zip and send to client by downloadZip(request,uniqueID):
public void downloadZipFile (HttpServletResponse response, String suuid) throws IOException
{
try
{
ZipOutputStream zipOutputStream = new ZipOutputStream(response.getOutputStream());
response.setContentType("application/octet-stream");
File loadableFolder = new File(uniqueID);
String timeStamp = new java.sql.Timestamp(Calendar.getInstance().getTime().getTime()).toString().replaceAll(
"-", "_").replaceAll(" ", "_").replaceAll(":", "_");
response.setHeader(
"Content-Disposition",
"attachment; filename=" + "File" + timeStamp.substring(
0, timeStamp.indexOf(".")) + ".ZIP");
File[] loadableMedias = //here we get the files from that uinqueID folder
ZipUtil.addFilesToZipStream(Medias, Folder,
zipOutputStream);
zipOutputStream.close();
response.flushBuffer();
response.getOutputStream().flush();
response.getOutputStream().close();
}
It works perfectly fine in above code.
When I used executorservice for downloading files in parallel
executorService.submit(new myThread(str[i]),uniqueID);
am facing error:
getOutputStream() has already been closed
Can anyone please explain me why am facing this error?
Why we face this error and how to resolve it?
I am working on a service project, who's UI allows its users to upload a file. I need to write a service that can upload this file to server and read and display the contents of this file. Can anyone tell me how to do this ?
//Controller definition begins
#RequestMapping(value = "/uploadFile", method = RequestMethod.POST)
public #ResponseBody
String uploadFileHandler(#RequestParam("name") String name,
#RequestParam("file") MultipartFile file) {
// Create a new file upload handler
ServletFileUpload upload = new ServletFileUpload();
if (!file.isEmpty()) {
try {
byte[] bytes = file.getBytes();
// Creating the directory to store file
String rootPath = System.getProperty("catalina.home");
File dir = new File(rootPath + File.separator + "tmpFiles");
if (!dir.exists())
dir.mkdirs();
// Create the file on server
File serverFile = new File(dir.getAbsolutePath()
+ File.separator + name);
BufferedOutputStream stream = new BufferedOutputStream(
new FileOutputStream(serverFile));
stream.write(bytes);
stream.close();
logger.info("Server File Location="
+ serverFile.getAbsolutePath());
return "You successfully uploaded file=" + name;
} catch (Exception e) {
return "You failed to upload " + name + " => " + e.getMessage();
}
} else {
return "You failed to upload " + name
+ " because the file was empty.";
}
}
Now i want to know how i can display the contents of the uploaded file. It is being converted to FILE format and is stored in the tempfiles directory of the tomcat server and the data is non-human readable.I need to convert this back to xlsx(file being uploaded is xlsx) or be able to read the data from it directly to update the Db.Also i am using apache commons-io and file upload in Spring MVC as you can see from above code.
You need a library capable of reading xlsx file types, such as Apache POI
In case you choose this library, there is a very good example section in the example secion of it's website
About Google Drive, in this Document, Google says there are 3 types of uploadType:
We have the media, multipart and resumable, as it says in the image above.
Also, in the same Document mentioned, Google give a example on Java, explaining how to upload an File into Google Drive.
public class MyClass {
private static File insertFile(Drive service, String title, String description,
String parentId, String mimeType, String filename) {
// File's metadata.
File body = new File();
body.setTitle(title);
body.setDescription(description);
body.setMimeType(mimeType);
// Set the parent folder.
if (parentId != null && parentId.length() > 0) {
body.setParents(
Arrays.asList(new ParentReference().setId(parentId)));
}
// File's content.
java.io.File fileContent = new java.io.File(filename);
FileContent mediaContent = new FileContent(mimeType, fileContent);
try {
File file = service.files().insert(body, mediaContent).execute();
// Uncomment the following line to print the File ID.
// System.out.println("File ID: " + file.getId());
return file;
} catch (IOException e) {
System.out.println("An error occured: " + e);
return null;
}
}
// ...
}
I want to set an Optional Query Parameter, as explained in the image. I can't find how to set the ocr and ocrLanguage in the java SDK of Google Drive. The example of upload above, don't have this parameter to set, also, is not clear what uploadType this java example uses to Upload.
You can use generic set property method
File body = new File();
body.setTitle(title);
body.set("ocr",true);
body.set("ocrLanguage", "zh");
How to store a file in Amazon S3 using jsp page ????
I used FileInputstream but it doesn't work. Please give any suggestion about jsp code.
try
{
String bucketName = "phr12345";
File file=new File("a.txt");
String key = "ads/" + file.getName();
s3.putObject(bucketName, key, file);
out.println("file uploaded !!!!!!");
}
catch(IOException e)
{
e.printStackTrace();
}