We Keep Coding

Java heap out of memory issue while zipping list of nio paths

I am asynchronously running the below method to zip the given set of nio paths. When there are multiple tasks running, java heap out of memory exception is encountered.
public InputStream compressToZip(String s3folderName, Set<Path> paths) throws Exception {
try {
ByteArrayOutputStream byteArrayOutputStream = new ByteArrayOutputStream();
ZipOutputStream zos = new ZipOutputStream(byteArrayOutputStream);
paths.forEach(path -> {
try {
System.out.println("Zipping " + path.getFileName());
zos.putNextEntry(new ZipEntry(path.getFileName().toString()));
FileInputStream ObjectInputStream = new FileInputStream(path.toFile());
IOUtils.copy(ObjectInputStream, zos);
zos.closeEntry();
} catch (Exception e) {
...
}
});
zos.close();
return new ByteArrayInputStream(byteArrayOutputStream.toByteArray());
} catch (Exception e) {
...
}
}
The input stream returned from this file will be written on sftp location.
org.springframework.integration.file.remote.session.Session session = this.createSession(deliveryType, deliveryLocation);
zippedIpStream = fileCompressionSvc.compressToZip(s3folderName, fileDir);
session.write(zippedIpStream, deliveryLocation.getLocation().getFolder() + "/"
+ getfileNameFormat(fileNameFormat, masterId) + ".zip");
I am not sure what went wrong to occur java heap issue. Could you please assist me.

Changed the implementation to write the file into a file in local path and then send that file to sftp and then delete the temp zip file.
public void compressToZip(String s3folderName, Set<Path> distinctPaths, String efsPathWithFileName) throws Exception {
try(FileOutputStream fos = new FileOutputStream(efsPathWithFileName);
ZipOutputStream zos = new ZipOutputStream(fos)) {
distinctPaths.forEach(path -> {
try {
zos.putNextEntry(new ZipEntry(path.getFileName().toString()));
final FileInputStream fis = new FileInputStream(path.toFile());
IOUtils.copy(fis, zos);
zos.closeEntry();
} catch (IOException e) {
...
}
});
} catch (Exception e) {
...
throw e;
}
}
calling method:
InputStream zippedIpStream = new FileInputStream(tempCompressedFilePath);
session.write(zippedIpStream, deliveryLocation.getLocation().getFolder() + "/" + fileNameToWrite);
...
...
zippedIpStream.close();
...
...
Files.deleteIfExists(Paths.get(tempCompressedFilePath));

Improper Resource Shutdown or Release

Description:
In the code below What am i doing wrong in the code below, in my code audit i got improper resource shutdown or release.
I tried taking out the close and flush from the code below
File someFile = new File(fileName);
fos = new FileOutputStream(someFile);
fos.write(data);
fos.flush();
fos.close();
Main code:
FileOutputStream fos=null;
try {
Hashtable hash = responseBlob.getAllAttachments();
Enumeration e = hash.elements();
while (e.hasMoreElements()) {
SBADataAttach tmpAttach = (SBADataAttach) e.nextElement();
String tag = tmpAttach.getTag();
byte[] data = tmpAttach.getData();
// encode compressed file
if (hasTag(tag))
mimeResponse.addPart(tag, Base64.encodeBytes(data)
.getBytes());
else
mimeResponse.addPart(tag, data);
try {
if (this.configData.getBlobPath() != null)
{
// save compressed file
String fileName = File.separatorChar + "tmp"
+ File.separatorChar + tag;
Log.theLogger.debug("XisServlet.process() ... "
+ "Save compressed file = " + fileName);
File someFile = new File(fileName);
fos = new FileOutputStream(
someFile);
fos.write(data);
fos.flush();
fos.close();
}
} catch (Exception zipe) {
Log.theLogger.error(zipe.getMessage(), zipe);
}
}
} catch (SBADataException sde) {
// cannot detach files from blob
Log.theLogger.error(sde.getMessage(), sde);
}
finally {
try {
if( fos!=null ) {
fos.close();
}
} catch(IOException e) {
Log.theLogger.error(e.getMessage(), e);
}
}
I don't get any error. But in the Appsec finding i getImproper Resource Shutdown or Release

try{
//open resources
File someFile = new File(fileName);
fos = new FileOutputStream(someFile);
fos.write(data);
}
catch(Exception e1){
//handle exception
}finally{
//close resources
fos.flush();
fos.close();
}
}

Create text file and add to zip file and download spring boot with out savind in local server

create and download zip file by adding list of text files. with out creating the file in local server, it should be download at client side direct,
Here i added a code snippet, it was creating in local server, but i dont want that, it should create and download at client side instant. Please help me in this way..
#GetMapping("/download/rawdata")
public void downloadRawdata(#RequestParam("date") String date){
log.info("date : "+date);
List<Rawdata> rawdatas = rawdataRepoisotry.findRawdataByDate(date);
log.info("size of rawdata : "+rawdatas.size());
List<File> files = new ArrayList<File>();
int i = 1;
for(Rawdata rawdata : rawdatas){
log.info("rawdata : "+ rawdata.getRawdata());
File file = new File(i+".txt");
try (Writer writer = new BufferedWriter(new FileWriter(file))) {
String contents = rawdata.getRawdata();
writer.write(contents);
files.add(file);
} catch (IOException e) {
e.printStackTrace();
}
i++;
}
try {
zipFile(files, new File(date+".zip"));
} catch (IOException e) {
e.printStackTrace();
throw new RuntimeException("Failed while creating Zip file");
}
}
public FileOutputStream zipFile(final List<File> files, final File targetZipFile) throws IOException {
try {
FileOutputStream fos = new FileOutputStream(targetZipFile);
ZipOutputStream zos = new ZipOutputStream(fos);
byte[] buffer = new byte[128];
for(File currentFile : files){
if (!currentFile.isDirectory()) {
ZipEntry entry = new ZipEntry(currentFile.getName());
FileInputStream fis = new FileInputStream(currentFile);
zos.putNextEntry(entry);
int read = 0;
while ((read = fis.read(buffer)) != -1) {
zos.write(buffer, 0, read);
}
zos.closeEntry();
fis.close();
}
}
zos.close();
fos.close();
return fos;
} catch (FileNotFoundException e) {
System.out.println("File not found : " + e);
throw new FileNotFoundException();
}
}

Here is an example using FileSystemResource.
What has been modified is (see the numbers in the commented code ) :
1) Declare that the response will be of type application/octet-stream
2) #ResponseBody
Annotation that indicates a method return value should be bound to the
web response body
3) Declare that the method returns a FileSystemResource body
4) Return the FileSystemResource entity based on your created zip file
Note that this will still create the file on the server side first, but you may want to use File.createTempFile and File.deleteOnExit.
#GetMapping("/download/rawdata", produces = MediaType.APPLICATION_OCTET_STREAM_VALUE)//1
#ResponseBody //2
public ResponseEntity<FileSystemResource> downloadRawdata(#RequestParam("date") String date){ //3
log.info("date : "+date);
List<Rawdata> rawdatas = rawdataRepoisotry.findRawdataByDate(date);
log.info("size of rawdata : "+rawdatas.size());
List<File> files = new ArrayList<File>();
int i = 1;
for(Rawdata rawdata : rawdatas){
log.info("rawdata : "+ rawdata.getRawdata());
File file = new File(i+".txt");
try (Writer writer = new BufferedWriter(new FileWriter(file))) {
String contents = rawdata.getRawdata();
writer.write(contents);
files.add(file);
} catch (IOException e) {
e.printStackTrace();
}
i++;
}
try {
File resultFile = new File(date+".zip");
zipFile(files, resultFile);
return new ResponseEntity<>(new FileSystemResource(resultFile), HttpStatus.OK); //4
} catch (IOException e) {
e.printStackTrace();
throw new RuntimeException("Failed while creating Zip file");
}
}

Unzipping File does not retrieve folders

I am trying to unzip some files from a zip file to an unzipped directory with the same file structure as the zipped file, but am having difficulty with the getNextEntry() method. It seems to be returning only the files in the zipped file and not the folders which is causing a FileNotFoundException when I try to create a file in a directory that doesn't exist.
For example the first level of my zip file is like the following:
Folder 1
file2.txt
Folder 2
Folder 3
file.txt
When I call getNextEntry() the first thing returned is file.txt, the second thing returned is Folder 1/file2.txt. Even the nested folders are being ignored. This was previously working, however, I'm not sure what I did to break it.
The file I'm passing in is a zipped file located in internal storage. Any help will be much appreciated!
public boolean unZipAndEncrypt(File file) {
boolean isSuccess = false;
ZipInputStream zin = null;
try {
ZipFile zipFile = new ZipFile(file);
FileInputStream fin = new FileInputStream(file);
zin = new ZipInputStream(fin);
ZipEntry ze;
File contentDir = new File(bookDirectory, contentId);
while ((ze = zin.getNextEntry()) != null) {
String name = ze.getName();
if (ze.isDirectory()) {
File dir = new File(contentDir, name);
dir.mkdirs();
continue;
}
FileModel fileModel = new FileModel(zipFile.getInputStream(ze), name);
if (!ze.getName().contains("cp_index")) {
fileModel = encryptor.encrypt(fileModel);
}
File toWrite = new File(contentDir, fileModel.getFullPathName());
toWrite.createNewFile();
OutputStream fout = new FileOutputStream(toWrite);
try {
byte[] buffer = new byte[1024];
int len = 0;
while ((len = fileModel.getInputStream().read(buffer)) != -1) {
fout.write(buffer, 0, len);
}
} finally {
fileModel.close();
zin.closeEntry();
fout.close();
}
}
isSuccess = true;
} catch (FileNotFoundException e) {
Log.e(TAG, "", e);
} catch (IOException e) {
Log.e(TAG, "", e);
} finally {
file.delete();
try {
zin.close();
} catch (IOException e) {
Log.e(TAG, "", e);
} catch (NullPointerException e) {
Log.e(TAG, "", e);
}
}
return isSuccess;
}

You could create the directory before creating the new file:
toWrite.getParentFile().mkdirs(); // do before createNewFile()

Unzipping with ZipInputStream and IoUtils.copyStream() freeze

I'm trying to unzip a File (30 Mb) programmatically, this is my script
public void unzip() {
dirChecker(location);
try {
FileInputStream fin = new FileInputStream(zipFile);
ZipInputStream zin = new ZipInputStream(fin);
ZipEntry ze = null;
while ((ze = zin.getNextEntry()) != null) {
Log.v("Decompress", "Unzipping " + location+ze.getName());
//create dir if required while unzipping
if (ze.isDirectory()) {
dirChecker(ze.getName());
} else {
FileOutputStream fout = new FileOutputStream(location + ze.getName());
IoUtils.copyStream(zin, fout);
zin.closeEntry();
fout.close();
}
}
zin.close();
} catch (Exception e) {
System.out.println(e);
}
}
private void dirChecker(String dir) {
File f = new File(location + dir);
if (!f.isDirectory()) {
f.mkdirs();
}
}
the unzipping goes well, until the 24th file in the zip, the application hangs whithout giving any error, simply freeze in the line:
IoUtils.copyStream(zin, fout);
no log, no exception trown, nothing... i discovered the right line putting a break point after the try and manually cicling until the 24th file arrive (it's an 1.1mb jpg, i had no trouble extracting with winzip)
any hint?