We Keep Coding

Java heap out of memory issue while zipping list of nio paths

I am asynchronously running the below method to zip the given set of nio paths. When there are multiple tasks running, java heap out of memory exception is encountered.
public InputStream compressToZip(String s3folderName, Set<Path> paths) throws Exception {
try {
ByteArrayOutputStream byteArrayOutputStream = new ByteArrayOutputStream();
ZipOutputStream zos = new ZipOutputStream(byteArrayOutputStream);
paths.forEach(path -> {
try {
System.out.println("Zipping " + path.getFileName());
zos.putNextEntry(new ZipEntry(path.getFileName().toString()));
FileInputStream ObjectInputStream = new FileInputStream(path.toFile());
IOUtils.copy(ObjectInputStream, zos);
zos.closeEntry();
} catch (Exception e) {
...
}
});
zos.close();
return new ByteArrayInputStream(byteArrayOutputStream.toByteArray());
} catch (Exception e) {
...
}
}
The input stream returned from this file will be written on sftp location.
org.springframework.integration.file.remote.session.Session session = this.createSession(deliveryType, deliveryLocation);
zippedIpStream = fileCompressionSvc.compressToZip(s3folderName, fileDir);
session.write(zippedIpStream, deliveryLocation.getLocation().getFolder() + "/"
+ getfileNameFormat(fileNameFormat, masterId) + ".zip");
I am not sure what went wrong to occur java heap issue. Could you please assist me.

Changed the implementation to write the file into a file in local path and then send that file to sftp and then delete the temp zip file.
public void compressToZip(String s3folderName, Set<Path> distinctPaths, String efsPathWithFileName) throws Exception {
try(FileOutputStream fos = new FileOutputStream(efsPathWithFileName);
ZipOutputStream zos = new ZipOutputStream(fos)) {
distinctPaths.forEach(path -> {
try {
zos.putNextEntry(new ZipEntry(path.getFileName().toString()));
final FileInputStream fis = new FileInputStream(path.toFile());
IOUtils.copy(fis, zos);
zos.closeEntry();
} catch (IOException e) {
...
}
});
} catch (Exception e) {
...
throw e;
}
}
calling method:
InputStream zippedIpStream = new FileInputStream(tempCompressedFilePath);
session.write(zippedIpStream, deliveryLocation.getLocation().getFolder() + "/" + fileNameToWrite);
...
...
zippedIpStream.close();
...
...
Files.deleteIfExists(Paths.get(tempCompressedFilePath));

Improper Resource Shutdown or Release

Description:
In the code below What am i doing wrong in the code below, in my code audit i got improper resource shutdown or release.
I tried taking out the close and flush from the code below
File someFile = new File(fileName);
fos = new FileOutputStream(someFile);
fos.write(data);
fos.flush();
fos.close();
Main code:
FileOutputStream fos=null;
try {
Hashtable hash = responseBlob.getAllAttachments();
Enumeration e = hash.elements();
while (e.hasMoreElements()) {
SBADataAttach tmpAttach = (SBADataAttach) e.nextElement();
String tag = tmpAttach.getTag();
byte[] data = tmpAttach.getData();
// encode compressed file
if (hasTag(tag))
mimeResponse.addPart(tag, Base64.encodeBytes(data)
.getBytes());
else
mimeResponse.addPart(tag, data);
try {
if (this.configData.getBlobPath() != null)
{
// save compressed file
String fileName = File.separatorChar + "tmp"
+ File.separatorChar + tag;
Log.theLogger.debug("XisServlet.process() ... "
+ "Save compressed file = " + fileName);
File someFile = new File(fileName);
fos = new FileOutputStream(
someFile);
fos.write(data);
fos.flush();
fos.close();
}
} catch (Exception zipe) {
Log.theLogger.error(zipe.getMessage(), zipe);
}
}
} catch (SBADataException sde) {
// cannot detach files from blob
Log.theLogger.error(sde.getMessage(), sde);
}
finally {
try {
if( fos!=null ) {
fos.close();
}
} catch(IOException e) {
Log.theLogger.error(e.getMessage(), e);
}
}
I don't get any error. But in the Appsec finding i getImproper Resource Shutdown or Release

try{
//open resources
File someFile = new File(fileName);
fos = new FileOutputStream(someFile);
fos.write(data);
}
catch(Exception e1){
//handle exception
}finally{
//close resources
fos.flush();
fos.close();
}
}

FileNotFoundException on FileOutputStream

My android application is very simply extract zip.
I want to create a folder with the FileName erase the extension (.zip) in the zip file.
And i'm succeeded.
However, the exception in some devices.
device name : KM-S300
os version : 2.3.4
source:
private void extractZip (File file) throws UTFDataFormatException {
FileInputStream fis = null;
FileOutputStream fos = null;
ZipInputStream zis = null;
ZipEntry ze = null;
byte[] data = new byte[1024];
int offset = 0;
String rootName = file.getAbsolutePath();
rootName = rootName.substring(0, rootName.lastIndexOf("."));
String rootFileName = rootName.substring(rootName.lastIndexOf("/") + 1);
File root = new File(rootName);
root.mkdirs();
try {
fis = new FileInputStream(file);
zis = new ZipInputStream(fis);
while (( ze = zis.getNextEntry() ) != null) {
try {
File f = new File(root, ze.getName());
if (!f.isDirectory()) {
f.getParentFile().mkdirs();
fos = new FileOutputStream(f); // <<-- ERROR
while (( offset = zis.read(data) ) != -1) {
fos.write(data, 0, offset);
}
}
} finally {
try {
fos.close();
} catch (Exception e) {}
}
}
file.delete();
}catch (UTFDataFormatException e){
throw e;
}catch (Exception e) {
e.printStackTrace();
} finally {
try {
zis.close();
} catch (Exception e) {}
try {
fis.close();
} catch (Exception e) {}
}
}
I can't understand Exception
java.io.FileNotFoundException: /mnt/sdcard/test/marker/Explosion/failed/0.png (No such file or directory)
at org.apache.harmony.luni.platform.OSFileSystem.open(Native Method)
at dalvik.system.BlockGuard$WrappedFileSystem.open(BlockGuard.java:232)
at java.io.FileOutputStream.<init>(FileOutputStream.java:94)
at java.io.FileOutputStream.<init>(FileOutputStream.java:66)
at com.sample.MainActivity.extractZip(MainActivity.java:507)
Made to the parent folder of the File prior to creating the FileOutputStream
Why did exception by application?

You're only creating the parent directory if the new file doesn't exist or isn't a directory. It's not a sensible test. You should create it if the parent directory doesn't exist.

Unzipping File does not retrieve folders

I am trying to unzip some files from a zip file to an unzipped directory with the same file structure as the zipped file, but am having difficulty with the getNextEntry() method. It seems to be returning only the files in the zipped file and not the folders which is causing a FileNotFoundException when I try to create a file in a directory that doesn't exist.
For example the first level of my zip file is like the following:
Folder 1
file2.txt
Folder 2
Folder 3
file.txt
When I call getNextEntry() the first thing returned is file.txt, the second thing returned is Folder 1/file2.txt. Even the nested folders are being ignored. This was previously working, however, I'm not sure what I did to break it.
The file I'm passing in is a zipped file located in internal storage. Any help will be much appreciated!
public boolean unZipAndEncrypt(File file) {
boolean isSuccess = false;
ZipInputStream zin = null;
try {
ZipFile zipFile = new ZipFile(file);
FileInputStream fin = new FileInputStream(file);
zin = new ZipInputStream(fin);
ZipEntry ze;
File contentDir = new File(bookDirectory, contentId);
while ((ze = zin.getNextEntry()) != null) {
String name = ze.getName();
if (ze.isDirectory()) {
File dir = new File(contentDir, name);
dir.mkdirs();
continue;
}
FileModel fileModel = new FileModel(zipFile.getInputStream(ze), name);
if (!ze.getName().contains("cp_index")) {
fileModel = encryptor.encrypt(fileModel);
}
File toWrite = new File(contentDir, fileModel.getFullPathName());
toWrite.createNewFile();
OutputStream fout = new FileOutputStream(toWrite);
try {
byte[] buffer = new byte[1024];
int len = 0;
while ((len = fileModel.getInputStream().read(buffer)) != -1) {
fout.write(buffer, 0, len);
}
} finally {
fileModel.close();
zin.closeEntry();
fout.close();
}
}
isSuccess = true;
} catch (FileNotFoundException e) {
Log.e(TAG, "", e);
} catch (IOException e) {
Log.e(TAG, "", e);
} finally {
file.delete();
try {
zin.close();
} catch (IOException e) {
Log.e(TAG, "", e);
} catch (NullPointerException e) {
Log.e(TAG, "", e);
}
}
return isSuccess;
}

You could create the directory before creating the new file:
toWrite.getParentFile().mkdirs(); // do before createNewFile()

What is the "Jodatime" / "Apache commons" of Zip/Unzip Java utilities?

I'm sure that there is a mature, widely used ZIP file utility out there, I just can't seem to find out. Something with the same maturity as Apache Commons, Google Collections, Joda Time
I'm trying to do the simplest task of getting a zip file as a byte array (ZipInputStream) and extract it to a folder. this seems like a very tedious task.
I would hope for a syntactic sugar API that does somethnig like this:
public class MyDreamZIPUtils
public static void extractToFolder(ZipInputStream zin, File outputFolderRoot){
...
}
public static void extractToFolder(ZipFile zf, File outputFolderRoot){
...
}
public static zipFolder(File folderToZip, File zippedFileLocation){
...
}
public static zipFolder(File folderToZip, ByteArrayOutputStream zipResult){
...
}
Anything like this?
Am I missing something?

http://commons.apache.org/compress/
I am sure you can write the "syntactic sugar" on top of that.
Javadoc: http://commons.apache.org/compress/apidocs/index.html

I used only Java API calls... I did not do all your methods. you can figure them out from here... Please note i do not claim that the code is bug free... use at your own risk :)
public static void extractToFolder(ZipInputStream zin, File outputFolderRoot)
throws IOException {
FileOutputStream fos = null;
byte[] buf = new byte[1024];
ZipEntry zipentry;
for (zipentry = zin.getNextEntry(); zipentry != null; zipentry = zin.getNextEntry()) {
try {
String entryName = zipentry.getName();
System.out.println("Extracting: " + entryName);
int n;
File newFile = new File(outputFolderRoot, entryName);
if (zipentry.isDirectory()) {
newFile.mkdirs();
continue;
} else {
newFile.getParentFile().mkdirs();
newFile.createNewFile();
}
fos = new FileOutputStream(newFile);
while ((n = zin.read(buf, 0, 1024)) > -1)
fos.write(buf, 0, n);
fos.close();
zin.closeEntry();
} catch (Exception e) {
e.printStackTrace();
} finally {
if (fos != null)
try {
fos.close();
} catch (Exception ignore) {
}
}
}
zin.close();
}
public static void zipFolder(File folderToZip, File zippedFileLocation) throws IOException {
// create a ZipOutputStream to zip the data to
ZipOutputStream zos = new ZipOutputStream(new FileOutputStream(zippedFileLocation));
String path = "";
zipDir(folderToZip, zos, path);
// close the stream
zos.close();
}
private static void zipDir(File directory, ZipOutputStream zos, String path) throws IOException {
File zipDir = directory;
// get a listing of the directory content
String[] dirList = zipDir.list();
byte[] readBuffer = new byte[2156];
int bytesIn = 0;
// loop through dirList, and zip the files
for (int i = 0; i < dirList.length; i++) {
File f = new File(zipDir, dirList[i]);
if (f.isDirectory()) {
zipDir(new File(f.getPath()), zos, path + f.getName() + "/");
continue;
}
FileInputStream fis = new FileInputStream(f);
try {
ZipEntry anEntry = new ZipEntry(path + f.getName());
zos.putNextEntry(anEntry);
bytesIn = fis.read(readBuffer);
while (bytesIn != -1) {
zos.write(readBuffer, 0, bytesIn);
bytesIn = fis.read(readBuffer);
}
} finally {
fis.close();
}
}
}
Reference Java2s