Concurrent use of ZipOutputStream uses 100% of CPU - java

I am working in a feature for an LMS to download a bunch of selected files and folders in a zip on-the-fly. I have used ZipOutputStream to prevent OutOfMemory issues.
The feature works nice, but we have done a stress test and when several users are downloading zips at the same time (lets say 10 users zipping about 100 MB each one), 4 out of 4 CPUs reach 100% of load until the zips are created. Our system admins think that this is not acceptable.
I wonder if there is some mechanism to do ZipOutputStream use less system resources, no matter if it takes more time to finish.
My current code:
protected void compressResource(ZipOutputStream zipOut, String collectionId, String rootFolderName, String resourceId) throws Exception
{
if (ContentHostingService.isCollection(resourceId))
{
try
{
ContentCollection collection = ContentHostingService.getCollection(resourceId);
List<String> children = collection.getMembers();
if(children != null)
{
for(int i = children.size() - 1; i >= 0; i--)
{
String child = children.get(i);
compressResource(zipOut,collectionId,rootFolderName,child);
}
}
}
catch (PermissionException e)
{
//Ignore
}
}
else
{
try
{
ContentResource resource = ContentHostingService.getResource(resourceId);
String displayName = isolateName(resource.getId());
displayName = escapeInvalidCharsEntry(displayName);
InputStream content = resource.streamContent();
byte data[] = new byte[1024 * 10];
BufferedInputStream bContent = null;
try
{
bContent = new BufferedInputStream(content, data.length);
String entryName = (resource.getContainingCollection().getId() + displayName);
entryName=entryName.replace(collectionId,rootFolderName+"/");
entryName = escapeInvalidCharsEntry(entryName);
ZipEntry resourceEntry = new ZipEntry(entryName);
zipOut.putNextEntry(resourceEntry); //A duplicate entry throw ZipException here.
int bCount = -1;
while ((bCount = bContent.read(data, 0, data.length)) != -1)
{
zipOut.write(data, 0, bCount);
}
try
{
zipOut.closeEntry();
}
catch (IOException ioException)
{
logger.error("IOException when closing zip file entry",ioException);
}
}
catch (IllegalArgumentException iException)
{
logger.error("IllegalArgumentException while creating zip file",iException);
}
catch (java.util.zip.ZipException e)
{
//Duplicate entry: ignore and continue.
try
{
zipOut.closeEntry();
}
catch (IOException ioException)
{
logger.error("IOException when closing zip file entry",ioException);
}
}
finally
{
if (bContent != null)
{
try
{
bContent.close();
}
catch (IOException ioException)
{
logger.error("IOException when closing zip file",ioException);
}
}
}
}
catch (PermissionException e)
{
//Ignore
}
}
}
Thanks in advance.

I have solved it with a simple hack told by #shmosel.
private static Semaphore mySemaphore= new Semaphore(ServerConfigurationService.getInt("content.zip.download.maxconcurrentdownloads",5),true);
(...)
ZipOutputStream zipOut = null;
try
{
mySemaphore.acquire();
ContentCollection collection = ContentHostingService.getCollection(collectionId);
(...)
zipOut.flush();
zipOut.close();
mySemaphore.release();
(...)
This is working in my test server. But if anybody has any objection or any extra advice, I will be happy to hear.

Related

File deleted without getting copied

I am trying to copy some Files to SD Card and then delete them . But many times the files are not getting copied and only getting Deleted.
And also many times the FileInputStreamis null where as i am checking if the file which has to be transferred exists or not and also if it is writable or not.
This is the Code i am using to move a file
public static void move(final File remove,final DocumentFile move_to_folder) {
final String mime = MimeTypes.getMimeType(remove);
final DocumentFile move = move_to_folder.createFile(mime, remove.getName());
try {
inStream = new FileInputStream(remove);
outStream =
con.getApplicationContext().getContentResolver().openOutputStream(move.getUri());
final byte[] buffer = new byte[4096];
int bytesRead;
while ((bytesRead = inStream.read(buffer)) != -1) {
outStream.write(buffer, 0, bytesRead);
}
} catch (IOException e) {
e.printStackTrace();
} finally {
try {
if(inStream!=null)
{
inStream.close();
}
if(outStream!=null)
{
outStream.close();
}
delete(remove);
} catch (IOException e) {
e.printStackTrace();
}
}
}
}
I am transferring many files at a time so i am using this code inside an Async Task .
Any help would be really Grateful.
If you have an exception, the remove gets deleted without consideration
Consider adding a boolean flag to prevent this
e.g.
before the try block add
boolean canDelete = true;
If you have an exception set
canDelete = false;
and then in the finally check this boolean
if (canDelete)
delete(remove);

Java Android - Still getting old file

I'm still receiving 1st file my app generated for me.
First , I thought it's because the file exists so I wrote
File file=new File(getCacheDir(), "Competition.xls");
if (file.exists()) {file.delete(); file =new File(getCacheDir(), "Competition.xls");}
But that didn't help me-I still receive first file that was made
I'm new to working with files so I decided to copy entire method here. Sorry for a lot of text.
private void createFileTosend() {
InputStream inputStream = null;
FileOutputStream outputStream = null;
try {
File toSend=null;
try {
toSend = getFile();
} catch (WriteException e) {
// TODO Auto-generated catch block
e.printStackTrace();
} catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
inputStream = new FileInputStream(toSend);
outputStream = openFileOutput("Competition.xls",
Context.MODE_WORLD_READABLE | Context.MODE_APPEND);
byte[] buffer = new byte[1024];
int length = 0;
try {
while ((length = inputStream.read(buffer)) > 0){
outputStream.write(buffer, 0, length);
}
} catch (IOException ioe) {
/* ignore */
}
} catch (FileNotFoundException fnfe) {
/* ignore */
} finally {
try {
inputStream.close();
} catch (IOException ioe) {
/* ignore */
}
try {
outputStream.close();
} catch (IOException ioe) {
/* ignore */
}
}
}
public File getFile() throws IOException, WriteException{
File file=new File(getCacheDir(), "Competition.xls");
if (file.exists()) {file.delete(); file =new File(getCacheDir(), "Competition.xls");}
WritableWorkbook workbook = Workbook.createWorkbook(file);
//then goes long block with creating a .xls file which is not important
workbook.write();
workbook.close();
return file;
}
Help on understanding where the problem is
You should never have a structure like :
catch(Exception ex ) {
//ignore (or log only)
}
Exception are there to tell you something went wrong. What you do is called (in french) "eating/hiding exceptions". You are loosing this very important information that something went abnormally.
You should always either throw the exception you catch to your caller, or process it locally. At the very least, and this is a poor practice, you should log it. But doing nothing is just very wrong.
Here, put the whole try catch in a method for instance :
private void createFileTosend() throws IOException {
InputStream inputStream = null;
FileOutputStream outputStream = null;
try {
File toSend = getFile();
inputStream = new FileInputStream(toSend);
outputStream = openFileOutput("Competition.xls",
Context.MODE_WORLD_READABLE | Context.MODE_APPEND);
byte[] buffer = new byte[1024];
int length = 0;
while ((length = inputStream.read(buffer)) > 0){
outputStream.write(buffer, 0, length);
}
} finally {
try {
if( inputStream != null ) {
inputStream.close();
}
} catch (IOException ioe) {
Log.e( ioe );
}
try {
if( outputStream != null ) {
outputStream.close();
}
} catch (IOException ioe) {
Log.e( ioe );
}
}
}
And now, when you call createFileToSend, do that in a try/catch structure and toast a message, or something if you catch an exception.

How to manage streams of bytes and when to close the streams

I'm experiencing java.lang.OutOfMemoryError: Java heap space whenever I try to execute my code. However, if I close my streams in certain instances the error goes away, but because my streams are closing prematurely I'm missing data.
I'm very new to Java and I'm clearly not understanding how to manage the streams. How and when should I close streams?
private void handleFile(File source)
{
FileInputStream fis = null;
try
{
if(source.isFile())
{
fis = new FileInputStream(source);
handleFile(source.getAbsolutePath(), fis);
}
else if(source.isDirectory())
{
for(File file:source.listFiles())
{
if(file.isFile())
{
fis = new FileInputStream(file);
handleFile(file, fis);
}
else
{
handleFile(file);
}
}
}
}
catch(IOException ioe)
{
ioe.printStackTrace();
}
finally
{
try
{
if(fis != null) { fis.close(); }
}
catch(IOException ioe) { ioe.printStackTrace(); }
}
}
private handleFile(String fileName, InputStream inputStream)
{
try
{
byte[] initialBytes = isToByteArray(inputStream);
byte[] finalBytes = initialBytes;
if(initialBytes.length == 0) return;
if(isBytesTypeB(initialBytes))
{
finalBytes = getBytesTypeB(startingBytes);
}
// Other similar method checks
// .....
map.put(fileName, finalBytes);
}
catch(IOException ioe)
{
ioe.printStackTrace();
}
}
private byte[] isToByteArray(InputStream inputStream)
{
ByteArrayOutputStream baos = new ByteArrayOutputStream();
byte[] buffer = new byte[1024];
int nRead;
while((nRead = inputStream.read(buffer)) != -1)
{
baos.write(buffer, 0, nRead);
}
return baos.toByteArray();
}
private boolean isBytesTypeB(byte[] fileBytes)
{
// Checks if these bytes match a particular type
if(BytesMatcher.matches(fileBytes, fileBytes.length))
{
return true;
}
return false;
}
private byte[] getBytesTypeB(byte[] fileBytes)
{
//decompress bytes
return decompressedBytes;
}
First of all, do not read the entire streams in memory. Use buffers when reading and writing.
Use ByteArrayInputStream and ByteArrayInputStream only if you're sure you'll be reading very small streams (whose data you will need to re-use for some operations) and it really makes sense to keep the data in memory. Otherwise, you will quickly (or unexpectedly) run out of memory.
Define the streams outside a try-catch block and close them in the finally block (if they are not null). For example:
void doSomeIOStuff() throws IOException
{
InputStream is = null;
try
{
is = new MyInputStream(...);
// Do stuff
}
catch (IOException ioExc)
{
// Either just inform (poor decision, but good for illustration):
ioExc.printStackTrace();
// Or re-throw to delegate further on:
throw new IOException(ioExc);
}
finally
{
if (is != null)
{
is.close();
}
}
}
This way your resources are always properly closed after use.
Out of curiosity, what should the handleFile(...) method really be doing?

Unnecessary file structure is added while "zipping" a directory using java?

well my question is really simple, is about an unexpected behavior (or at least is unexpected to me) while I try to zip a directory, I have the following methods that I've created on my own (I'm quite aware that I'm not handling exceptions and all that stuff, It is because (by now) I'm just doing this to learn how to do it so stability "is not really important"), here is the code:
public static void zipDirectory(File srcDirectory, File zipFile) throws IllegalArgumentException {
if (!srcDirectory.isDirectory()) {
throw new IllegalArgumentException("The first parameter (srcDirectory) MUST be a directory.");
}
int bytesRead;
byte[] dataRead = new byte[1000];
BufferedInputStream in = null;
ZipOutputStream zOut;
try {
zOut = new ZipOutputStream(new BufferedOutputStream(new FileOutputStream(zipFile)));
for (File f : srcDirectory.listFiles()) {
if (f.isDirectory()) {
FileUtilities.zipInnerDirectory(f,zOut);
}else {
in = new BufferedInputStream(new FileInputStream(f.getAbsolutePath()), 1000);
zOut.putNextEntry(new ZipEntry(f.getPath()));
while((bytesRead = in.read(dataRead,0,1000)) != -1) {
zOut.write(dataRead, 0, bytesRead);
}
zOut.closeEntry();
}
}
zOut.flush();
zOut.close();
} catch (FileNotFoundException e) {
// TODO Auto-generated catch block
e.printStackTrace();
} catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}
private static void zipInnerDirectory(File dir, ZipOutputStream zOut) throws IllegalArgumentException {
if (!dir.isDirectory()) {
throw new IllegalArgumentException("The first parameter (srcDirectory) MUST be a directory.");
}
BufferedInputStream in = null;
int bytesRead;
byte[] dataRead = new byte[1000];
try {
for (File f : dir.listFiles()) {
if (f.isDirectory()) {
FileUtilities.zipInnerDirectory(f,zOut);
}else {
in = new BufferedInputStream(new FileInputStream(f.getAbsolutePath()), 1000);
zOut.putNextEntry(new ZipEntry(f.getPath()));
while((bytesRead = in.read(dataRead,0,1000)) != -1) {
zOut.write(dataRead, 0, bytesRead);
}
zOut.closeEntry();
}
}
zOut.flush();
zOut.close();
} catch (FileNotFoundException e) {
// TODO Auto-generated catch block
e.printStackTrace();
} catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}
As I said is not my best coding so please don't judge the code (or at least don't be too strict ;) ), I know it can be so much better; ok the "unexpected behavior" is this, let's say that I have the following directory:
H:\MyDir1\MyDir2\MyDirToZip
when i send as a parameter a file created with that path (new File("H:\\MyDir1\\MyDir2\\MyDirToZip")) everything's work pretty fine the zip is created successfully, the thing is that when I open (unzip) the files inside the zip they have the next structure:
H:\MyDir1\MyDir2\MyDirToZip
when I was expecting to find inside just:
\MyDirToZip
without H: \MyDir1 \MyDir2 which are "unnecessary" (BTW they just contain one to each other in the appropriate order, i mean, the other files that are in them are not compressed, that is why I say they are unnecessary) so the question is, what I'm I doing wrong? how can I specify that I just want to zip the structure down the srcDirectory?
zOut.putNextEntry(new ZipEntry(f.getPath()));
This should be the problem. f.getPath() will return a path that's relative to some root directory (probably your current working dir), but not relative to the directory you are zipping. You need to figure out a way to get the relative path from the zip directory, possibly this will do:
new ZipEntry(f.getAbsolutePath().substring(zipDir.getAbsolutePath().length()))
or, if you want the root directory added:
new ZipEntry(zipDir.getName() + "/"
+ f.getAbsolutePath().substring(zipDir.getAbsolutePath().length()))

Zip files based on InputStreams

I have a method to zip files in Java:
public void compress(File[] inputFiles, OutputStream outputStream) {
Validate.notNull(inputFiles, "Input files are required");
Validate.notNull(outputStream, "Output stream is required");
int BUFFER = 2048;
BufferedInputStream origin = null;
ZipOutputStream out = new ZipOutputStream(new BufferedOutputStream(outputStream));
byte data[] = new byte[BUFFER];
for (File f : inputFiles) {
FileInputStream fi;
try {
fi = new FileInputStream(f);
} catch (FileNotFoundException e) {
throw new RuntimeException("Input file not found", e);
}
origin = new BufferedInputStream(fi, BUFFER);
ZipEntry entry = new ZipEntry(f.getName());
try {
out.putNextEntry(entry);
} catch (IOException e) {
throw new RuntimeException(e);
}
int count;
try {
while ((count = origin.read(data, 0, BUFFER)) != -1) {
out.write(data, 0, count);
}
} catch (IOException e) {
throw new RuntimeException(e);
}
try {
origin.close();
} catch (IOException e) {
throw new RuntimeException(e);
}
}
try {
out.close();
} catch (IOException e) {
throw new RuntimeException(e);
}
}
As you can see parameter inputFiles is an Array of File objects. This all works, but I'd like to have instead a collection of InputStream objects as parameter to make it more flexible.
But then I have the problem that when making a new ZipEntry (as in code above)
ZipEntry entry = new ZipEntry(f.getName());
I don't have a filename to give as parameter.
How should i solve this? Maybe a Map with (fileName,inputStream) pairs?
Any thoughts on this are appreciated!
Thanks,
Nathan
I think your suggestion Map<String, InputStream> is a good solution.
Just a side note: Remember to close the inputstreams after you are done
If you want to make it more "fancy" you can always use create an interface:
interface ZipOuputInterface {
String getName();
InputStream getInputStream();
}
And have it implemented differently in your different cases for instance File:
class FileZipOutputInterface implements ZipOutputInterface {
File file;
public FileZipOutputInterface(File file) {
this.file = file;
}
public String getName() {
return file.getAbstractName();
}
public InputStream getInputStream() {
return new FileInputStream(file);
}
}
I think that map is good. Just pay attention on the type ofmap you are using if you wish to preserve the original order of files in your ZIP. In this case use LinkedHashMap.

Categories