We Keep Coding

Concurrent use of ZipOutputStream uses 100% of CPU

I am working in a feature for an LMS to download a bunch of selected files and folders in a zip on-the-fly. I have used ZipOutputStream to prevent OutOfMemory issues.
The feature works nice, but we have done a stress test and when several users are downloading zips at the same time (lets say 10 users zipping about 100 MB each one), 4 out of 4 CPUs reach 100% of load until the zips are created. Our system admins think that this is not acceptable.
I wonder if there is some mechanism to do ZipOutputStream use less system resources, no matter if it takes more time to finish.
My current code:
protected void compressResource(ZipOutputStream zipOut, String collectionId, String rootFolderName, String resourceId) throws Exception
{
if (ContentHostingService.isCollection(resourceId))
{
try
{
ContentCollection collection = ContentHostingService.getCollection(resourceId);
List<String> children = collection.getMembers();
if(children != null)
{
for(int i = children.size() - 1; i >= 0; i--)
{
String child = children.get(i);
compressResource(zipOut,collectionId,rootFolderName,child);
}
}
}
catch (PermissionException e)
{
//Ignore
}
}
else
{
try
{
ContentResource resource = ContentHostingService.getResource(resourceId);
String displayName = isolateName(resource.getId());
displayName = escapeInvalidCharsEntry(displayName);
InputStream content = resource.streamContent();
byte data[] = new byte[1024 * 10];
BufferedInputStream bContent = null;
try
{
bContent = new BufferedInputStream(content, data.length);
String entryName = (resource.getContainingCollection().getId() + displayName);
entryName=entryName.replace(collectionId,rootFolderName+"/");
entryName = escapeInvalidCharsEntry(entryName);
ZipEntry resourceEntry = new ZipEntry(entryName);
zipOut.putNextEntry(resourceEntry); //A duplicate entry throw ZipException here.
int bCount = -1;
while ((bCount = bContent.read(data, 0, data.length)) != -1)
{
zipOut.write(data, 0, bCount);
}
try
{
zipOut.closeEntry();
}
catch (IOException ioException)
{
logger.error("IOException when closing zip file entry",ioException);
}
}
catch (IllegalArgumentException iException)
{
logger.error("IllegalArgumentException while creating zip file",iException);
}
catch (java.util.zip.ZipException e)
{
//Duplicate entry: ignore and continue.
try
{
zipOut.closeEntry();
}
catch (IOException ioException)
{
logger.error("IOException when closing zip file entry",ioException);
}
}
finally
{
if (bContent != null)
{
try
{
bContent.close();
}
catch (IOException ioException)
{
logger.error("IOException when closing zip file",ioException);
}
}
}
}
catch (PermissionException e)
{
//Ignore
}
}
}
Thanks in advance.

I have solved it with a simple hack told by #shmosel.
private static Semaphore mySemaphore= new Semaphore(ServerConfigurationService.getInt("content.zip.download.maxconcurrentdownloads",5),true);
(...)
ZipOutputStream zipOut = null;
try
{
mySemaphore.acquire();
ContentCollection collection = ContentHostingService.getCollection(collectionId);
(...)
zipOut.flush();
zipOut.close();
mySemaphore.release();
(...)
This is working in my test server. But if anybody has any objection or any extra advice, I will be happy to hear.

Java Android - Still getting old file

I'm still receiving 1st file my app generated for me.
First , I thought it's because the file exists so I wrote
File file=new File(getCacheDir(), "Competition.xls");
if (file.exists()) {file.delete(); file =new File(getCacheDir(), "Competition.xls");}
But that didn't help me-I still receive first file that was made
I'm new to working with files so I decided to copy entire method here. Sorry for a lot of text.
private void createFileTosend() {
InputStream inputStream = null;
FileOutputStream outputStream = null;
try {
File toSend=null;
try {
toSend = getFile();
} catch (WriteException e) {
// TODO Auto-generated catch block
e.printStackTrace();
} catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
inputStream = new FileInputStream(toSend);
outputStream = openFileOutput("Competition.xls",
Context.MODE_WORLD_READABLE | Context.MODE_APPEND);
byte[] buffer = new byte[1024];
int length = 0;
try {
while ((length = inputStream.read(buffer)) > 0){
outputStream.write(buffer, 0, length);
}
} catch (IOException ioe) {
/* ignore */
}
} catch (FileNotFoundException fnfe) {
/* ignore */
} finally {
try {
inputStream.close();
} catch (IOException ioe) {
/* ignore */
}
try {
outputStream.close();
} catch (IOException ioe) {
/* ignore */
}
}
}
public File getFile() throws IOException, WriteException{
File file=new File(getCacheDir(), "Competition.xls");
if (file.exists()) {file.delete(); file =new File(getCacheDir(), "Competition.xls");}
WritableWorkbook workbook = Workbook.createWorkbook(file);
//then goes long block with creating a .xls file which is not important
workbook.write();
workbook.close();
return file;
}
Help on understanding where the problem is

You should never have a structure like :
catch(Exception ex ) {
//ignore (or log only)
}
Exception are there to tell you something went wrong. What you do is called (in french) "eating/hiding exceptions". You are loosing this very important information that something went abnormally.
You should always either throw the exception you catch to your caller, or process it locally. At the very least, and this is a poor practice, you should log it. But doing nothing is just very wrong.
Here, put the whole try catch in a method for instance :
private void createFileTosend() throws IOException {
InputStream inputStream = null;
FileOutputStream outputStream = null;
try {
File toSend = getFile();
inputStream = new FileInputStream(toSend);
outputStream = openFileOutput("Competition.xls",
Context.MODE_WORLD_READABLE | Context.MODE_APPEND);
byte[] buffer = new byte[1024];
int length = 0;
while ((length = inputStream.read(buffer)) > 0){
outputStream.write(buffer, 0, length);
}
} finally {
try {
if( inputStream != null ) {
inputStream.close();
}
} catch (IOException ioe) {
Log.e( ioe );
}
try {
if( outputStream != null ) {
outputStream.close();
}
} catch (IOException ioe) {
Log.e( ioe );
}
}
}
And now, when you call createFileToSend, do that in a try/catch structure and toast a message, or something if you catch an exception.

Download and save a file using Java

Let's say I have an URL, like something.domain/myfile.txt then I want to save this file, with that "Save File" dialog.
I tried my best to do it, but everytime I save the file using the dialog the file is not there.
An example or somewhere I can find information on this would help a lot!
URL website = null;
try {
website = new URL(<insert url here>);
} catch (MalformedURLException e) {
e.printStackTrace();
}
ReadableByteChannel rbc = null;
try {
rbc = Channels.newChannel(website.openStream());
} catch (IOException e2) {
e2.printStackTrace();
}
FileOutputStream fos = null;
try {
fos = new FileOutputStream(new File("minecraft.jar"));
} catch (FileNotFoundException e1) {
e1.printStackTrace();
}
try {
fos.getChannel().transferFrom(rbc, 0, 1 << 24);
} catch (IOException e) {
e.printStackTrace();
}
JFileChooser fileChooser = new JFileChooser();
if (fileChooser.showSaveDialog(fileChooser) == JFileChooser.APPROVE_OPTION) {
File dir = fileChooser.getCurrentDirectory();
dir.mkdir();
//After this point is where I need help.

I trust that this is what you're looking for:
if (fileChooser.showSaveDialog(fileChooser) == JFileChooser.APPROVE_OPTION)
{
File file = fileChooser.getSelectedFile();
// whatever you want to do with the file
System.out.println("The file is "+file.getAbsolutePath());
// fos = new FileOutputStream(file) ...
}

Did you notice that in your code you are trying to save/download the file before giving the user the option to chose the destination?
I would split the code into three different operations:
A method in charge of transferring the bytes from the InputStream (the web) to the OutputStream (the file).
a method to show the user a dialog so he can chose where to store the file.
the method that completes the whole task: choose a file and transfer the bytes from the web to it.
1) Would be something like this (you don't need to use the NIO API to implement it):
public void transfer(InputStream in, OutputStream out) throws IOException {
byte[] buffer = new byte[2048];
int bytesRead;
while ((bytesRead = in.read(buffer)) > 0) {
out.write(buffer, 0, bytesRead);
}
}
2) would be something very similar to what Dukeling has already stated:
public File chooseFile() {
File result = null;
JFileChooser fileChooser = new JFileChooser();
if (fileChooser.showSaveDialog(fileChooser) == JFileChooser.APPROVE_OPTION) {
result = fileChooser.getSelectedFile();
}
return result;
}
3) then, combining these two operations together is really simple:
public void saveFileFromWeb(URL url) {
File file = chooseFile(); // 1. Choose the destination file
if (file != null) {
// 2. Create parent folder structure
File folder = file.getParentFile();
if (!folder.exist()) {
folder.mkdirs();
}
InputStream in = null;
OutputStream out = null;
try {
// 3. Initialise streams
in = url.openStream();
out = new FileOuputStream(file);
// 4. Transfer data
transfer(in, out);
} catch (IOException e) {
e.printStackTrace();
} finally {
// 5. close streams
if (in != null) {
try {
in.close();
} catch (IOException e) { /* ignore */ }
}
if (out != null) {
try {
out.close();
} catch (IOException e) { /* ignore */ }
}
}
}
NOTE: 1) and 2) could be private methods. Of course you could do this is just one single operation but splitting it would give you an overview of the different steps to perform.
NOTE 2: I simplified the exception handling part

Using JarEntry to write sound files is not working as intended

public static void writeFile(String theFileName, String theFilePath)
{
try {
File currentFile = new File("plugins/mcMMO/Resources/"+theFilePath+theFileName);
//System.out.println(theFileName);
#SuppressWarnings("static-access")
JarFile jar = new JarFile(plugin.mcmmo);
JarEntry entry = jar.getJarEntry("resources/"+theFileName);
InputStream is = jar.getInputStream(entry);
byte[] buf = new byte[(int)entry.getSize()];
is.read(buf, 0, buf.length);
FileOutputStream os = new FileOutputStream(currentFile);
os.write(buf);
os.close();
} catch (FileNotFoundException e) {
// TODO Auto-generated catch block
e.printStackTrace();
} catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}
Alright so in my program I have various resources kept within the Jar of the program, when the program runs it has specific files passed to this function that are written to the HDD of the users computer. Everything gets written, but only the images come out 100% correct. The sound files are not so lucky.
Basically, I CANNOT get the sounds to write correctly, their file sizes are correct but they only contain a split second of audio instead of their full length audio. Am I missing something here? I seem to have done everything right, but if that was true I wouldn't be posting here.
I tried my best at googling this problem but it has failed me.
Any guess as to why this doesn't work would be AMAZING!! :)

As JarEntry extends ZipEntry, I would recommend not to rely on the ZipEntry.getSize() method as it return -1. See the doc.
Moreover, it's in general much more common to take advantage of buffering when reading a stream. In your example, you put everything inside your byte array, so I guess for big files you could end up in an OutOfMemoryError.
Here would be the code I would test:
public static void writeFile(String theFileName, String theFilePath)
{
try {
File currentFile = new File("plugins/mcMMO/Resources/"+theFilePath+theFileName);
#SuppressWarnings("static-access")
JarFile jar = new JarFile(plugin.mcmmo);
JarEntry entry = jar.getJarEntry("resources/"+theFileName);
InputStream is = jar.getInputStream(entry);
byte[] buf = new byte[2048];
int nbRead;
OutputStream os = new BufferedOutputStream(new FileOutputStream(currentFile));
while((nbRead = is.read(buf)) != -1) {
os.write(buf, 0, nbRead);
}
os.flush();
os.close();
} catch (FileNotFoundException e) {
e.printStackTrace();
} catch (IOException e) {
e.printStackTrace();
}
}

How to check if a generated zip file is corrupted?

we have a piece of code which generates a zip file on our system. Everything is ok, but sometimes this zip file while opened by FilZip or WinZip is considered to be corrupted.
So here is my question: how can we check programatically if a generated zip file is corrupted?
Here is the code we are using to generate our zip files:
try {
ZipOutputStream zos = new ZipOutputStream(new FileOutputStream(tmpFile));
byte[] buffer = new byte[16384];
int contador = -1;
for (DigitalFile digitalFile : document.getDigitalFiles().getContent()) {
ZipEntry entry = new ZipEntry(digitalFile.getName());
FileInputStream fis = new FileInputStream(digitalFile.getFile());
try {
zos.putNextEntry(entry);
while ((counter = fis.read(buffer)) != -1) {
zos.write(buffer, 0, counter);
}
fis.close();
zos.closeEntry();
} catch (IOException ex) {
throw new OurException("It was not possible to read this file " + arquivo.getId());
}
}
try {
zos.close();
} catch (IOException ex) {
throw new OurException("We couldn't close this stream", ex);
}
Is there anything we are doing wrong here?
EDIT:
Actually, the code above is absolutely ok. My problem was that I was redirecting the WRONG stream for my users. So, instead of opening a zip file they where opening something completely different. Mea culpa :(
BUT the main question remains: how programatically I can verify if a given zip file is not corrupted?

You can use the ZipFile class to check your file :
static boolean isValid(final File file) {
ZipFile zipfile = null;
try {
zipfile = new ZipFile(file);
return true;
} catch (IOException e) {
return false;
} finally {
try {
if (zipfile != null) {
zipfile.close();
zipfile = null;
}
} catch (IOException e) {
}
}
}

I know its been a while that this has been posted, I have used the code that all of you provided and came up with this. This is working great for the actual question. Checking if the zip file is corrupted or not
private boolean isValid(File file) {
ZipFile zipfile = null;
ZipInputStream zis = null;
try {
zipfile = new ZipFile(file);
zis = new ZipInputStream(new FileInputStream(file));
ZipEntry ze = zis.getNextEntry();
if(ze == null) {
return false;
}
while(ze != null) {
// if it throws an exception fetching any of the following then we know the file is corrupted.
zipfile.getInputStream(ze);
ze.getCrc();
ze.getCompressedSize();
ze.getName();
ze = zis.getNextEntry();
}
return true;
} catch (ZipException e) {
return false;
} catch (IOException e) {
return false;
} finally {
try {
if (zipfile != null) {
zipfile.close();
zipfile = null;
}
} catch (IOException e) {
return false;
} try {
if (zis != null) {
zis.close();
zis = null;
}
} catch (IOException e) {
return false;
}
}
}

I think you'll see correspondent exception stack trace during zip-file generation. So, you probably wan't to enhance your exception handling.

in my implementation it looks like that. maybe it helps you:
//[...]
try {
FileInputStream fis = new FileInputStream(file);
BufferedInputStream bis = new BufferedInputStream(fis);
zos.putNextEntry(new ZipEntry(file.getName()));
try {
final byte[] buf = new byte[BUFFER_SIZE];
while (true) {
final int len = bis.read(buf);
if (len == -1) {
break;
}
zos.write(buf, 0, len);
}
zos.flush();
zos.closeEntry();
} finally {
try {
bis.close();
} catch (IOException e) {
LOG.debug("Buffered Stream closing failed");
} finally {
fis.close();
}
}
} catch (IOException e) {
throw new Exception(e);
}
//[...]
zos.close

Perhaps swap the following two lines?;
fis.close();
zos.closeEntry();
I can imagine that the closeEntry() will still read some data from the stream.

Your code is basically OK, try to find out which file is responsible for the corrupted zip file. Check whether digitalFile.getFile() always returns a valid and accessible argument to FileInputStream. Just add a bit logging to your code and you will find out what's wrong.

new ZipFile(file)
compress again the file, so duplicate efforts and that is not what you are looking for. Despite of the fact that only check one file and the question compress n-files.
Take a look to this: http://www.kodejava.org/examples/336.html
Create a checksum for your zip:
CheckedOutputStream checksum = new CheckedOutputStream(fos, new CRC32());
ZipOutputStream zos = new ZipOutputStream(new BufferedOutputStream(checksum));
...
And when you finish the compression show it
System.out.println("Checksum : " + checksum.getChecksum().getValue());
You must do the same reading the zip with java or others tools checking if checksums match.
see https://stackoverflow.com/a/10689488/848072 for more information

ZipOutputStream does not close the underlying stream.
What you need to do is:
FileOutputStream fos = new FileOutputStream(...);
ZipOutputStream zos = new ZipOutputStream(fos);
Then in your closing block:
zos.close();
fos.flush(); // Can't remember whether this is necessary off the top of my head!
fos.close();