I am trying to add a text file to a zip archive through a Java program on Linux. The program spawns a process (using java.lang.Process) to execute the commandline "zip -j .zip .txt", reads the output and error streams of the spawned process and waits for the process to complete using waitFor(). Though the program seems to run fine (spawned process exits with exit code 0, indicating that the zip commandline was executed successfully) and the output read from output and error streams do not indicate any errors, at the end of the program the zip archive doesn't always contain the file supposed to have been added. This problem doesn't happen consistently though (even with the same existing-archive and file-to-add) - once in a while (perhaps once in 4 attempts) the zip is found to have been updated correctly. Strangely, the problem doesn't occur at all when the program is run through Eclipse debugger mode. Any pointers on why this problem occurs and how it can be addressed would be helpful. Thanks!
Below is the code snippet. The program calls addFileToZip(File, File, String):
public static void addFileToZip(final File zipFile, final File fileToBeAdded,
final String fileNameToBeAddedAs) throws Exception {
File tempDir = createTempDir();
File fileToBeAddedAs = new File(tempDir, fileNameToBeAddedAs);
try {
FileUtils.copyFile(fileToBeAdded, fileToBeAddedAs);
addFileToZip(zipFile, fileToBeAddedAs);
} finally {
deleteFile(fileToBeAddedAs);
deleteFile(tempDir);
}
}
public static void addFileToZip(final File zipFile, final File fileToBeAdded) throws Exception {
final String[] command = {"zip", "-j", zipFile.getAbsolutePath(), fileToBeAdded.getAbsolutePath()};
ProcessBuilder procBuilder = new ProcessBuilder(command);
Process proc = procBuilder.start();
int exitCode = proc.waitFor();
/*
* Code to read output/error streams of proc and log/print them
*/
if (exitCode != 0) {
throw new Exception("Unable to add file, error: " + errMsg);
}
}
Make sure no other process has the zip file locked for write, or the file being added locked for read. If you're generating the file to be added, make sure the stream is flushed and closed before spawning the zip utility.
I am trying to add a text file to a zip archive through a Java program on Linux.
Use the java.util.zip API, which:
Provides classes for reading and writing the standard ZIP and GZIP file formats.
If you intend to stick with using a Process to do this, be sure to implement all the suggestions of When Runtime.exec() won't.
Related
I have a process created as follows:
Process p = Runtime.getRuntime().exec(new String[]{"su"});
In my program, I only want to create this process once. I am developing a root file explorer application for Android, and whenever this process is created, the Android device will prompt the user to grant root permissions. This is a very slow operation, and as this is a file browser, it will need root permissions often. So, I have decided to create this process once and write commands to its OutputStream in the following manner (stdin is this OutputStream):
stdin.writeBytes(command + "\n");
Before I can read the output of the command, I need my program to wait until the command written by writeBytes has terminated. I have tried p.waitFor(), but this causes the program to hang.
Here is how I read bytes from the InputStream:
int read;
String out = "";
stdout = p.getInputStream();
byte[] buffer = new byte[262144];
while (true) {
read = stdout.read(buffer);
out += new String(buffer, 0, read);
if (read < BUFF_LEN) {
//we have read everything
break;
}
}
Note that although the read(buffer) method blocks until input data is available, it does not block in this case because it thinks it has reached the end of the InputStream.
I have tried to include only relevant portions of my code in this post, but if you would like to take a look at the entire source code of the class where this is contained, see here: http://pastebin.com/t6JdWmQr.
How can I make sure the command has finished running before reading the process' InputStream?
I also encounter similar problem, and I found the answer here:
Wait until a command in su finishes
If you don't need any read stream in this shell process, simply add shell read stream may completed the shell process.
Or in XDA also have better way:
[HowTo]Execute Root Commands and read output
I open office files (docx, xlsx) by using Runtime.getRuntime().exec(String cmd) function. Simultaneously I store meta data of these files in a database. In order to keep integrity I lock the file with a flag in the meta data so that no other user concurrently can modify the file. This implies that the flag must be automatically resetted after the user closes the file (e.g. closes the external process).
Here's the snippet that opens the file:
File file = new File("c:/test.docx");
Process process = null;
if(file.getName().endsWith("docx")) {
process = Runtime.getRuntime().exec("c:/msoffice/WINWORD.EXE "+file.getAbsolutePath());
} else if(file.getName().endsWith("xlsx")) {
process = Runtime.getRuntime().exec("c:/msoffice/EXCEL.EXE "+file.getAbsolutePath());
}
if(process!=null) {
new ProcessExitListener(file, process);
}
Here's my listener that waits until the user closes the file (and finally unlocks the file by setting the flag in the meta data):
private class ProcessExitListener extends Thread {
private File file;
private Process process;
public ProcessExitListener(File file, Process process) throws IOException {
this.setName("File-Thread ["+process.toString()+"]");
this.file = file;
this.process = process;
this.start();
}
#Override
public void run() {
try {
process.waitFor();
database.unlock(file);
} catch (InterruptedException ex) {
// print exception
}
}
}
This works fine for different file types, e.g. if I open 1 docx and 1 xlsx file simultaneously. But when opening 2 docx files, one of the process exits right after it has been initialized.
Any ideas why ?
Thanks for your help in advance!
But when opening 2 docx files, one of the process exists right after it has been initialized.
Probably because winword.exe process finds out that there is already one instance of it running, so instead of keeping two instances in memory, it just asks the first instance to open the second document. Don't know how it looks from GUI perspective, but looking at the task manager, try opening two Word documents from Windows Explorer. The second file won't cause second winword.exe process to start.
I can reproduce the exact same behaviour on Ubuntu Linux. When I ran:
$ geany foo.txt
and the geany editor wasn't yet running, the console hangs until I close the editor. But if instead I open another terminal and call:
$ geany bar.txt
this returns immediately and bar.txt is simply opened as another tab in already existing process.
I am writing a program that needs to zip a file.
This will run over both linux and windows machines. It works just fine in Linux but I am not able to get anything done in windows.
To send commands I am using the apache-net project. I've also tried using Runtime().exec
but it isn't working.
Can somebody suggest something?
CommandLine cmdLine = new CommandLine("zip");
cmdLine.addArgument("-r");
cmdLine.addArgument("documents.zip");
cmdLine.addArgument("documents");
DefaultExecutor exec = new DefaultExecutor();
ExecuteWatchdog dog = new ExecuteWatchdog(60*1000);
exec.setWorkingDirectory(new File("."));
exec.setWatchdog(dog);
int check =-1;
try {
check = exec.execute(cmdLine);
} catch (ExecuteException e) {
} catch (IOException e) {
}
Java provides its own compression library in java.util.zip.* that supports the .zip format. An example that zips a folder can be found here. Here's a quickie example that works on a single file. The benefit of going with native Java is that it will work on multiple operating systems and is not dependent on having specific binaries installed.
public static void zip(String origFileName) {
try {
String zipName=origFileName + ".zip";
ZipOutputStream out = new ZipOutputStream(new BufferedOutputStream(new FileOutputStream(zipName)));
byte[] data = new byte[1000];
BufferedInputStream in = new BufferedInputStream(new FileInputStream(origFileName));
int count;
out.putNextEntry(new ZipEntry(origFileName));
while((count = in.read(data,0,1000)) != -1) {
out.write(data, 0, count);
}
in.close();
out.flush();
out.close();
} catch (Exception ex) {
ex.printStackTrace();
}
}
The same code won't work in Windows. Windows doesn't have a "zip" program the way that Linux does. You will need to see if Windows 7 has a command line zip program (I don't think it does; see here: http://answers.microsoft.com/en-us/windows/forum/windows_vista-files/how-to-compress-a-folder-from-command-prompt/02f93b08-bebc-4c9d-b2bb-907a2184c8d5). You will likely need to do two things
Make sure the user has a suitable 3rd party zip program
Do OS detection to execute the proper command.
You can use inbuilt compact.exe to compress/uncompress in dos
It displays or alters the compression of files on NTFS partitions.
COMPACT [/C | /U] [/S[:dir]] [/A] [/I] [/F] [/Q] [filename [...]]
/C Compresses the specified files. Directories will be marked so that files added afterward will be compressed.
/U Uncompresses the specified files. Directories will be marked so that files added afterward will not be compressed.
/S Performs the specified operation on files in the given directory and all subdirectories. Default "dir" is the current directory.
/A Displays files with the hidden or system attributes. These files are omitted by default.
/I Continues performing the specified operation even after errors have occurred. By default, COMPACT stops when an error is encountered.
/F Forces the compress operation on all specified files, even those that are already compressed. Already-compressed files are skipped by default.
/Q Reports only the most essential information.
filename Specifies a pattern, file, or directory.
Used without parameters, COMPACT displays the compression state of the current directory and any files it contains. You may use multiple filenames and wildcards. You must put spaces between multiple parameters.
Examples
compact
Display all the files in the current directory and their compact status.
compact file.txt
Display the compact status of the file file.txt
compact file.txt /C
Compacts the file.txt file.
Let's say you had an external process writing files to some directory, and you had a separate process periodically trying to read files from this directory. The problem to avoid is reading a file that the other process is currently in the middle of writing out, so it would be incomplete. Currently, the process that reads uses a minimum file age timer check, so it ignores all files unless their last modified date is more than XX seconds old.
I'm wondering if there is a cleaner way to solve this problem. If the filetype is unknown (could be a number of different formats) is there some reliable way to check the file header for the number of bytes that should be in the file, vs the number of bytes currently in the file to confirm they match?
Thanks for any thoughts or ideas!
The way I've done this in the past is that the process writing the file writes to a "temp" file, and then moves the file to the read location when it has finished writing the file.
So the writing process would write to info.txt.tmp. When it's finished, it renames the file to info.txt. The reading process then just had to check for the existence of info.txt - and it knows that if it exists, it has been written completely.
Alternatively you could have the write process write info.txt to a different directory, and then move it to the read directory if you don't like using weird file extensions.
You could use an external marker file. The writing process could create a file XYZ.lock before it starts creating file XYZ, and delete XYZ.lock after XYZ is completed. The reader would then easily know that it can consider a file complete only if the corresponding .lock file is not present.
I had no option of using temp markers etc as the files are being uploaded by clients over keypair SFTP. they can be very large in size.
Its quite hacky but I compare file size before and after sleeping a few seconds.
Its obviously not ideal to lock the thread but in our case it is merely running as a background system processes so seems to work fine
private boolean isCompletelyWritten(File file) throws InterruptedException{
Long fileSizeBefore = file.length();
Thread.sleep(3000);
Long fileSizeAfter = file.length();
System.out.println("comparing file size " + fileSizeBefore + " with " + fileSizeAfter);
if (fileSizeBefore.equals(fileSizeAfter)) {
return true;
}
return false;
}
Note: as mentioned below this might not work on windows. This was used in a Linux environment.
One simple solution I've used in the past for this scenario with Windows is to use boolean File.renameTo(File) and attempt to move the original file to a separate staging folder:
boolean success = potentiallyIncompleteFile.renameTo(stagingAreaFile);
If success is false, then the potentiallyIncompleteFile is still being written to.
This possible to do by using Apache Commons IO maven library FileUtils.copyFile() method. If you try to copy file and get IOException its means that file is not completely saved.
Example:
public static void copyAndDeleteFile(File file, String destinationFile) {
try {
FileUtils.copyFile(file, new File(fileDirectory));
} catch (IOException e) {
e.printStackTrace();
copyAndDeleteFile(file, fileDirectory, delayThreadPeriod);
}
Or periodically check with some delay size of folder that contains this file:
FileUtils.sizeOfDirectory(folder);
Even the number of bytes are equal, the content of the file may be different.
So I think, you have to match the old and the new file byte by byte.
2 options that seems to solve this issue:
the best option- writer process notify reading process somehow that
the writing was finished.
write the file to {id}.tmp, than when finish- rename it to {id}.java, and the reading process run only on *.java files. renaming taking much less time and the chance this 2 process work together decrease.
First, there's Why doesn't OS X lock files like windows does when copying to a Samba share? but that's variation of what you're already doing.
As far as reading arbitrary files and looking for sizes, some files have that information, some do not, but even those that do do not have any common way of representing it. You would need specific information of each format, and manage them each independently.
If you absolutely must act on the file the "instant" it's done, then your writing process would need to send some kind of notification. Otherwise, you're pretty much stuck polling the files, and reading the directory is quite cheap in terms of I/O compared to reading random blocks from random files.
One more method to test that a file is completely written:
private void waitUntilIsReadable(File file) throws InterruptedException {
boolean isReadable = false;
int loopsNumber = 1;
while (!isReadable && loopsNumber <= MAX_NUM_OF_WAITING_60) {
try (InputStream in = new BufferedInputStream(new FileInputStream(file))) {
log.trace("InputStream readable. Available: {}. File: '{}'",
in.available(), file.getAbsolutePath());
isReadable = true;
} catch (Exception e) {
log.trace("InputStream is not readable yet. File: '{}'", file.getAbsolutePath());
loopsNumber++;
TimeUnit.MILLISECONDS.sleep(1000);
}
}
}
Use this for Unix if you are transferring files using FTP or Winscp:
public static void isFileReady(File entry) throws Exception {
long realFileSize = entry.length();
long currentFileSize = 0;
do {
try (FileInputStream fis = new FileInputStream(entry);) {
currentFileSize = 0;
while (fis.available() > 0) {
byte[] b = new byte[1024];
int nResult = fis.read(b);
currentFileSize += nResult;
if (nResult == -1)
break;
}
} catch (Exception e) {
e.printStackTrace();
}
System.out.println("currentFileSize=" + currentFileSize + ", realFileSize=" + realFileSize);
} while (currentFileSize != realFileSize);
}
When I try to execute an external program from java I use this code below :
Process p;
rn = Runtime.getRuntime();
String[] unzip = new String[2];
unzip[0]="unzip";
unzip[1]=archive ;
public void dezip() throws IOException{
p = rn.exec(unzip);
int ret = p.exitValue();
System.out.println("End of unzip method");
But my last System.out is never executed, as if we exit from unzip method.
The unzip() call does only the half of the work, only a part of my archive is unzipped.
When I use ps -x or htop from command line I see that unzip process is still here.
Help please.
You probably need to read the InputStream from the process. See the javadoc of Process
Which states:
Because some native platforms only provide limited buffer size for
standard input and output streams, failure to promptly write the input
stream or read the output stream of the subprocess may cause the
subprocess to block, and even deadlock.
Check if the unzip command is prompting for something, perhaps a warning if the file already exists and if you want to overwrite it.
Also, is that a backquote I see in the middle of a java program?
Make sure external program doesn't wait for user input
Check if the the executable path is quoted when launching on Windows systems to handle directories with spaces or special characters.
PS.
I was using the java.lang.Runtime class but found that the java.lang.ProcessBuilder class is far superior. You can specify current working directory, and most importantly the system environment.
Please try the following:
p = rn.exec(unzip);
p.waitFor()
I hope it will change something.