I have a simple java program that creates a series of temporary files stored in a local tmp directory. I have added a simple shutdown hook that walks through all files and deletes them, then deletes the tmp directory, before exiting the program. here is the code:
Runtime.getRuntime().addShutdownHook(new Thread(new Runnable() {
#Override
public void run() {
File tmpDir = new File("tmp/");
for (File f : tmpDir.listFiles()) {
f.delete();
}
tmpDir.delete();
}
}));
My problem is that the thread that creates these files may not have terminated upon launch of the shutdown hook, and therefore, there may be a file created after listFiles() is called. this causes the tmp dir not to get deleted. I have come up with 2 hacks around this:
Hack # 1:
Runtime.getRuntime().addShutdownHook(new Thread(new Runnable() {
#Override
public void run() {
File tmpDir = new File("tmp/");
while (!tmp.delete()){
for (File f : tmpDir.listFiles()) {
f.delete();
}
}
}
}));
Hack # 2:
Runtime.getRuntime().addShutdownHook(new Thread(new Runnable() {
#Override
public void run() {
try{
Thread.sleep(1000);
} catch(InterruptedException e){
e.printStackTrace();
}
File tmpDir = new File("tmp/");
for (File f : tmpDir.listFiles()) {
f.delete();
}
tmpDir.delete();
}
}));
Neither is a particularly good solution. What would be ideal is to have the shutdown hook wait until all threads have terminated before continuing. Does anyone know if this can be done?
Just keep track of all your running threads and then.join() them before shutting down the program.
This is an answer to the question title as the ewok has said he can't use .deleteOnExit()
What Tyler said, but with a little more detail:
Keep references to the threads where the shutdown hook can access them.
Have the shutdown hook call interrupt on the threads.
Review the code of the threads to make sure they actually respond to interruption (instead of eating the InterruptedException and blundering on, which is typical of a lot of code). An interrupt should prompt the thread to stop looping or blocking, wrap up unfinished business, and terminate.
For each thread where you don't want to proceed until it finishes, check whether the thread is alive and if so call join on it, setting a timeout in case it doesn't finish in a reasonable time, in which case you can decide whether to delete the file or not.
UPDATE: Tyler Heiks accurately pointed out that deleteOnExit() isn't a valid solution since the OP tried it and it did not work. I am providing an alternate solution. It is again indirect, but mainly because the original design using threads and a ShutdownHook is fatally flawed.
Use finally blocks to delete the temp files.
Relying on ShutdownHooks for resource management is a very bad idea and makes the code very difficult to compose or reuse in a larger system. It's an even worse idea to hand resources from thread to thread. Resources like files and streams are among the most dangerous things to share between threads. There is likely very little to gain from this and it would make far more sense for each thread to independently obtain temp files using the library's createTempFile methods and manage their use and deletion using try/finally.
The convention for dealing with the temporary files on the system is to treat them as block boxes where:
location on disk is opaque (irrelevant to and not used directly by the program)
filename is irrelevant
filename is guaranteed to be mutually exclusive
The third above is very difficult to achieve if you hand-roll code to create and name temp files yourself. It is likely to be brittle and fail at the worst times (3AM pager anyone?).
The algorithm you present could delete files created by other processes that coincidentally share the same parent directory. That is unlikely to be a good thing for the stability of those other programs.
Here's the high-level process:
Get Path with Files.createTempFile() (or with legacy pre-Java 7 code File with File.createTempFile())
Use temp file however desired
Delete file
This is similar to InputStream or other resources that need to be manually managed.
That general pattern for explicit resource management (when AutoCloseable and try-with-resources aren't available) is as follows.
Resource r = allocateResource();
try {
useResource(r);
} finally {
releaseResource(r);
}
In the case of Path it looks like this:
Path tempDir = Paths.get("tmp/);
try {
Path p = Files.createTempFile(tempDir, "example", ".tmp");
try {
useTempFile(f);
} finally {
Files.delete(f);
}
} finally {
Files.delete(tempDir);
}
On pre-Java 7 legacy, the use with File looks like this:
File tempDir = new File("tmp/");
try {
File f = File.createTempFile(tempDir, "example", ".tmp");
try {
useTempFile(f);
} finally {
if (!f.delete()) {
handleFailureToDeleteTempFile(f);
}
}
} finally {
if (!tempDir.delete()) {
handleFailureToDeleteTempDir(tempDir);
}
}
Related
This code simply processes some files whenever they are present in a preset folder. But while running on Eclipse, the thread is terminated when there are no files in the folder, although files are generally added later on. To keep it running always, I have added an infinite while loop in the run function. This works, but is there any other way to achieve this so that it uses lesser memory?
Here's the code:-
import java.io.*;
public class AddService extends Thread{
public void run(){
while(true){
Handler obj=new Handler();
File[] listOfFiles=obj.getFileList();
if(listOfFiles.length>0)
{
obj.process(listOfFiles);
}
else{
try{
AddService.sleep(1000);
}
catch(Exception ex){
}
}
}
}
public static void main(String args[]){
AddService t1=new AddService();
t1.start();
}
}
You could do the same thing by ScheduledExecutorService
ScheduledExecutorService s = Executors.newScheduledThreadPool(1);
s.schedule(myrunnable, 1000, TimeUnit.SECONDS);
To check for the file if any present. Altrenatively if you are allowed to use third party libraries, you could observe for changes in directories using FileAlterationObserver
Apart from the approaches suggested by the other answers, you could also use the NIO WatchService provided by the JDK itself.
You can follow the tutorial here, which shows you how to monitor a directory for changes (new files added, existent files modified, or files deleted).
http://docs.oracle.com/javase/tutorial/essential/io/notification.html
So you could check if there are any files in the directory and if not, create a watcher that notifies you when there is a change.
A nice approach is to use a BlockingQueue (from the Java API).
One thread tries to take a File from the queue, waiting until one is provided.
One thread (or threads) are executed in a scheduled fashion trying to discover new files:
fileQueue = new LinkedBlockingQueue<>();
threadPool = Executors.newFixedThreadPool(1);
threadPool.submit(new Runnable() {
#Override
public void run() {
while (true)
{
//Block and wait for a file
File file = fileQueue.take();
// Do your stuff here
}
}
}
You can wrap this code in a loop, waiting for files and, as soon as they appear, put them in the Queue so the blocked thread can process them for you.
I have a situation where I have a large number of classes that need to do file (read only) access. This is part of a web app running on top of OSGI, so there will be a lot of concurrent needs to access.
So I'm building an OSGI service to access the file system for all the other pieces that will need it and provide a centralized access as this also simplifies configuration of file locations, etc.
It occurs to me that a multi-threaded approach makes the most sense along with a thread pool.
So the question is this:
If I do this and I have a service with an interface like:
FileService.getFileAsClass(class);
and the method getFileAsClass(class) looks kinda like this: (this is a sketch it may not be perfect java code)
public < T> T getFileAsClass(Class< T> clazz) {
Future<InputStream> classFuture = threadpool.submit(new Callable< InputStream>() {
/* initialization block */
{
//any setup from configs.
}
/* implement Callable */
public InputStream call() {
InputStream stream = //new inputstream from file location;
boolean giveUp = false;
while(null == stream && !giveUp) {
//Code that tries to read in the file 4
// times with a Thread.sleep() then gives up
// this is here t make sure we aren't busy updating file.
}
return stream;
}
});
//once we have the file, convert it and return it.
return InputStreamToClassConverter< T>.convert(classFuture.get());
}
Will that correctly wait until the relevant operation is done to call InputStreamtoClassConverter.convert?
This is my first time writing multithreaded java code so I'm not sure what I can expect for some of the behavior. I don't care about order of which threads complete, only that the file handling is handled async and once that file pull is done, then and only then is the Converter used.
I'm using the WatchService API to watch a directory, and getting ENTRY_CREATE events when a user starts copying a file into the directory. The files I'm working with can be large, though, and I'd like to know when the copy is finished. Is there any built in java API I can use to accomplish this, or am I best off to just keep track of the created files' size and start processing when the size stops growing?
EDIT: Here is my example code:
package com.example;
import java.io.File;
import java.nio.file.FileSystems;
import java.nio.file.Path;
import java.nio.file.StandardWatchEventKinds;
import java.nio.file.WatchEvent;
import java.nio.file.WatchKey;
import java.nio.file.WatchService;
public class Monitor {
public static void main(String[] args) {
try {
String path = args[0];
System.out.println(String.format( "Monitoring %s", path ));
WatchService watcher = FileSystems.getDefault().newWatchService();
Path watchPath = FileSystems.getDefault().getPath(path);
watchPath.register(watcher, StandardWatchEventKinds.ENTRY_CREATE, StandardWatchEventKinds.ENTRY_MODIFY);
while (true) {
WatchKey key = watcher.take();
for (WatchEvent<?> event: key.pollEvents()) {
Object context = event.context();
System.out.println( String.format( "Event %s, type %s", context, event.kind() ));
}
}
} catch (Exception e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}
}
Which Produces this output:
Monitoring /Users/ericcobb/Develop/watch
Event .DS_Store, type ENTRY_MODIFY
Event 7795dab5-71b1-4b78-952f-7e15a2f39801-84f3e5daeca9435aa886fbebf7f8bd61_4.mp4, type ENTRY_CREATE
When an entry is created, you will get an ENTRY_CREATE event. For every subsequent modification, you will get an ENTRY_MODIFY event. When copying is completed, you will be notified with an ENTRY_MODIFY.
The only valid option to ensure, no other process is writing to the file anymore is to successfully obtain a Write lock on the file. Although a bit clumsy, you could use FileChannel and FileLock for that purpose
try(FileChannel ch = FileChannel.open(p, StandardOpenOption.WRITE);
FileLock lock = ch.tryLock()){
if(lock == null) {
//no lock, other process is still writing
} else {
//you got a lock, other process is done writing
}
} catch (IOException e) {
//something else went wrong
}
You could as well obtain a lock using ch.lock() which is a blocking call. But since ENTRY_MODIFY events are continously generated when the file is being written to, you can as well use tryLock and wait for the next event.
I think that the most common way to accomplish this is to copy the file into a temporary directory on the same partition and then move it to the monitored directory by an atomic move command.
Perhaps a solution could also be achieved by playing around with the files time stamps
However both of these solutions need to be realized in the writing application. I think you cannot tell surely from the reading application.
This is the library that we use today:
https://github.com/levelsbeyond/jpoller
It's a fork of an open-source project that the original author no longer maintains. We've fixed a few bugs in it, and today use it in production.
It works by checking the size of files in a directory, and considers the file done (ready for processing) once the file size stops growing for a period of time.
If you have the ability to control the file writes, you can write to a temp file, then rename it to the target. That should generate a single ENTRY_MODIFY.
Something like:
Path tempFile = Files.createTempFile("tmp", "tmp");
writeSomeContents(tempFile);
Path target = Paths.get(targetFilename);
Files.move(tempFile, target);
I recommend waiting a few MS after receiving the notification, to wait for the move to complete.
Maybe you can tell by trying to acquire write access to the file, but this depends on two factors and requires research:
The writing application must acquire exclusive write permission to the file and mustn’t close the file before it is in fact complete.
It must be possible to test for this from Java. I would give it a try whether opening the file as FileChannel for WRITE access throws an exception if the file is still locked for writing by another program. Unluckily, the Javadoc doesn’t go into detail about this.
File file = child.toAbsolutePath().toFile();
if(file.isFile() && file.isWrite())
{
}
maybe this would be the way to find out what you want but not in the while statement.
It will work when you really access to file.
I have a write method that is supposed to safely write data to a file.
// The current file I am writing to.
FileOutputStream file = null;
...
// Synchronized version.
private void write(byte[] bytes) {
if (file != null && file.getChannel() != null) {
try {
boolean written = false;
do {
try {
// Lock it!
FileLock lock = file.getChannel().lock();
try {
// Write the bytes.
file.write(bytes);
written = true;
} finally {
// Release the lock.
lock.release();
}
} catch (OverlappingFileLockException ofle) {
try {
// Wait a bit
Thread.sleep(0);
} catch (InterruptedException ex) {
throw new InterruptedIOException("Interrupted waiting for a file lock.");
}
}
} while (!written);
} catch (IOException ex) {
log.warn("Failed to lock " + fileName, ex);
}
} else {
log.warn("Failing - " + (file == null ? "file" : "channel") + " is null!!");
}
}
It has worked fine for me for a while now, although I know there are some wrinkles in it.
I have recently changed a project that uses this code to build and run under Java 5 (from Java 6) and now it looks like it is deadlocked awaiting a lock on the file. It is a multithreaded app and it is quite possible for several threads to attempt to write to the same file.
The debugger tells me that the hung threads are waiting for the FileLock lock = file.getChannel().lock() call to return.
Some research brought up this interesting little nugget which mentions:
File locks are held on behalf of the entire Java virtual machine. They are not suitable for controlling access to a file by multiple threads within the same virtual machine.
So am I doing it wrong? If so what is the right way? If I am doing it right how come I hit a deadlock?
Added: Forgot to mention - each thread holds its own copy of this object so there should not be any synchronisation issues within the code. I felt safe to rely on the FileChannel.lock() method to ensure writes do not interleave.
Added too: I have indeed solved the issue using various synchronized mechanisms. I do, however, have outstanding questions:
Why is FileLock lock = file.getChannel().lock(); not suitable ...?
Why did my issues only appear when switching back to Java-5 when everything worked fine with Java-6?
FileLock is only for interprocess locking, javadoc reads:
"File locks are held on behalf of the entire Java virtual machine.
They are not suitable for controlling access to a file by multiple
threads within the same virtual machine."
To lock between java threads (same JVM) you need to use some shared lock. I would suggest within the file writing class to use a synchronized block (which according to these articles is likely to perform best):
final Object lock = new Object();
public void write(...){
synchronized(lock){
// do writing
}
}
Another approach is to use a ReentrantLock and then use the proven idiom of
final ReentrantLock lock = new ReentrantLock();
public void write(...){
try {
lock.lock()
// do the writing
} finally {
// forget this and you're screwed
lock.unlock();
}
}
you may need to implement critical section concept on the actual code using the file, rather than the hashmap. You can either create a synchronized block or separate the file access code into a separate procedure and make that method synchronized.
Essentially, only one thread executes a synchronized block at a time. It gives you the exclusive access you need.
Another way of doing it is, to use a serial thread Executor, depending on your functional requirements.
You may want to look at this thread:
Howto synchronize file access in a shared folder using Java (OR: ReadWriteLock on network level)
Edit: It seems my test to determine whether the original JVM had exited was flawed to begin with (see comments on accepted answer). Sorry for the noise.
I have a need to have a running JVM start another JVM and then exit. I'm currently trying to do this via Runtime.getRuntime().exec(). The other JVM starts, but my original JVM won't exit until the "child" JVM process stops. It appears that using Runtime.getRuntime().exec() creates a parent-child relationship between the processes. Is there some way to de-couple the spawned process so that the parent can die, or some other mechanism to spawn a process without any relationship to the creating process?
Note that this seems exactly like this question: Using Java to spawn a process and keep it running after parent quits but the accepted answer there doesn't actually work, at least not on my system (Windows 7, Java 5 and 6). It seems that maybe this is a platform-dependent behavior. I'm looking for a platform independent way to reliably invoke the other process and let my original process die.
For example, suppose I have a jar file at C:\myjar.jar and I want to run the class com.example.RunMe that lives in that jar. Lets say that class pops up a JOptionPane, and then exits once the user has hit OK.
Now, the following is the program running in JVM #1:
public static void main(String[] args) {
String javaHome = System.getProperty("java.home");
String os = System.getProperty("os.name");
String javawBin = javaHome + File.separator + "bin" + File.separator + "javaw";
if (os.toLowerCase().contains("win")) {
javawBin += ".exe";
}
List<String> cmd = new ArrayList<String>();
cmd.add("\"" + javawBin + "\"");
cmd.add("-cp");
cmd.add("\"C:\\myjar.jar\"");
cmd.add("com.example.RunMe");
System.out.println("Running: " + cmd);
try {
System.out.println("Launching...");
Process p = Runtime.getRuntime().exec(cmd.toArray(new String[cmd.size()]));
new Thread(new StreamGobbler(p.getInputStream())).start();
new Thread(new StreamGobbler(p.getErrorStream())).start();
System.out.println("Launched JVM.");
System.exit(0);
} catch (IOException e) {
e.printStackTrace();
}
}
private static class StreamGobbler implements Runnable {
InputStream stream;
StreamGobbler(InputStream stream) {
this.stream = stream;
}
public void run() {
byte[] buf = new byte[64];
try {
while (stream.read(buf) != -1)
;
} catch (IOException e) {
}
}
}
The observed behavior is that both "Launching..." and "Launched JVM." are printed, but JVM #1 only exits after you hit OK in the JOptionPane launched by JVM #2. Also - the behavior is the same whether or not you start up the stream gobbler threads or not.
Also, to save someone the breath, yes I know I could create a new URLClassLoader with that jar file and run it that way, but thats not what I'm trying to do here.
I just tried the following code, and I see processes being spawned and main one exiting on Vista and Java 6. I think something else might be going on with your code.
public class Test {
public static void main(String[] args) throws Exception {
if(args.length == 0)
Runtime.getRuntime().exec("javaw Test loop");
else
while(true){}
}
}
As far as I know, killing a process fairly often kills all child processes. I doubt there's a platform independent way to do this.
Windows doesn't establish the same kind of parent-child relationship between processes that Unix systems do. It is likely that your parent process isn't exiting because there's a thread still running in it. This thread may be waiting for the child process to terminate, which could explain why your parent exits when the child exits.
Your threads running StreamGobblers are within Process #1, and are not daemon threads, so Process #1 doesn't end till those threads complete, when the Streams they are gobbling go away as Process #2 ends.
Take out the two lines that create those threads.