We Keep Coding

Java watch folder and action when all files and folders in watch folder have finished downloading

I am trying to write the facility to process media files using a watch folder. The Oracle example WatchDir demonstrates how to know when there are changes in a folder. However the issue with this is that I don't know when all the media has finished uploading. So for example when an SD card containing media which contains multiple files in different folders is dragged onto the watch folder, I need to be able process the media once all the files and sub folders are present. Media is not always just stored in single files but may have sidecar files so both sets of files need to be present in order to process the files correctly. Can anyone suggest how I can know that all files and sub folders have finished being copied into the watch folder?
This is my slightly modified version of the WatchDir to include logging:
public class WatchDir {
private final WatchService watcher;
private final Map<WatchKey,Path> keys;
private final boolean recursive;
private boolean trace = false;
#SuppressWarnings("unchecked")
static <T> WatchEvent<T> cast(WatchEvent<?> event) {
return (WatchEvent<T>)event;
}
/**
* Register the given directory with the WatchService
*/
private void register(Path dir) throws IOException {
WatchKey key = dir.register(watcher, ENTRY_CREATE, ENTRY_DELETE, ENTRY_MODIFY);
if (trace) {
Path prev = keys.get(key);
if (prev == null) {
System.out.format("register: %s%n", dir);
} else {
if (!dir.equals(prev)) {
System.out.format("update: %s -> %s%n", prev, dir);
}
}
}
keys.put(key, dir);
}
/**
* Register the given directory, and all its sub-directories, with the
* WatchService.
*/
private void registerAll(final Path start) throws IOException {
// register directory and sub-directories
Files.walkFileTree(start, new SimpleFileVisitor<Path>() {
#Override
public FileVisitResult preVisitDirectory(Path dir, BasicFileAttributes attrs)
throws IOException
{
register(dir);
return FileVisitResult.CONTINUE;
}
});
}
/**
* Creates a WatchService and registers the given directory
*/
WatchDir(Path dir, boolean recursive) throws IOException {
this.watcher = FileSystems.getDefault().newWatchService();
this.keys = new HashMap<WatchKey,Path>();
this.recursive = recursive;
if (recursive) {
System.out.format("Scanning %s ...\n", dir);
registerAll(dir);
System.out.println("Done.");
} else {
register(dir);
}
// enable trace after initial registration
this.trace = true;
}
/**
* Process all events for keys queued to the watcher
*/
void processEvents() {
System.out.println("process event");
boolean processing = false;
for (;;) {
System.out.println("loop");
// wait for key to be signalled
WatchKey key;
try {
processing = false;
System.out.println("about to take");
key = watcher.take();
processing = true;
} catch (InterruptedException x) {
System.out.println("take interrupted");
return;
}
Path dir = keys.get(key);
if (dir == null) {
System.err.println("WatchKey not recognized!!");
continue;
}
for (WatchEvent<?> event: key.pollEvents()) {
System.out.println("poll");
WatchEvent.Kind kind = event.kind();
// TBD - provide example of how OVERFLOW event is handled
if (kind == OVERFLOW) {
System.out.println("Overflow");
continue;
}
// Context for directory entry event is the file name of entry
WatchEvent<Path> ev = cast(event);
Path name = ev.context();
Path child = dir.resolve(name);
// print out event
System.out.format("%s: %s\n", event.kind().name(), child);
// if directory is created, and watching recursively, then
// register it and its sub-directories
if (recursive && (kind == ENTRY_CREATE)) {
try {
if (Files.isDirectory(child, NOFOLLOW_LINKS)) {
registerAll(child);
}
} catch (IOException x) {
// ignore to keep sample readable
System.out.println("ex: " + x.getMessage());
}
}
}
// reset key and remove from set if directory no longer accessible
boolean valid = key.reset();
if (!valid) {
keys.remove(key);
System.out.println("finished this set of files");
// all directories are inaccessible
if (keys.isEmpty()) {
break;
}
}
if (processing) {
System.out.println("processing files...");
} else {
System.out.println("not processing files");
}
System.out.println("End of loop\n\n");
}
}
static void usage() {
System.err.println("usage: java WatchDir [-r] dir");
System.exit(-1);
}
public static void main(String[] args) throws IOException {
// parse arguments
if (args.length == 0 || args.length > 2)
usage();
boolean recursive = false;
int dirArg = 0;
if (args[0].equals("-r")) {
if (args.length < 2)
usage();
recursive = true;
dirArg++;
}
// register directory and process its events
Path dir = Paths.get(args[dirArg]);
new WatchDir(dir, recursive).processEvents();
}
}

Maybe you could not do that. According to the doc, The WatchService only provided these events:
static WatchEvent.Kind<Path> ENTRY_DELETE Directory entry deleted.
static WatchEvent.Kind<Path> ENTRY_MODIFY Directory entry modified.
static WatchEvent.Kind<Object> OVERFLOW A special event to indicate that events may have been lost or discarded. ```
So you won't know whether there are new files to be create/copy to your directory.
Maybe there are some workarounds you could consider:
Set a timeout, for that time, if no new file is created, consider the file transfer is finished and start to do the work.
Let your application to handle the copy, so it could know the progress and trigger work after all file is copied.

Java WatchService stops watching folder if renamed (OSX)

I'm trying to setup a WatchService on an MacOS (High Sierra) and it seems to be working ok except when I rename dir. As an example I rename "test" to "hello" and here's what I get:
ENTRY_CREATE: /Users/david/Desktop/watchme/hello
update: /Users/david/Desktop/watchme/test -> /Users/david/Desktop/watchme/hello
ENTRY_DELETE: /Users/david/Desktop/watchme/test
I would expect to see:
register: /Users/david/Desktop/watchme/hello
But that isn't happening so when I modify anything inside of "hello", the only response I get is:
ENTRY_MODIFY: /Users/david/Desktop/watchme/hello
Nothing about the files or folders inside "hello"
I copied the following code right from Oracle with the exception of hard coding the attrs so I can run it in Idea:
import java.nio.file.*;
import static java.nio.file.StandardWatchEventKinds.*;
import static java.nio.file.LinkOption.*;
import java.nio.file.attribute.*;
import java.io.*;
import java.util.*;
/**
* Example to watch a directory (or tree) for changes to files.
*/
public class WatchDir {
private final WatchService watcher;
private final Map<WatchKey,Path> keys;
private final boolean recursive;
private boolean trace = false;
#SuppressWarnings("unchecked")
private static <T> WatchEvent<T> cast(WatchEvent<?> event) {
return (WatchEvent<T>)event;
}
/**
* Register the given directory with the WatchService
*/
private void register(Path dir) throws IOException {
WatchKey key = dir.register(watcher, ENTRY_CREATE, ENTRY_DELETE, ENTRY_MODIFY);
if (trace) {
Path prev = keys.get(key);
if (prev == null) {
System.out.format("register: %s\n", dir);
} else {
if (!dir.equals(prev)) {
System.out.format("update: %s -> %s\n", prev, dir);
}
}
}
keys.put(key, dir);
}
/**
* Register the given directory, and all its sub-directories, with the
* WatchService.
*/
private void registerAll(final Path start) throws IOException {
// register directory and sub-directories
Files.walkFileTree(start, new SimpleFileVisitor<Path>() {
#Override
public FileVisitResult preVisitDirectory(Path dir, BasicFileAttributes attrs)
throws IOException
{
register(dir);
return FileVisitResult.CONTINUE;
}
});
}
/**
* Creates a WatchService and registers the given directory
*/
private WatchDir(Path dir, boolean recursive) throws IOException {
this.watcher = FileSystems.getDefault().newWatchService();
this.keys = new HashMap<>();
this.recursive = recursive;
if (recursive) {
System.out.format("Scanning %s ...\n", dir);
registerAll(dir);
System.out.println("Done.");
} else {
register(dir);
}
// enable trace after initial registration
this.trace = true;
}
/**
* Process all events for keys queued to the watcher
*/
private void processEvents() {
for (;;) {
// wait for key to be signalled
WatchKey key;
try {
key = watcher.take();
} catch (InterruptedException x) {
return;
}
Path dir = keys.get(key);
if (dir == null) {
System.err.println("WatchKey not recognized!!");
continue;
}
for (WatchEvent<?> event: key.pollEvents()) {
WatchEvent.Kind kind = event.kind();
// TBD - provide example of how OVERFLOW event is handled
if (kind == OVERFLOW) {
continue;
}
// Context for directory entry event is the file name of entry
WatchEvent<Path> ev = cast(event);
Path name = ev.context();
Path child = dir.resolve(name);
// print out event
System.out.format("%s: %s\n", event.kind().name(), child);
if(child.toString().contains("testdir")) {
System.out.println("This is an update");
}
// if directory is created, and watching recursively, then
// register it and its sub-directories
if (recursive && (kind == ENTRY_CREATE)) {
try {
if (Files.isDirectory(child, NOFOLLOW_LINKS)) {
registerAll(child);
}
} catch (IOException x) {
// ignore to keep sample readbale
}
}
}
// reset key and remove from set if directory no longer accessible
boolean valid = key.reset();
if (!valid) {
keys.remove(key);
// all directories are inaccessible
if (keys.isEmpty()) {
break;
}
}
}
}
private static void usage() {
System.err.println("usage: java WatchDir [-r] dir");
System.exit(-1);
}
#SuppressWarnings("ParameterCanBeLocal")
public static void main(String[] args) throws IOException {
// parse arguments
args = new String[2]; // Added for debug
args[0] = "-r"; // Added for debug
args[1] = "/Users/david/Desktop/watchme"; // Added for debug
if (args.length == 0 || args.length > 2)
usage();
boolean recursive = false;
int dirArg = 0;
if (args[0].equals("-r")) {
if (args.length < 2)
usage();
recursive = true;
dirArg++;
}
// register directory and process its events
Path dir = Paths.get(args[dirArg]);
new WatchDir(dir, recursive).processEvents();
}
}

It's an interesting question. I tried on my own Mac and get the same result.
When we rename test to hello, OS first create hello and then delete test, both of watchme and test will get event. WatchKey of watchme folder gets ENTRY_CREATE and ENTRY_DELETE events.
In this step, though new dir name is hello, but after this code
WatchKey key = dir.register(watcher, ENTRY_CREATE, ENTRY_DELETE, ENTRY_MODIFY);
the key we get is still test's watch key, that's why we see update log in console.
ENTRY_CREATE: /Users/david/Desktop/watchme/hello
update: /Users/david/Desktop/watchme/test -> /Users/david/Desktop/watchme/hello
ENTRY_DELETE: /Users/david/Desktop/watchme/test
WatchKey of test also recieved an event. But this event is an invalid event, the key(Infact this is hello's watchKey now) will be removed from keys map. So the change of hello folder will show nothing in console.
Here is the the interesting thing. Why we get test's watchKey when input dir is hello. When I debug code, stop it at
WatchKey key = dir.register(watcher, ENTRY_CREATE, ENTRY_DELETE, ENTRY_MODIFY);
for a while when receive hello's ENTRY_CREATE event, and I get hello's key not test's key and register log not update log. It seems we can't see the hello folder at once when call dir.register method. I don't know more about what OS do when we rename a folder, hope others' reply.
This is my test code added logs in processEvents, you may try it yourself.
private void processEvents() {
for (; ; ) {
// wait for key to be signalled
WatchKey key;
try {
key = watcher.take();
} catch (InterruptedException x) {
return;
}
Path dir = keys.get(key);
if (dir == null) {
System.err.println("WatchKey not recognized!!");
continue;
} else {
System.out.println("CurrentDir=" + dir.getFileName() + " keySize=" + keys.size() + " valid=" + key.isValid());
}
for (WatchEvent<?> event : key.pollEvents()) {
WatchEvent.Kind kind = event.kind();
// TBD - provide example of how OVERFLOW event is handled
if (kind == OVERFLOW) {
continue;
}
// Context for directory entry event is the file name of entry
WatchEvent<Path> ev = cast(event);
Path name = ev.context();
Path child = dir.resolve(name);
// print out event
System.out.format("%s: %s\n", event.kind().name(), child);
// if directory is created, and watching recursively, then
// register it and its sub-directories
if (recursive && (kind == ENTRY_CREATE)) {
try {
if (Files.isDirectory(child, NOFOLLOW_LINKS)) {
registerAll(child);
}
} catch (IOException x) {
// ignore to keep sample readbale
}
}
}
// reset key and remove from set if directory no longer accessible
boolean valid = key.reset();
if (!valid) {
System.out.println("Remove From keys");
keys.remove(key);
// all directories are inaccessible
if (keys.isEmpty()) {
break;
}
}
}
}

WatchService fires ENTRY_MODIFY sometimes twice and sometimes once

I am using this WatchService example from Oracle:
import java.nio.file.*;
import static java.nio.file.StandardWatchEventKinds.*;
import static java.nio.file.LinkOption.*;
import java.nio.file.attribute.*;
import java.io.*;
import java.util.*;
public class WatchDir {
private final WatchService watcher;
private final Map<WatchKey,Path> keys;
private final boolean recursive;
private boolean trace = false;
#SuppressWarnings("unchecked")
static <T> WatchEvent<T> cast(WatchEvent<?> event) {
return (WatchEvent<T>)event;
}
/**
* Register the given directory with the WatchService
*/
private void register(Path dir) throws IOException {
WatchKey key = dir.register(watcher, ENTRY_CREATE, ENTRY_DELETE, ENTRY_MODIFY);
if (trace) {
Path prev = keys.get(key);
if (prev == null) {
System.out.format("register: %s\n", dir);
} else {
if (!dir.equals(prev)) {
System.out.format("update: %s -> %s\n", prev, dir);
}
}
}
keys.put(key, dir);
}
/**
* Register the given directory, and all its sub-directories, with the
* WatchService.
*/
private void registerAll(final Path start) throws IOException {
// register directory and sub-directories
Files.walkFileTree(start, new SimpleFileVisitor<Path>() {
#Override
public FileVisitResult preVisitDirectory(Path dir, BasicFileAttributes attrs)
throws IOException
{
register(dir);
return FileVisitResult.CONTINUE;
}
});
}
/**
* Creates a WatchService and registers the given directory
*/
WatchDir(Path dir, boolean recursive) throws IOException {
this.watcher = FileSystems.getDefault().newWatchService();
this.keys = new HashMap<WatchKey,Path>();
this.recursive = recursive;
if (recursive) {
System.out.format("Scanning %s ...\n", dir);
registerAll(dir);
System.out.println("Done.");
} else {
register(dir);
}
// enable trace after initial registration
this.trace = true;
}
/**
* Process all events for keys queued to the watcher
*/
void processEvents() {
for (;;) {
// wait for key to be signalled
WatchKey key;
try {
key = watcher.take();
} catch (InterruptedException x) {
return;
}
Path dir = keys.get(key);
if (dir == null) {
System.err.println("WatchKey not recognized!!");
continue;
}
for (WatchEvent<?> event: key.pollEvents()) {
WatchEvent.Kind kind = event.kind();
// TBD - provide example of how OVERFLOW event is handled
if (kind == OVERFLOW) {
continue;
}
// Context for directory entry event is the file name of entry
WatchEvent<Path> ev = cast(event);
Path name = ev.context();
Path child = dir.resolve(name);
// print out event
System.out.format("%s: %s\n", event.kind().name(), child);
// if directory is created, and watching recursively, then
// register it and its sub-directories
if (recursive && (kind == ENTRY_CREATE)) {
try {
if (Files.isDirectory(child, NOFOLLOW_LINKS)) {
registerAll(child);
}
} catch (IOException x) {
// ignore to keep sample readbale
}
}
}
// reset key and remove from set if directory no longer accessible
boolean valid = key.reset();
if (!valid) {
keys.remove(key);
// all directories are inaccessible
if (keys.isEmpty()) {
break;
}
}
}
}
static void usage() {
System.err.println("usage: java WatchDir [-r] dir");
System.exit(-1);
}
public static void main(String[] args) throws IOException {
// parse arguments
if (args.length == 0 || args.length > 2)
usage();
boolean recursive = false;
int dirArg = 0;
if (args[0].equals("-r")) {
if (args.length < 2)
usage();
recursive = true;
dirArg++;
}
// register directory and process its events
Path dir = Paths.get(args[dirArg]);
new WatchDir(dir, recursive).processEvents();
}
}
I am developing an app in Windows 7, and deployment environment is rhel 7.2. At first, in both OS'es, whenever I copied a file, it fired one ENTRY_CREATED and then two ENTRY_MODIFY. The first ENTRY_MODIFY was at the beginning of copying, and the second ENTRY_MODIFY was at the end of copying. So I was able to understand the copying process was over. However, It only fires one ENTRY_MODIFY in rhel 7.2 now. It still fires two ENTRY_MODIFY events in Windows 7 though.
I have found this in stackoverflow. That question asks why two ENTRY_MODIFY are fired. It is not exactly my question, but one of its answers disputes what I'm asking. Sadly, there is no solution to my question in that dispute though.
Because there is no ENTRY_MODIFY fired at the end but only in the beginning of copying, I can not understand when the copying is finished. What do you think might be the cause for this? Can it be fixed, how can I understand the copying is finished? I can't change rhel 7.2, but anything other than that I would gladly accept. Thanks in advance.

I just check if the file length is zero or not.
Here an example of
for (WatchEvent<?> event : key.pollEvents()) {
Path fileName = (Path) event.context();
if("tomcat-users.xml".equals(fileName.toString())) {
Path tomcatUsersXml = tomcatConf.resolve(fileName);
if(tomcatUsersXml.toFile().length() > 0) {
load(tomcatUsersXml);
}
}
}

You can use a DelayQueue to deduplicate the events.

Java Get notification if file is modified

I'm looking for a way to get a notification when a certain file is modified. I want to call a certain method when this happens, but in some cases I also want that the method is not called.
I tried the following:
class FileListener extends Thread {
private Node n;
private long timeStamp;
public FileListener(Node n) {
this.n = n;
this.timeStamp = n.getFile().lastModified();
}
private boolean isModified() {
long newStamp = n.getFile().lastModified();
if (newStamp != timeStamp) {
timeStamp = newStamp;
return true;
} else {
return false;
}
public void run() {
while(true) {
if (isModified()) {
n.setStatus(STATUS.MODIFIED);
}
try {
Thread.sleep(1000);
} catch(Exception e) {
e.printStackTrace();
}
}
}
The Node class contains a reference to the file, a STATUS (enum) and a reference to the FileListener of that file. When the file is modified, I want the status to change to STATUS.MODIFIED. But, in some cases, the file where the Node refers to changes to a new File and I don't want it to automatically change the status to Modified. In that case I tried this:
n.listener.interrupt(); //interrupts the listener
n.setListener(null); //sets listener to null
n.setFile(someNewFile); //Change the file in the node
//Introduce a new listener, which will look at the new file.
n.setListener(new FileListener(n));
n.listener.start(); // start the thread of the new listener
But what I get is an Exception thrown by 'Thread.sleep(1000)', because the sleep was interrupted and when I check the status, it is still modified to STATUS.MODIFIED.
Am I doing something wrong?

What about the watch service : http://docs.oracle.com/javase/7/docs/api/java/nio/file/WatchService.html?
WatchService watcher = FileSystems.getDefault().newWatchService();
Path dir = ...;
try {
WatchKey key = dir.register(watcher, ENTRY_MODIFY);
} catch (IOException x) {
System.err.println(x);
}
And then:
for (;;) {
//wait for key to be signaled
WatchKey key;
try {
key = watcher.take();
} catch (InterruptedException x) {
return;
}
for (WatchEvent<?> event: key.pollEvents()) {
WatchEvent.Kind<?> kind = event.kind();
if (kind == OVERFLOW) {
continue;
}
...
}

How to make sure all files uploaded via SFTP in watched directory are ok to use via Java7?

I'm using WatchService to monitor a directory. Another third party will upload large CSV files to that directory via SFTP. I have to wait until all the files are finished to start processing the files.
My trouble right now is that SFTP creates the file as soon as the uploading starts I get ENTRY_CREATE and continuously get ENTRY_MODIFY until the file is done. Is there anyway to tell if the file is actually done.
This is the code I use which I got it from Java Documentation
public class WatchDir {
private final WatchService watcher;
private final Map<WatchKey, Path> keys;
private final boolean recursive;
private boolean trace = false;
#SuppressWarnings("unchecked")
static <T> WatchEvent<T> cast(WatchEvent<?> event) {
return (WatchEvent<T>) event;
}
/**
* Register the given directory with the WatchService
*/
private void register(Path dir) throws IOException {
WatchKey key = dir.register(watcher, ENTRY_CREATE, ENTRY_DELETE, ENTRY_MODIFY);
if (trace) {
Path prev = keys.get(key);
if (prev == null) {
System.out.format("register: %s\n", dir);
} else {
if (!dir.equals(prev)) {
System.out.format("update: %s -> %s\n", prev, dir);
}
}
}
keys.put(key, dir);
}
/**
* Register the given directory, and all its sub-directories, with the
* WatchService.
*/
private void registerAll(final Path start) throws IOException {
// register directory and sub-directories
Files.walkFileTree(start, new SimpleFileVisitor<Path>() {
#Override
public FileVisitResult preVisitDirectory(Path dir, BasicFileAttributes attrs)
throws IOException {
register(dir);
return FileVisitResult.CONTINUE;
}
});
}
/**
* Creates a WatchService and registers the given directory
*/
WatchDir(Path dir, boolean recursive) throws IOException {
this.watcher = FileSystems.getDefault().newWatchService();
this.keys = new HashMap<WatchKey, Path>();
this.recursive = recursive;
if (recursive) {
System.out.format("Scanning %s ...\n", dir);
registerAll(dir);
System.out.println("Done.");
} else {
register(dir);
}
// enable trace after initial registration
this.trace = true;
}
/**
* Process all events for keys queued to the watcher
*/
void processEvents() {
for (; ; ) {
// wait for key to be signalled
WatchKey key;
try {
key = watcher.take();
} catch (InterruptedException x) {
return;
}
Path dir = keys.get(key);
if (dir == null) {
System.err.println("WatchKey not recognized!!");
continue;
}
for (WatchEvent<?> event : key.pollEvents()) {
WatchEvent.Kind kind = event.kind();
// TBD - provide example of how OVERFLOW event is handled
if (kind == OVERFLOW) {
continue;
}
// Context for directory entry event is the file name of entry
WatchEvent<Path> ev = cast(event);
Path name = ev.context();
Path child = dir.resolve(name);
// print out event
System.out.format("%s: %s\n", event.kind().name(), child);
// if directory is created, and watching recursively, then
// register it and its sub-directories
if (recursive && (kind == ENTRY_CREATE)) {
try {
if (Files.isDirectory(child, NOFOLLOW_LINKS)) {
registerAll(child);
}
} catch (IOException x) {
// ignore to keep sample readbale
}
}
}
// reset key and remove from set if directory no longer accessible
boolean valid = key.reset();
if (!valid) {
keys.remove(key);
// all directories are inaccessible
if (keys.isEmpty()) {
break;
}
}
}
}
static void usage() {
System.err.println("usage: java WatchDir [-r] dir");
System.exit(-1);
}
public static void main(String[] args) throws IOException {
// parse arguments
if (args.length == 0 || args.length > 2)
usage();
boolean recursive = false;
int dirArg = 0;
if (args[0].equals("-r")) {
if (args.length < 2)
usage();
recursive = true;
dirArg++;
}
// register directory and process its events
Path dir = Paths.get(args[dirArg]);
new WatchDir(dir, recursive).processEvents();
}
}

Under Linux, you can use the "inotify" tools. they probably arrive with all major destributions. here is wiki for it : wiki - Inotify
Note in the supported events list you have:
IN_CLOSE_WRITE - sent when a file opened for writing is closed
IN_CLOSE_NOWRITE - sent when a file opened not for writing is closed
these are what you look for. I did not manage to see something similar on windows.
Now as for using them, there can be various ways. I used a java library jnotify
Note that the library is cross platform, so you don't want to use the main class as windows does not support events for close file. you will want to use the linux API which exposes the full linux capabilities. just read the description page and you know its what you request : jnotify - linux
Note that in my case, I had to download the library source because I needed compile the shared object file "libjnotify.so" for 64 bit. the one provided worked only under 32 bit. Maybe they provide it now you can check.
check the examples for code and how to add and remove watches. just remember to use the "JNotify_linux" class instead of "JNotify" and then you can use a mask with your operation, for example.
private final int MASK = JNotify_linux.IN_CLOSE_WRITE;
I hope it will work for you.