I want to monitor a folder for newly created subfolders for specific .csv files using Java WatchService API (Windows 7 x64, Java 8). I expect the final application like this: folders\files will be created somewhere else and loaded into a cloud (e.g. GDrive). From there I will synchronize them with a local folder (using stock software). This local folder I want to monitor and process files within minutes after they appear. Also very important is not to miss any new files.
I use WatchService as in many tutorials and questions here, but still it behaves strangely.
When the folder (already with files) is created it is both CREATED and MODIFIED - thus two events for a folder. I can live with it, but it sometimes misses files in it completely. (I suppose, files are copied so fast, that they are there even before this new folder is registered and monitored).
When I copy several folders with files at once (F1,F2,F3) then it:
registers F1, detects modified files in F1, processes them, registers F2, registers F3.
Again, the files in the last two folders are not detected anyhow.
Here is a simplified code that I use. I am looking for a cause of this issues and how to make it more robust.
I can cut and paste all files from every new folder and hope then it will detect them, but this is the worst case solution.
More general question - what happens between ws.take() and ws.pollEvents() and ws.reset()? Are there blind-moments, when events are not registered?
public static void main(String args[]) {
// SET UP LOGGER AND CONFIG VARIABLES HERE
// CREATE AN OBJECT AND DEFINE AN ABSTRACT startListening()
WatchServiceClass wsc = new WatchServiceClass() {
public void startListening(WatchService watchService) {
while (true) {
WatchKey queuedKey = watchService.take();
// DOESNT HELP: Thread.sleep(1000);
List<WatchEvent<?>> events = queuedKey.pollEvents();
for (WatchEvent<?> watchEvent : events) {
if (watchEvent.kind() == StandardWatchEventKinds.OVERFLOW) {
continue;
}
String action // create, modify or delete
String fileName // self explanatory
String fullPath
String fullPathParent
if (action.equals("create")) { // TRACK NEW FOLDERS
registerDir(Paths.get(fullPath), watchService);
}
if (new File(fullPath).isDirectory()) {
continue; // NOT DOING ANYTHING WITH DIRECTORIES
}
// IF HERE, THEN THIS IS THE FILE, DO SOMETHING
// TAKES SEVERAL MINUTES
}
if (!queuedKey.reset()) {
keyPathMap.remove(queuedKey);
}
if (keyPathMap.isEmpty()) {
break;
}
}
}
};
// END OF startListening() METHOD DEFINITION
while (true) { // MAIN INFINITE LOOP
try (WatchService watchService = FileSystems.getDefault().newWatchService()) {
wsc.registerDir(Paths.get(DATA_DIR), watchService);
wsc.startListening(watchService);
} catch (Exception e) {
e.printStackTrace();
}
}
}
Related
I'm using multiple JVMs, but I need that each JVM to use a specific folder. What I'm trying to do is iterate through folders till it finds a file that is not locked and then lock it to use that specific folder.
Here I'm filtering the folders I want to use:
// Filter 'fran' folders
String dir = System.getProperty("user.dir");
FilenameFilter filter = new FilenameFilter() {
public boolean accept(File dir, String name) {
String lowercaseName = name.toLowerCase();
if (lowercaseName.startsWith("fran")) {
return true;
} else {
return false;
}
}
};
File[] dirs = new File(dir).listFiles(filter);
Then I'm trying to go through the folders and check if it is locked or not with f.canWrite(). However it always appears to use only one folder and ignore the others.
// Find available folder
boolean lock = true;
String lock_folder = "";
FileChannel fileChannel = null;
FileLock lockfile = null;
File f = null;
while (lock) {
for (File folder : dirs) {
f = new File(folder + "\\lock.txt");
Boolean isnotlocked = f.canWrite();
if (isnotlocked) {
fileChannel = new RandomAccessFile(f, "rw").getChannel();
lockfile = fileChannel.lock();
lock = false;
lock_folder = folder.getAbsolutePath();
break;
}
}
}
I+ve previously tried to accomplish what I needed without FileLock, creating a file in the specific folder and then deleting after completed. If the folder did not have that file it would create and lock that JVM. However I think the JVMs were getting mixed cause the results were bad.
Hope u can understand what my problem is, would really appreciate some help.
Here are some ideas:
Assuming a process class - CustomProcess.java. This runs in a separate thread. The class has a constructor which takes a file folder as an argument. Lets assume the filePath1 is the folder's path and is accepted from a FileChooser.
(a) How the application works:
Put the selected folder filePath1 in a collection like List<File> or List<Path> - lets call it processFilesList; this is shared (perhaps a static member) by all the processes (this needs to be a concurrent collection from java.util.concurrent package). This list tracks the folders which are already being processed. Before the process starts check if the filePath1 is already in the processFilesList.
(b) Create and start the process:
CustomProcess p1 = new CustomProcess(filePath1);
p1.startProcess(); // here the application does whatever with the files in the folder.
Option 2:
Put all the folder file paths that need to be processed in a Queue collection. Process each folder (and its files as needed) one at a time or by multiple processes. The queue can be a first-in-first-out (FIFO) or a last-in-first-out (LIFO). One can consider these concurrent queue implementations based on the requirement: ConcurrentLinkedQueue, LinkedBlockingQueue, ConcurrentLinkedDeque or LinkedBlockingDeque.
I want to determine if a new file or document is placed inside a specific folder/directory using java. For example, There are no files inside the "C:\Users\User\Documents" directory and then I downloaded a pdf file from the Internet and was placed on the mentioned directory. How can I determine if a new file is detected on the directory using java programming language? (It should also print-out the name of the directory and the new file name). Can I have any tips on how to create this kind of program using Java language? It should be continuous or in an infinite loop.
I tried this by using this:
package readfilesfromfolder;
import java.io.File;
public class ReadFilesFromFolder {
public static File folder = new File("C:/Documents and Settings/My Documents/Downloads");
static String temp = "";
public static void main(String[] args) {
// TODO Auto-generated method stub
System.out.println("Reading files under the folder "+ folder.getAbsolutePath());
listFilesForFolder(folder);
}
public static void listFilesForFolder(final File folder) {
for (final File fileEntry : folder.listFiles()) {
if (fileEntry.isDirectory()) {
listFilesForFolder(fileEntry);
} else {
if (fileEntry.isFile()) {
temp = fileEntry.getName();
if ((temp.substring(temp.lastIndexOf('.') + 1, temp.length()).toLowerCase()).equals("txt"))
System.out.println("File= " + folder.getAbsolutePath()+ "\\" + fileEntry.getName());
}
}
}
}
}
But based on the outcome, it just accessed the directory but did not list for any new items. Also, it is not yet in loop because I haven't placed it yet. Thank you :) (*Note: I am still new to Java programming :) *)
You could use the Watch Service. A watch service that watches registered objects for changes and events. For example a file manager may use a watch service to monitor a directory for changes so that it can update its display of the list of files when files are created or deleted.
A good example can be found here.
You too can use the Commons IO library from the Apache Foundation, mainly the org.apache.commons.io.monitor package.
Thank you guys for the tip! :) I found out how to do this using the WatchService :)
This is the output based on my research and reading :)
public static void main(String[] args) throws IOException{
// TODO code application logic here
WatchService watchService = FileSystems.getDefault().newWatchService();
//The path needed for changes
Path directory = Paths.get("C:\\Users\\User\\Documents");
//To determine whether a file is created, deleted or modified
//ENTRY_CREATE can be changed to ENTRY_MODIFY and ENTRY_DELETE
WatchKey watchKey = directory.register(watchService, StandardWatchEventKinds.ENTRY_CREATE);
//This portion is for the output of what file is created, modified, or deleted
while (true){
for (WatchEvent<?> event : watchKey.pollEvents()) {
System.out.println(event.kind());
Path file = directory.resolve((Path) event.context());
System.out.println(file);
}
}
}
Hope this can help other people. Thanks also to those who helped me as well as to the authors of different research materials used to create this one :) Credits to Mr.Kriechel for this one :)
So I'm trying to create a program in Unix that will take in a directory as a parameter and then recursively go through, open all of the folders, look through all of the files, and then delete all of the class files. I thought I was taking the correct steps as I was given code for a similar program and told to use it as a basis, but upon testing my program and I discover that nothing happens.
import java.io.File;
import java.nio.file.Files;
import java.nio.file.Path;
import java.nio.file.Paths;
import java.text.ParseException;
public class ClassFileDeleter {
public static void main(String[] args) throws ParseException {
String dirName = args[0];
deleteFile(dirName);
}
private static void deleteFile(String dirName) {
Path path = Paths.get(dirName);
File dir = path.toFile();
if(dir.exists()) {
File[] files = dir.listFiles();
if(dir.isDirectory()) {
for(File f:files) {
if(!f.isDirectory())
if(f.toString().endsWith(".class"))
System.out.println("yes");
else deleteFile(dirName + "/" + f.getName());
}}}
}}
I am at a loss at what I should do. I haven't attempted to delete anything yet because I don't want to delete anything that isn't a class file so I am using some dummy code that should print 'yes' once the program finds a class file. However when I run my code, absolutely nothing happens. I believe that there is either an issue with the way I am searching for class files (We are supposed to use endsWith) or with the way I am attempting to use recursion to look through all of the files in the specified directory. If I could have some assistance, that would be great.
I would start with a isFile check (and then test the extension of a file and log it if it matches), then you could recursively descend any directories. Something like,
private static void deleteFile(String dirName) {
File dir = new File(dirName);
if (dir.isFile()) {
if (dir.getName().endsWith(".class")) {
try {
System.out.println("Delete: " + dir.getCanonicalPath());
// dir.delete();
} catch (IOException e) {
e.printStackTrace();
}
}
} else if (dir.isDirectory()) {
File[] files = dir.listFiles();
for (File f : files) {
try {
deleteFile(f.getCanonicalPath());
} catch (IOException e) {
e.printStackTrace();
}
}
}
}
It strikes me that the code that you have to recurse through the directory, is creating a file object not a directory.
A quick google gave me this from the Oracle java tutorial (http://docs.oracle.com/javase/tutorial/essential/io/dirs.html#listdir).
Listing a Directory's Contents
You can list all the contents of a directory by using the newDirectoryStream(Path) method. This method returns an object that implements the DirectoryStream interface. The class that implements the DirectoryStream interface also implements Iterable, so you can iterate through the directory stream, reading all of the objects. This approach scales well to very large directories.
Remember: The returned DirectoryStream is a stream. If you are not using a try-with-resources statement, don't forget to close the stream in the finally block. The try-with-resources statement takes care of this for you.
The following code snippet shows how to print the contents of a directory:
Path dir = ...;
try (DirectoryStream<Path> stream = Files.newDirectoryStream(dir)) {
for (Path file: stream) {
System.out.println(file.getFileName());
}
} catch (IOException | DirectoryIteratorException x) {
// IOException can never be thrown by the iteration.
// In this snippet, it can only be thrown by newDirectoryStream.
System.err.println(x);
}
The Path objects returned by the iterator are the names of the entries resolved against the directory. So, if you are listing the contents of the /tmp directory, the entries are returned with the form /tmp/a, /tmp/b, and so on.
This method returns the entire contents of a directory: files, links, subdirectories, and hidden files. If you want to be more selective about the contents that are retrieved, you can use one of the other newDirectoryStream methods, as described later in this page.
Note that if there is an exception during directory iteration then DirectoryIteratorException is thrown with the IOException as the cause. Iterator methods cannot throw exception exceptions.
So I'd take a look there and see what you can work out.
When a directory monitored by a WatchService gets deleted, its parent directory does not immediately reflect the deletion in its File's listFiles method and cannot be deleted. Until the entire service is explicitly stopped the consequences for the parent appear to be:
The recommended recursive solution for deleting a non-empty directory failing.
deleteOnExit not being carried out on normal termination
Calls to delete returning false and having no effect on the filesystem.
To demonstrate, this test code:
import java.io.*;
import java.nio.file.*;
class DirectoryTester {
static WatchService watcher;
static {
try{watcher = FileSystems.getDefault().newWatchService();}
catch (IOException e) {e.printStackTrace();}
}
public static void main(String[] args) throws IOException {
String SEPARATE = System.getProperty("file.separator");
String testDirName = System.getProperty("user.dir") + SEPARATE + "testDir";
String subDirName = testDirName + SEPARATE + "subDir";
String fileName = subDirName + SEPARATE +"aFile";
create(fileName);
Paths.get(subDirName).register(watcher, StandardWatchEventKinds.ENTRY_DELETE);
delete(new File(testDirName));
}
static void create(String nameOfFile) throws IOException {
new File(nameOfFile).getParentFile().mkdirs();
Files.createFile(Paths.get(nameOfFile));
System.out.println("Created " + nameOfFile);
}
static void delete(File toDelete) throws IOException {
if (toDelete.isDirectory())
for (File c : toDelete.listFiles())
delete(c);
int numContainedFiles = toDelete.listFiles() != null ? toDelete.listFiles().length : 0;
if (!toDelete.delete()) {
System.out.println("Failed to delete " + toDelete + " containing " + numContainedFiles);
}
else {
System.out.println("Deleted " + toDelete + " containing " + numContainedFiles);
}
}
}
gives the following output on windows, which corresponds with testDir not being deleted on the filesystem.
Created C:\Dropbox\CodeSpace\JavaTestbed\src\testDir\subDir\aFile
Deleted C:\Dropbox\CodeSpace\JavaTestbed\src\testDir\subDir\aFile containing 0
Deleted C:\Dropbox\CodeSpace\JavaTestbed\src\testDir\subDir containing 0
Failed to delete C:\Dropbox\CodeSpace\JavaTestbed\src\testDir containing 1
If I put a breakpoint after the subDir deletion I can see that it has actually been deleted on the filesystem. Resuming from the breakpoint causes the last deletion to suceed, suggesting that this might be an issue with the visibility of changes made by the watch service thread. Does anyone know what is going on here, and if it is a bug? What I am actually trying to do is to delete directories that are monitored without stopping the monitoring on other directories, given that there does not appear to be an unregister path method provided by the API what are other standard Java ways of accomplishing this?
possibly related:
http://bugs.sun.com/view_bug.do?bug_id=6972833
The WatchService has an open handle to each watched directory. If a a watch directory is deleted then the WatchService closes the handle so that the directory entry can be removed from the parent directory. A problem arises for utilities and application that expect to be able to delete the parent directory immediately as it can take a few milliseconds for the watch service to get the notificationa and close the handle. If during that time that the tool attempts to delete the parent directory then it will fail. We don't have a solution to this issue at this time.
Specifically I am using lucene to perform full text searching and in certain scenarios the index file might become corrupted or simply has not been created yet at which point I would delete the file and rewrite the index to said file. My question pertains to the actual act of deleting and re-writing to a file in a multi threaded Java program.
Will synchronizing protect the IO File while its being deleted and restored? In other words will it block access to another thread coming along and attempting to use the same method and begin rewriting while its already in the process?
The setDirectory method needs to be run before any other methods in the class will work (it will throw errors otherwise), so does the way I have the synchronization setup protect me from any multi threaded mishaps?
When another thread attempts to use the setDirectory method and the buildCompleteIndex method is already in progress, will the thread simply wait for that to finish and then its check on whether the path exists will pass and it will move on to opening the index?
In lucene do I have to synchronize when writing, deleting, or searching the index or can these tasks be done concurrently?
public void setDirectory(int organizationId) throws IOException {
this.organizationId = organizationId;
File path = new File(INDEX_PATH + "/" + String.valueOf(organizationId));
//If path does not exist, create it and create new index for organization
synchronized(this) {
if(!path.exists()) {
path.mkdirs();
buildCompleteIndex(organizationId, false);
}
}
this.directory = FSDirectory.open(path); //Open directory
}
private void buildCompleteIndex(int organizationId, boolean rebuildDir) {
if(rebuildDir) {
File path = new File(INDEX_PATH + "/" + String.valueOf(organizationId));
try {
Utils.deleteDirectory(path);
} catch (IOException e) {
throw new LuceneIndexException("Error rebuilding index directory.", e);
}
path.mkdirs();
}
List<Tag> tagList = tagDAO.findAll(organizationId);
for(Tag tag : tagList) {
add(tag);
}
}