I need to monitor changes( uploaded files ) in two directories in a web application. I created a ServletContextListener that triggers the monitoring of these two directories.
My problem is that when the first monitoring starts, the thread is blocked and the second monitoring does not start.
Is possible to keep the monitoring of two different folders running in parallel and background?
I know that the problem is due to an infinite loop , but do not know how to put this thread in the background. Any help will be appreciated . Thank you very much in advance
ContextListener
#Override
public void contextInitialized(ServletContextEvent event) {
Path pathFolder1 = Paths.get("my_folder_1_path");
MyWatcher watcher1 = new MyWatcher();
Path pathFolder2 = Paths.get("my_folder_2_path");
MyWatcher watcher2 = new MyWatcher();
watcher1.startMonitoring(pathFolder1);
watcher2.startMonitoring(pathFolder2);
}
MyWatcher
public void startMonitoring(Path directory) {
try {
FileSystem fs = directory.getFileSystem ();
WatchService watcher = fs.newWatchService();
while(true) {
directory.register(watcher, StandardWatchEventKinds.ENTRY_CREATE, StandardWatchEventKinds.ENTRY_DELETE);
WatchKey watckKey = watcher.take();
List<WatchEvent<?>> events = watckKey.pollEvents();
for (WatchEvent event : events) {
if (event.kind() == StandardWatchEventKinds.ENTRY_CREATE) {
System.out.println("File created: " + event.context().toString());
}
if (event.kind() == StandardWatchEventKinds.ENTRY_DELETE) {
System.out.println("File removed: " + event.context().toString());
}
}
watckKey.reset();
}
} catch (IOException e) {
e.printStackTrace();
} catch (InterruptedException e) {
e.printStackTrace();
}
}
You should configure a thread factory on the app server, inject it through a #Resource annotation, and use threads from there. An googled example would be a blog entry about this.
Related
I'm using Java's WatchService API within my Spring Boot application to monitor a directory, and perform some actions on created files. This process is executed asynchronously: it starts automatically right after the application is ready and monitors the directory in the background until the application is stopped.
This is the configuration class:
#Configuration
public class DirectoryWatcherConfig {
#Value("${path}")
private String path;
#Bean
public WatchService watchService() throws IOException {
WatchService watchService = FileSystems.getDefault().newWatchService();
Path directoryPath = Paths.get(path);
directoryPath.register(watchService, StandardWatchEventKinds.ENTRY_CREATE);
return watchService;
}
}
And this is the monitoring service:
#Service
#RequiredArgsConstructor
public class DirectoryWatcherService {
private final WatchService watchService;
#Async
#EventListener(ApplicationReadyEvent.class)
public void startWatching() throws InterruptedException {
WatchKey key;
while ((key = watchService.take()) != null) {
for (WatchEvent<?> event : key.pollEvents()) {
// actions on created files
}
key.reset();
}
}
}
This code is working as expected, with the following exception, which I'd like to fix:
Any failure during the execution makes the monitoring to stop (obviously), and I don't know how to restart the monitoring after such events occur
You should change the loop with and add a try/Catch, in the catch restarting the service. As you commented you need to keep in even if is interrupted , so you will need the use of ExecutorService. Declare the Executor out of the method
#Autowired
private ExecutorService executor;
and inside the method something similar to my last answer but using the executor
Runnable task = () -> {
while (true) {
try {
WatchKey key = watchService.take();
if (key != null) {
for (WatchEvent<?> event : key.pollEvents()) {
// Perform actions on created files here
}
key.reset();
}
} catch (Exception e) {
// Wait for some time before starting the thread again
Thread.sleep(5000);
}
}
};
//submit the task to the executor
executor.submit(task);
Actually the solution was quite simple. Wrapping the desired actions with try/catch (catching desired exceptions) in DirectoryWatcherService like this allows the thread to continue monitoring directories:
#Service
#RequiredArgsConstructor
public class DirectoryWatcherService {
private final WatchService watchService;
#Async
#EventListener(ApplicationReadyEvent.class)
public void startWatching() throws InterruptedException {
WatchKey key;
while ((key = watchService.take()) != null) {
for (WatchEvent<?> event : key.pollEvents()) {
try {
// actions on created files
} catch (RuntimeException ex) {
// log exception or whatever you choose, as long as execution continues
}
}
key.reset();
}
}
}
I have a class that looks in a specific folder for new image files that have been moved there. How can I return the names of these files back to my main class so that I can manipulate the images once they've been moved into the folder?
In my main class I have
Thread thread = new Thread(new DirectoryWatcher(directory.toPath()));
thread.start();
And the DirectoryWatcher class
public class DirectoryWatcher implements Runnable{
private Path path;
private FileSystem fs;
private WatchKey watchKey;
private Path newFile;
private static final Pattern pattern = Pattern.compile("[A-Za-z0-9-_\\s()]+.(jpg|png|bmp|JPG|PNG|BMP)");
private Matcher matcher;
private String patternMatcher;
public DirectoryWatcher(Path path){
this.path = path;
fs = this.path.getFileSystem();
watchKey = null;
}
#Override
public void run() {
try{
Boolean isFolder = (boolean) Files.getAttribute(path, "basic:isDirectory", NOFOLLOW_LINKS);
if(!isFolder) {
throw new IllegalArgumentException("Path: " + path + " is not a folder");
}
} catch (IOException e){
e.printStackTrace();
}
System.out.println("Watching directory: " + path);
try(WatchService service = fs.newWatchService()) {
path.register(service, ENTRY_CREATE);
//WatchKey watchKey = null;
while(true) {
watchKey = service.take();
Kind<?> kind = null;
for(WatchEvent<?> watchEvent : watchKey.pollEvents()){
kind = watchEvent.kind();
if(OVERFLOW == kind) {
continue;
} else if (ENTRY_CREATE == kind){
newFile = ((WatchEvent<Path>) watchEvent).context();
patternMatcher = newFile.toString();
matcher = pattern.matcher(patternMatcher);
//
//If the file is an image
if(matcher.matches()){
//Return image to main class
}
}
}
if (!watchKey.reset()){
break;
}
}
} catch (IOException e) {
e.printStackTrace();
} catch (InterruptedException e){
e.printStackTrace();
}
}
}
Hum, I'm not sure but I think you should manipulate the image either from the DirectoryWatcher thread itself, or from other threads (Callable) created by the DirectoryWatcher thread.
In that last scenario, the DirectoryWatcher would start a thread(callable) whenever a new image is detected in the folder. Each created threads would just do their job with the new file and then terminates. To make that I suggest you take a look at ExecutorService, Callable and Future. With that api you will have no thread synchonization to worry about.
Otherwise, if you really want the main thread to do something each time the directory watcher finds a new image, your main thread would have to first create a blocking collection, like ArrayBlockingQueue, pass it to the DirectoryWatcher thread before starting it.
The DirectoryWatcher thread would then add file names to the queue whenever it needs to, and the main thread could loop forever, calling the take() method of the blocking queue.
Have in mind that if you do that, the main thread could do nothing else than listen to filenames returned from the DirectoryWatcher.
And since there would be only one thread executing for all file found, it would be less performant than the solution with the ExecutorService,Callable,Future proposed above in the case more than one image are copied in the folder at the same time.
I'm working on an application that uses the Java watchservice (Java 8) under Linux Mint. One interesting problem I am encountering is running out of inotify watches.
I'm developing under Eclipse and the behavior is as follows:
When the app starts, it recurses a directory structure, putting a watch on each directory found. The current test case uses 13,660 paths. My maximum is set to 16384.
If I stop and restart the app several (20+ times), it seems to function normally. Eventually, however, I will get a cascade of system errors indicating the maximum number of watches has been reached. However, if I restart Eclipse, the issue goes away.
Obviously, the Watch Service isn't releasing all of it's resources, but of the 13,660 watches it acquires, only a few (I'm guessing less than a hundred) are retained. It appears they aren't released unless I shut down Eclipse's Java instance and restart it.
To address this, I've ensured the watch service's close method is called when the application shuts down and the watch service task is cancelled.
The only other thing that I'm doing differently is I'm running two separate watch services for two different purposes. I'm told that you shouldn't need to run more than one, and perhaps this is the problem, but I'd rather not run one watch service if I can help it.
That said, are there any thoughts or suggestions on how I might be able to determine the cause of this bug?
Apologies for the massive code posting. This is my implementation of the WatchService class.
A few notes:
The pathFinder runs in a separate thread and is just a file visitor - walking the directory tree and returning paths to all dirs / files found.
Register is called only when changes are posted to the pathsChanged property (from the pathFinder's onSucceeded callback).
The pathsChanged property is always updated by a setAll() call. It only posts the latest changes and is not meant to be cumulative. Beyond the watchservice, other classes listen to these properties and respond accordingly.
public final class LocalWatchService extends BaseTask {
private final static String TAG = "LocalWatchService";
//watch service task
private WatchService watcher;
//path finding task and associated executor
private LocalPathFinder finder;
//root path where the watch service begins
private final Path mRootPath;
private final ExecutorService pathFinderExecutor =
createExecutor ("pathFinder", false);
//class hash map which keys watched paths to generated watch keys
private final Map<WatchKey, Path> keys = new HashMap<WatchKey, Path>();
//reference to model property of watched paths.
private final SimpleListProperty <SyncPath> mChangedPaths =
new SimpleListProperty <SyncPath>
(FXCollections.<SyncPath> observableArrayList());
public LocalWatchService (String rootPath) {
super ();
mRootPath = Paths.get(rootPath);
//create the watch service
try {
this.watcher = FileSystems.getDefault().newWatchService();
} catch (IOException e) {
e.printStackTrace();
}
setOnCancelled(new EventHandler() {
#Override
public void handle(Event arg0) {
pathFinderExecutor.shutdown();
}
});
mChangedPaths.addListener(new ListChangeListener <SyncPath> (){
#Override
public void onChanged(
javafx.collections.ListChangeListener.Change<? extends SyncPath>
arg0) {
for (SyncPath path: arg0.getList()) {
//call register only when a directory is found
if (path.getFile() == null) {
try {
register (path.getPath());
} catch (IOException e) {
e.printStackTrace();
} catch (InterruptedException e) {
Thread.currentThread().interrupt();
}
}
}
}
});
};
public SimpleListProperty<SyncPath> changedPaths() { return mChangedPaths; }
public void initializeWatchPaths() {
ArrayList <Path> paths = new ArrayList <Path> ();
//create a DirectoryStream filter that finds only directories
//and symlinks
DirectoryStream.Filter<Path> filter =
new DirectoryStream.Filter<Path>() {
public boolean accept(Path file) throws IOException {
return (Files.isDirectory(file) ||
Files.isSymbolicLink(file));
}
};
//apply the filter to a directory stream opened on the root path
//and save everything returned.
paths.addAll(utils.getFiles(mRootPath, filter));
runPathFinder (paths);
}
private void runPathFinder (ArrayList <Path> paths) {
//need to add blocking code / mechanism in case a path finder is
//currently running (rare case)
finder = new LocalPathFinder();
finder.setPaths (paths);
//callbacks on successful completion of pathfinder
EventHandler <WorkerStateEvent> eh =
new EventHandler <WorkerStateEvent> () {
ArrayList <SyncPath> paths = new ArrayList <SyncPath>();
#Override
public void handle(WorkerStateEvent arg0) {
for (Path p: finder.getPaths()) {
paths.add(
new SyncPath(mRootPath, p, SyncType.SYNC_NONE));
}
addPaths(paths);
}
};
finder.setOnSucceeded(eh);
pathFinderExecutor.execute (finder);
}
private void addPath(Path path, SyncType syncType) {
mChangedPaths.setAll(new SyncPath(mRootPath, path, syncType));
}
private void addPaths(ArrayList<SyncPath> paths) {
mChangedPaths.setAll(paths);
}
/**
* Register the given directory with the WatchService
* #throws InterruptedException
*/
public final void register(Path dir)
throws IOException, InterruptedException {
//register the key with the watch service
WatchKey key =
dir.register (watcher, ENTRY_CREATE, ENTRY_DELETE, ENTRY_MODIFY);
if (!keys.isEmpty()) {
Path prev = keys.get(key);
if (prev == null) {
//This is a new key
}
else if (!dir.equals(prev)) {
//This is an update
}
}
keys.put(key, dir);
}
private void processWatchEvent (WatchKey key, Path dir) throws IOException, InterruptedException {
for (WatchEvent<?> event: key.pollEvents()) {
WatchEvent.Kind kind = event.kind();
// TBD - provide example of how OVERFLOW event is handled
if (kind == OVERFLOW) {
System.out.println ("Overflow encountered");
}
WatchEvent<Path> ev = (WatchEvent<Path>)event;
Path target = dir.resolve(ev.context());
if (kind == ENTRY_DELETE) {
ArrayList <Path> finderList = new ArrayList <Path> ();
if (Files.isDirectory(target)) {
//directory deletion is not implemented apart from
//file deletion
}
else
addPath (target, SyncType.SYNC_DELETE);
} else if (kind == ENTRY_CREATE) {
/*
* Added paths are passed to the pathfinder service for
* subdirectory discovery. Path and subpaths are then added
* to the AddedPaths property via an event listener on
* service's onSucceeded() event.
*
* Added files are added directly to the AddedPaths property
*/
ArrayList <Path> finderList = new ArrayList <Path> ();
if (Files.isDirectory(target)) {
finderList.add (target);
runPathFinder (finderList);
}
//add files directly to the addedPaths property
else {
//a newly created file may not be immediately readable
if (Files.isReadable(target)) {
addPath (target, SyncType.SYNC_CREATE);
}
else
System.err.println ("File " + target + " cannot be read");
}
} else if (kind == ENTRY_MODIFY) {
System.out.println ("File modified: " + target.toString());
}
boolean valid = key.reset();
if (!valid)
break;
}
}
#SuppressWarnings("unchecked")
<T> WatchEvent<T> cast(WatchEvent<?> event) {
return (WatchEvent<T>)event;
}
#Override
protected Void call () throws IOException, InterruptedException {
boolean interrupted = false;
register (mRootPath);
initializeWatchPaths();
try {
// enter watch cycle
while (!interrupted) {
//watch for a key change. Thread blocks until a change occurs
WatchKey key = null;
interrupted = isCancelled();
//thread blocks until a key change occurs
// (whether a new path is processed by finder or a watched item changes otherwise)
try {
key = watcher.take();
} catch (InterruptedException e) {
interrupted = true;
try {
watcher.close();
} catch (IOException e1) {
// TODO Auto-generated catch block
e1.printStackTrace();
}
// fall through and retry
}
Path dir = keys.get (key);
if (dir == null) {
System.out.println ("Null directory key encountered.");
continue;
}
//process key change once it occurs
processWatchEvent(key, dir);
// reset key and remove from set if directory no longer accessible
if (!key.reset()) {
keys.remove(key);
// all directories are inaccessible
if (keys.isEmpty())
break;
}
}
} finally {
if (interrupted)
Thread.currentThread().interrupt();
}
keys.clear();
watcher.close();
return null;
};
}
I am trying a program with many phidget rfid readers. This test code works fine and I can load up all the readers and have it worked.
Vector phidgetList = manager.getPhidgets();
Enumeration phidgetListEnum = phidgetList.elements();
int count=phidgets.size();
while(phidgetListEnum.hasMoreElements()) {
Phidget phidgetElement = (Phidget) phidgetListEnum
.nextElement();
if (!phidgets.containsKey(phidgetElement.getSerialNumber())) {
RFIDTracking rfi = (RFIDTracking) ct.getTracking("rfid")
.clone();
rfi.setName("rfid clone " + count++);
rfi.detect();
rfi.setCode(phidgetElement.getSerialNumber());
phidgets.put(phidgetElement.getSerialNumber(), rfi);
Thread t = new Thread(rfi);
t.start();
}
}
The problem is when I tried to detect the new readers attached or detached from the system. I used this code
Manager manager;
manager = new Manager();
try {
manager.addAttachListener(new AttachListener() {
public void attached(AttachEvent ae)
{
try
{
System.out.println("attached" + ((RFIDPhidget)ae.getSource()).getSerialNumber());
}
catch (PhidgetException ex) { }
}
});
manager.open();
} catch (PhidgetException exception) {
System.err.println(exception.getErrorNumber()+ exception.getDescription());
}
// Allow the Phidgets time to attach
Thread.sleep(1000);
This code could not detect any reader attachment. I found there is no waitForAttachment(time) in the manager. May I know how to solve this. Thank you in advanced
It's Phidget, but not RFIDPhidget. There is no WaitForAttachment in the manager class because it is not necessary. The previous code works fine, but the wait time must be a little bit longer and the program won't terminate before something is attached.
Manager manager;
manager = new Manager();
try {
manager.addAttachListener(new AttachListener() {
public void attached(AttachEvent ae)
{
try
{
System.out.println("attached" + ((Phidget)ae.getSource()).getSerialNumber());
} catch (PhidgetException ex) { }
}
});
manager.open();
} catch (PhidgetException exception) {
System.err.println(exception.getErrorNumber()+ exception.getDescription());
}
// Allow the Phidgets time to attach
Thread.sleep(1000);
This question already has answers here:
How to run a background task in a servlet based web application?
(5 answers)
Closed 7 years ago.
EDIT:
The current code is the working solution, the one which does not block the application, it
incorporates the suggestion made in the approved answer.
I want a background thread to download an MS Access database continuously, while my tomcat 7 web application is running, the thread does download the database, however it seems to block my application's startup as I'm unable to access any page from the service, this is the code that I'm using:
public class DatabaseUpdater implements ServletContextListener {
private Thread thread = null;
private final Runnable updater = new Runnable() {
private boolean hasExpired(File mdbFile) throws IOException {
if (!mdbFile.exists())
return true;
Long ttl = Long.parseLong(Configuration.getValueForOS("db.http-expiration"));
Date now = new Date();
Date fileDate = new Date(mdbFile.lastModified());
return (now.getTime() - fileDate.getTime()) > ttl;
}
#Override
public void run() {
while (true) {
if (Thread.currentThread().isInterrupted())
throw new RuntimeException("Application Shutdown");
try {
String databases[] = new String[]{"database1", "database2"};
for (String database : databases) {
String fileName = database + "." + StringUtil.randomString(8) + ".mdb";
String fileLocation = Configuration.getValueForOS("db.path");
File mdbFile = new File(fileLocation, fileName);
File currentDatabaseFile = new File(fileLocation, database + ".mdb");
if (hasExpired(currentDatabaseFile)) {
URL url = new URL(Configuration.getValueForOS("db.url." + database));
InputStream in = url.openConnection().getInputStream();
OutputStream out = new FileOutputStream(mdbFile);
FileUtil.streamBridge(in, out);
FileUtil.close(in, out);
while (currentDatabaseFile.exists() && !currentDatabaseFile.delete()) ;
while (!mdbFile.renameTo(currentDatabaseFile)) ;
}
}
// Put the thread to sleep so the other threads do not starve
Thread.sleep(Long.parseLong(
Configuration.getValueForOS("db.http-expiration"));
} catch (IOException ioe) {
ioe.printStackTrace();
} catch (InterruptedException ie) {
ie.printStackTrace();
}
}
}
};
#Override
public void contextInitialized(ServletContextEvent servletContextEvent) {
this.thread = new Thread(updater);
thread.start();
}
#Override
public void contextDestroyed(ServletContextEvent servletContextEvent) {
if (this.thread.isAlive())
this.thread.interrupt();
}
}
What could be causing?
I based my implementation on this question: Background Thread for a Tomcat servlet app
Given that your code loops forever, you're probably starving all the other threads in the VM. Try sleeping the thread once in a while.