How to watch a complete file system for changes in Java? - java

Problem description
I would like to watch a complete file system for changes. I'm talking about watching changes in a directory recursively. So, when watching a directory (or a whole file system) all changes in sub-directories need to be captured too. The application needs to be able to track all changes by getting notified.
Java's WatchService isn't suitable
Java already has a WatchService feature, which allows you to monitor a directory for changes. The problem is however, that this isn't a recursive process as far as I know, thus you can't use this to monitor all changes in the root directory of a file system.
Watching all sub-directories explicitly
A solution I've thought of would be to register each directory inside the specified root directory explicitly. The problem with this is however, that walking through, and registering these directories is very resource expensive on a system with more than a million sub-directories. This is because the system would need to go through the whole file system recursively to only register all directories in the first place. The performance impact of this feature would be too big, if it's even possible without crashing the application.
Logical solution
I would assume an operating system would fire/call some sort of event when anything is changed on the file system, that an application is able to listen to. I did however, not find anything like this yet. This would allow the application to listen to all changes without the need to register all sub-directories explicitly. Thus the performance impact with such a method would be minimal.
Question
Is watching a whole file system, or watching a directory recursively possible in Java, and how would this be achieved?

The question should be split into several:
How to track file events across the disk on certain OS
How to use this mechanism in Java
The answer to the first question is that the approaches are different. On Windows there exist Windows API functions that let you do this (and famous FileSystemWatcher class in .NET Framework is a kind of wrapper around this API function set). The more robust method on windows is to create or use a pre-created file system filter driver. On Linux there exists inotify. On MacOS X there exist several approaches (there was a question on this topic somewhere around), none of them being universal or always available.
Also all approaches except a filesystem filter driver are good only for being notified after the event happens, but they don't let you intercept and deny the request (AFAIK, I can be mistaken here).
As for the second question, there seems to be no universal solution that would cover all or most variants that I mentioned above. You would need to first choose the mechanism for each OS, then find some wrappers for Java to use those mechanisms.

Here is an example to to watch a directory (or tree) for changes to file

Please find example https://github.com/syncany/syncany/blob/59cf87c72de4322c737f0073ce8a7ddd992fd898/syncany-lib/src/main/java/org/syncany/operations/watch/RecursiveWatcher.java
Even you can filtered our directory that you don't want to watch

Related

Using Java, is there a way to move file on the file system despite if its being used by another process? For Windows

Using Java, I want to move a file on the file system despite if it's being used by another process? I've found this for deleting FileDeleteStrategy.FORCE.delete() is there an equivalent one for moving the file into another directory?
As others already pointed out, on Windows there is no such thing as move file while it's being used by other process. There is however standard way of handling such situation, but it will require OS restart.
From documentation of MoveFileEx:
MOVEFILE_DELAY_UNTIL_REBOOT 4 (0x4)
The system does not move the file until the operating system is restarted. The system moves the file immediately after AUTOCHK is executed, but before creating any paging files.
To use this function from java application, you could use JNA, but your application needs to be running with admin privileges with appropriate UAC.
From UX point of view, Jared Smith's answer make much more sense.

How to open a file in Java that does not prevent external "Safe Save"?

We want to open a file in Java and read its contents.
This file may be updated by an external application using Safe Save. That means the file will be externally read and its updated contents will be stored to a new file. Eventually the original file is deleted and the new file is renamed to match the original file's name.
Unfortunately the external process fails during rename (last part of the Safe Save) when our Java Application is reading the original file at the same time.
We played with different kind of open modes but could not get a solution that does not fail the external reader.
Is there some way to open a file that does not interfere with external processes accessing the same file? Ideally, whenever an external process moves or deletes the file we would like to get an exception in our Java application. And only there.
Do you have any ideas on how to achieve that?
EDIT:
Just some clarification regarding the use case:
This an indexer like scenario. We want to index contents of a potentially very large filesystem where 3rd party independent processes can concurrently read from or write to as well. We have no control over the 3rd party processes.
Copying the original file seems like a big overhead and we are not sure if that helps with the original problem as it will probably fail the external reader on a Safe Save as well.
Last but not least: This should work on Windows and Linux. But we are experiencing this problems on Windows.
On Windows, whether a file can be renamed or deleted while it's open is controlled by the FILE_SHARE_DELETE sharing mode flag. This flag should be passed in when the file is opened with the low level CreateFile function.
Unfortunately, Java API does not give you control over low level Windows-specific flags. There is an open bug report to have FILE_SHARE_DELETE added by default, but it's unlikely it will be done because of backwards compatibility (some applications may depend on this behavior). the A comment in the report suggests a workaround: instead of new FileInputStream(file) use the java.nio API.
InputStream in = Files.newInputStream(file.toPath());
I don't have access to Windows right now to verify that this workaround uses the right sharing mode.
Make a copy of the original file an use this within your Java program, and at the same time keep track of the original file.
Here, this might help you out:
The java.nio.file package provides a file change notification API, called the Watch Service API. This API enables you to register a directory (or directories) with the watch service. When registering, you tell the service which types of events you are interested in: file creation, file deletion, or file modification. When the service detects an event of interest, it is forwarded to the registered process. The registered process has a thread (or a pool of threads) dedicated to watching for any events it has registered for. When an event comes in, it is handled as needed. Official docs
You cannot achieve this only with files, at least not without making additional assumptions. If the processes are not synchronized you will get either (a) errors (b) corrupted data or (c) both. Furthermore, such system will be unstable, prone to race conditions and implementation-specific details. This means that even if it looks like it's working it will not work correctly always and in each case.
Depending on your circumstances you might try to use a combination of scehduling (i.e. process A runs every even minute, process B every odd minute), exclusive/shared open flags, range locks, copying files, file change notifiers, retrying on failure etc. If you can somehow ensure that your assumptions are never broken you might end up with something which is "good enough". But all in all, this is a bad engineering practice and should be avoided.
For a proper solution, you need to make both processes aware that they are talking to each other. What you have is really a textbook use case for a database. Besides using a database there are plenty of other ways to synchronize access to data - messaging, streams, locks, shared memory etc. Each way has its own benefits and downsides and without knowing more about your specific situation it is impossible to say which would be better.

How to check that my jar file is not modified?

I hope you're doing well in these complicated times, good containment all around :D
So I come to my problem, currently I'm working on a game in Java and I know that it's possible to modify the jar file of my game which annoys me a little bit because it can give cheating possibilities for malicious players... I had thought of a solution even if it's not infallible, it would be to make sure to check if the jar file has not been modified. But the problem is that I don't know how to check this, I had thought of a system that would check the point of the file even if I doubt it would be the best solution.
If you have any other ideas to secure my game I would be interested :D
It is possible to check if a JAR file on your machine has been modified. Simply compare a cryptographic hash of the current JAR file with a previously recorded hash for a pristine copy. (Or just do a byte-by-byte comparison with the pristine copy.)
But you can't do this for JAR file on the user's machine:
You can't login to the user's machine and access their file system to look at the JAR file. (Even if you could, there is no guarantee that you would see the file that the cheater is actually using.)
If your application (running on the user's machine) tries to report on the integrity of its JAR files, this can be defeated by the cheater. All they need to do is to modify the JAR file containing the reporting code to report a fake hash.
Basically, there is no reliable way to detect that a cheater is running a modified JAR ... apart from detecting the anomalous behavior of the cheat itself.
But think of it this way. If there was a good (reliable, no circumvention) mechanism for detect that a cheater is running a modified client, then cheats would not be a problem in the many online games that exist out there. And (by extension) there would be no way to defeat software license enforcement schemes ... because software vendors would use a similar mechanism.
So, my advice would be not waste too much time on this approach. It only works against people with limited technical expertise or limited motivation.
The only way to completely prevent cheats is to control the platform on which the client runs. That is usually impractical.
In regards to your question about other ideas, the best thing you can do is validate everything from the client. One thing you should always remember is that the client cannot be trusted because you cannot verify anything from it. All you can do is store the state on a remote server and when the client tells you something, validate it, and give a response if necessary or prevent the action if necessary.
You will need to somehow find out where the jar file is: Java - Search for files in a directory
Then you can check for the last modified date of the file: https://www.boraji.com/java-get-last-modified-date-of-a-file
However, I would not consider this a very powerful defense against cheating, because one can modify the jar file and remove the validation of the file date.
You will need to think about the kind of cheating that can occur and to come up with other security measures as well.
EDIT
As Dave Thompson pointed out, the modified timestamp can be changed as well, which makes the modification of the jar file unnecessary, even though reverse engineering is still needed by the hackers, because that's how they find out what the rules of the application are.

How can I watch any particular file / directory based upon any file(located under specific directory if directory base watching) attempt for OPEN?

I want to execute few task based open targeted file's OPEN event.
For example, I am watching Sample.docx & whenever user will go for OPEN it, few subsequent task will be performed based upon it's OPEN action.
I have searched on internet & find out few solution but that are based upon file's MODIFICATION & DELETION operation. none of them shows based upon OPEN action which is actually I am looking for.
Any hint/suggestion would help me.
Thanks.
That is an operating-system specific functionality and is not something Java comes with out of the box. If you are on Windows you would use a FileSystemWatcher which exists in .NET, but if you need it in Java you would have to create native bindings if there isn't a library that already exists. Chances are this does not exist as not many people would have a valid use-case to do this and I don't think security people would be happy to see this either.
You could I suppose, in a specific thread, regularly poll currently running processes to see if the file name is contained within a process title.
As for .docx files for example, WORD would have this as its process title:
Sample.docx - Microsoft Word
You would need to utilize a JNA method named getAllWindowNames() to acquire a list of Window Names. This method works quite well. When Sample.docx is detected within the acquired list then start whatever file or files you like.
Keep in mind however, your Java application would need to always be running in the background and because of file association (as mentioned to you in a previous post) this technique would run the files you have associated with Sample.docx regardless of how the file was run (from a double-click in Windows Explorer, a shortcut on Desktop, opened from MS WORD itself, etc).
I have actually created a small demo application that does exactly what you are trying to accomplish however it is too large to post here. There is no tutorial that I know of for this sort of thing, it's just a matter of doing it....that is if the concept appeals to you.
Yes!...most people would not want this sort of thing dancing around on their System(s).

Lock future file

So I have a Samba file server on which my Java app needs to write some files. The thing is that there is also another php application (if a php script is even considered an application) that is aggressively pulling the same directory for new files.
Sometimes, the php script is pulling the file before my Java app is done writing it completely to the disk. Here is a little bit of ascii art to help visualize what I currently have (but doesn't work):
Samba share
/foo (my java app drops file here)
/bar (the directory that the php is pulling)
What I'm currently doing is when the file meets some criterias, it's being moved to /bar and then picked up by the php for more processing. I've tried different thing such has setting the file non writable and non readable before calling renameTo.
I've looked a little bit at FileLocks but it doesn't seem to be able to lock future files. So I am wondering what kind of possiblities I have here? What could I use to lock the file from being picked up before it's fully written without touching the php (because, well, it's php and I don't really have the right to modify it right now).
Thanks
Edit 1
I've got some insight on what the php script is really doing if it can help in any way.
It's reading the directory file in loop (using readdir without sleeping).
As soon as it finds a filename other than "." and "..", it calls file_get_contents and that's where it fails because the file is not completely written to disk (or not even there since the Java code might not even had time to write it between the readdir and file_get_contents.
Edit 2
This Java application is replacing an old php script. When they implemented it, they had the same problem I'm having right now. They solved it by writing the new file in /bar/tmp(with file_put_contents) and then use rename to move it to bar (it looks like rename is supposed to be atomic). And it's been working fine so far. I can't and won't believe that Java can't do something better than what php does...
I think this is due to the fact read locks are shared (multiple process can apply read locks to the same file and read it together).
One approach you can do is to create a separate temporary lock file (eg: /bar/file1.lock) while /bar/file1 hasn't finished copying. Delete the lock file as soon as the file copying is finished.
Then alter the php code to ensure the file isn't being locked before it reads.
You mentioned that you tried FileLock, but keep in mind the disclaimer in the javadoc for that method:
Whether or not a lock actually prevents another program from accessing
the content of the locked region is system-dependent and therefore
unspecified. The native file-locking facilities of some systems are
merely advisory, meaning that programs must cooperatively observe a
known locking protocol in order to guarantee data integrity.
You also mentioned you are using File.renameTo, which also has some caveats (mentioned in the javadoc):
Many aspects of the behavior of this method are inherently
platform-dependent: The rename operation might not be able to move a
file from one filesystem to another, it might not be atomic, and it
might not succeed
Instead of File.renameTo, Try Files.move with the ATOMIC_MOVE option. You'll have to catch AtomicMoveNotSupportedException and possibly fall back to some alternative workaround in case an atomic move is not possible.
You could create a hardlink with Files.createLink(Paths.get('/foo/myFile'), 'Paths.get('/bar/myFile')) then delete the original directory entry (in this example, /foo/myFile.
Failing that, a simple workaround that doesn't require modification to the PHP is to use a shell command or system call to move the file from /foo to /bar. You could, for example, use ProcessBuilder to call mv, or perhaps call ln to create a symlink or hardlink in /bar. You might still have the same problem with mv if /foo and /bar are on different filesystems.
If you have root privileges on the server, you could also try implementing mandatory file locking. I found an example in C, but you could call the C program from Java or adapt the example to Java using JNA (or JNI if you want to punish yourself).

Categories