How safe is it to use java.nio.channels.FileLock for locking files among processes? It is stated that other processes can not access the file if we have an exclusive lock. However, the below answer on another SO question states other processes have to check for the filelock too in order for our process to be safe.
(a) Are you aware that locking the file won't keep other processes from touching it unless they also use locks?
So I tested my code and tried to change, a file which I have the lock already, with Windows Text Editor and I was safe from harm but not when I test with Notepad++..
Is there a solution for locking a file appropriately in Java 6?
Java FileLock uses advisory (not mandatory) locks on many platforms. That means it may only provide locking against other applications that also use FileLock (or the equivalent in other languages).
Neither Linux or Windows implement mandatory locking across the board. For instance:
For Linux and similar, file locking is advisory only.
For Windows, according to Wikipedia:
"For applications that use the file read/write APIs in Windows, byte-range locks are enforced .... by the file systems that execute
within Windows. For applications that use the file mapping APIs in
Windows, byte-range locks are not enforced ..."
In other words, locking on Windows can be either mandatory or advisory, depending on which API an Windows application uses to access files.
How safe is it to use Java FileLock?
If you are actually asking if it is safe to assume that FileLock provides mandatory file locking with respect to all other applications (Java & non-Java) irrespective of how they are written, the answer is No. It is NOT safe to make that assumption.
Is there a solution for locking a file appropriately in Java 6?
Only if all of the applications (Java & other) cooperate; e.g. by using FileLock or the equivalent.
If you can't make that assumption, there is no solution using portable Java. Indeed, on most (if not all) common OS platforms, there is no solution at all, AFAIK ... because the platform itself doesn't support mandatory file locking independent of the application.
From the Javadoc of java.nio.channels.FileLock under Platform Dependencies:
The native file-locking facilities of some systems are merely advisory, meaning that programs must cooperatively observe a known locking protocol in order to guarantee data integrity. On other systems native file locks are mandatory, meaning that if one program locks a region of a file then other programs are actually prevented from accessing that region in a way that would violate the lock. On yet other systems, whether native file locks are advisory or mandatory is configurable on a per-file basis. To ensure consistent and correct behavior across platforms, it is strongly recommended that the locks provided by this API be used as if they were advisory locks.
As you discovered from your testing, other non-Java code running on your version of Windows does not have to honor your exclusive lock.
Your only solution is to read the file into memory as fast as you can, take your time processing the information, then write the file to disk as fast as you can.
It is stated that other processes can not access the file if we have an exclusive lock.
It is stated where? Not in the Javadoc.
However, the below answer on another [SO question][2] states other processes have to check for the filelock too in order for our process to be safe.
That is correct.
So I tested my code and tried to change, a file which I have the lock already, with Windows Text Editor and I was safe from harm but not when I test with Notepad++.
You're already doing something invalid by testing on the one platform where file locks affect ordinary opens, but the only conclusion to be drawn from this is that it isn't safe. Notepad++ keeps the file open, and so encounters your locks, but Windows Text Editor doesn't, and so doesn't see the locks either, until you try to save.
Is there a solution for locking a file appropriately in Java 6?
Not unless the applications you're locking against also use file locks.
Related
So I need to create a file then write one line to it and this must be atomic. So that no other process may tinker with the file whilst it is under initialization.
I have one idea, to lock on something different then while the lock is held, do the operations then release the lock to let the other party in. But this is quite tedious, also may be erroneus because creating the lock and acquiring it might be not atomic (I guess). No other way to do it?
I'd suggest you to write temporary file and then rename it to your file. I am not sure this operation is implemented in java as atomic for all operating system but at least on Unix you have a chance because I think it uses the same call as mv that is atomic.
It will not be truly atomic on windows, I guess. It will be "almost atomic" that is enough for most applications.
If you want something that will be safe guarded from other processes, I suggest that you read http://en.wikipedia.org/wiki/File_locking and utilize JNI to get to the OS level.
The project I am working on would trigger various asynchronous jobs to do some work. As I look into it more these asynchronous jobs are actually being run as separate JVMs (separate java processes). Does it mean I would not be able to use any of the following if I need to synchronize between these processes:
synchronized methods/blocks
any lock that implements java.util.concurrent.locks
Because it seems to me they are all thread-level?
Does Java provide support for IPC like semaphores between processes?
That's right. You can not use any standard synchronization mechanisms because they are working into one JVM.
Solutions
You can use file locks introduced in java 7.
You can use synchronization via database entities.
One of already implemented solutions like Terracota may be helpful
Re-think your design. If you are beginner in java world try to talk in details with more experienced engineers. Your question shows that IMHO you are just on wrong way.
You can use synchronized keyword, locks, atomic objects, etc. - but they are local to the JVM. So if you have two JVMs running the same program, they can still e.g. run the same synchronized method at the same time - one on each JVM, but not more.
Solutions:
terracotta provides distributed locking
hazelcast as well
you can use manual synchronization on file system or database
I'm using distributed lock provided by Redisson to synchronize work of different JVMs
they are all thread-level?
That's correct, synchronized etc only work within the context of a single process.
Does Java provide support for IPC like semaphores between processes?
One way to implement communication between Java processes is using RMI.
I have implemented a java IPC Lock implementation using files: FileBasedLock and a IPC Semaphore implementation using a shared DB (jdbc): JdbcSemaphore. Both implementations are part of spf4j.
If you have a zookeeper instance take a look at the Zookeeper based Lock recipes from Apache Curator
Are Memory-mapped files in Java like Memory-mapped files for Windows? Or is it only emulation based on memory and file common operations in Java?
It uses the OS support for memory mapped files.
I'm trying to find documentation to back that up, but I haven't found anything conclusive yet. However, various bits of the docs do say things like this:
Many of the details of memory-mapped files are inherently dependent upon the underlying operating system and are therefore unspecified. The behavior of this method when the requested region is not completely contained within this channel's file is unspecified. Whether changes made to the content or size of the underlying file, by this program or another, are propagated to the buffer is unspecified. The rate at which changes to the buffer are propagated to the file is unspecified.
If everything were just emulated, there'd be no need for such unspecified behaviour.
Reading between the lines of your question and comment, what you are trying to do is use a memory mapped file to share memory/objects between a Java and C++ application.
My advice is don't try to do this.
There are difficult issues that would need to be addressed to make this reliable:
synchronizing the two applications' use of the shared data structure,
ensuring that changes made by one application get reliably written to main memory and read by the other one,
ensuring that changes get flushed to disc in the expected order.
A Java specific problem is that you cannot put your Java objects in the memory mapped area. Instead you have to serialize and deserialize them in some way that is compatible with the representations that the C++ side is expecting.
Finally, even if you do succeed in addressing all of those issues, your solution is likely to be fragile because it depends on unspecified behaviour of the OS, C++ and Java implementations that potentially may change if you change versions of any of the above.
suppose I have a file that might gets written by one thread/process Writer and read by another thread/process Reader.
Writer updates the file every x time interval, and Reader reads it every y time interval,
if they happen to read and write to the file at the same time, will there be any issues? would the read block until writes finishes? or would the read fails? and vice versa?
What's the best practice here?
You'll need to devise your own locking protocol to implement in the applications. Specifics depend on the underlying operating system, but in general, nothing will stop one process from reading a file even when another process is writing to it.
Java has a FileLock class that can be used to coordinate access to a file. However, you'll need to read the caveats carefully, especially those relating to the system-dependence of this feature. Testing the feature on the target operating system is extremely important.
A key concept of Java's FileLock is that it is only "advisory". Your process should be able to detect that another process holds a lock on a file, but your process can ignore it and do what it likes with the file, no restrictions.
The question is ambiguous whether multiple process will use the file, or merely separate threads within a single Java process. That's a big difference. If the problem requires only thread safety within a single process, a ReentrantReadWriteLock can provide a robust, high performance solution, without any platform-specific pitfalls.
Best practice is to not use a file for communication between processes. File are not designed for this purposes. Instead you should use messaging which IS designed for communication between processes. You can use files as well to audit what has been sent/received,
If you use files alone, you could come up with a solution which is good enough, but I don't believe you will have a solution which could be considered best practice.
this is a bit related to this question.
I'm using make to extract some information concerning some C programs. I'm wrapping the compilation using a bash script that runs my java program and then gcc. Basically, i'm doing:
make CC=~/my_script.sh
I would like to use several jobs (-j option with make). It's running several processes according to the dependency rules.
If i understood well, I would have as many instances of the jvm as jobs, right ?
The thing is that i'm using sqlite-jdb to collect some info. So the problem is how to avoid several processes trying to modify the db at the same time ?
It seems that the sqlite lock is jvm-dependant (i mean one lock can be "see" only inside the locking jvm), and that this is the same for RandomAccessFile.lock().
Do you have any idea how to do that ? (creating a tmp file and then looking if it exists or not seems to be one possibility but may be expensive. A locking table in the dB ? )
thanks
java.nio.channels.FileLock allows OS-level cross-process file locking.
However, using make to start a bash scripts that runs several JVMs in parallel before calling gcc sounds altogether too Rube-Goldbergian and brittle to me.
there are several solutions for this.
if your lock should be within the same machine, you can use a server socket to implement it (The process that manages to bind to the port first owns the lock, other processes waits for the port to become available).
if you need a lock that span across multiple machines you can use a memcached lock. this will require a memcached server running. I can paste some code if you are interested in this solution.
you can get Java library to connect to memcached here.
You may try Terracotta for sharing objects between various JVM instances. It may appear as a too heavy solution for your needs, but at least worth considering.