Create and write a file in one atomic operation on OS level - java

So I need to create a file then write one line to it and this must be atomic. So that no other process may tinker with the file whilst it is under initialization.
I have one idea, to lock on something different then while the lock is held, do the operations then release the lock to let the other party in. But this is quite tedious, also may be erroneus because creating the lock and acquiring it might be not atomic (I guess). No other way to do it?

I'd suggest you to write temporary file and then rename it to your file. I am not sure this operation is implemented in java as atomic for all operating system but at least on Unix you have a chance because I think it uses the same call as mv that is atomic.
It will not be truly atomic on windows, I guess. It will be "almost atomic" that is enough for most applications.

If you want something that will be safe guarded from other processes, I suggest that you read http://en.wikipedia.org/wiki/File_locking and utilize JNI to get to the OS level.

Related

Is this a valid use of Thread.stop()?

Okay, I know it's dangerous, I know it's deprecated, and I know using it would make baby Jesus cry. I think I'm aware of the implications of calling it and have read this related question.
Here's my scenario. I would like to test a data processing library. It runs multiple jobs, one per thread. Each job only communicates with other jobs via an out-of-process queueing system. Otherwise, jobs are independent: there is no shared state between threads, at least not in my code base.
I would like to test that if some terrible thing such as an OutOfMemoryError or a cosmic ray killing the VM happens at some random point in a job, that the rest of the system is okay. Therefore I want to stop a thread at a completely arbitrary point, and killing the thread should not leave resources accessible by other threads in an undefined state. The job logic is part of a framework that I don't want to compromise for the purposes of this test so it's not viable to intersperse random exits throughout the job code.
Is this an appropriate use of Thread.stop()? And so that this is not an XY question, is there any other practical way to accomplish my goal? (I suppose it could be done with bytecode instrumentation but I think that would be tremendously difficult.)

How safe is it to use Java FileLock?

How safe is it to use java.nio.channels.FileLock for locking files among processes? It is stated that other processes can not access the file if we have an exclusive lock. However, the below answer on another SO question states other processes have to check for the filelock too in order for our process to be safe.
(a) Are you aware that locking the file won't keep other processes from touching it unless they also use locks?
So I tested my code and tried to change, a file which I have the lock already, with Windows Text Editor and I was safe from harm but not when I test with Notepad++..
Is there a solution for locking a file appropriately in Java 6?
Java FileLock uses advisory (not mandatory) locks on many platforms. That means it may only provide locking against other applications that also use FileLock (or the equivalent in other languages).
Neither Linux or Windows implement mandatory locking across the board. For instance:
For Linux and similar, file locking is advisory only.
For Windows, according to Wikipedia:
"For applications that use the file read/write APIs in Windows, byte-range locks are enforced .... by the file systems that execute
within Windows. For applications that use the file mapping APIs in
Windows, byte-range locks are not enforced ..."
In other words, locking on Windows can be either mandatory or advisory, depending on which API an Windows application uses to access files.
How safe is it to use Java FileLock?
If you are actually asking if it is safe to assume that FileLock provides mandatory file locking with respect to all other applications (Java & non-Java) irrespective of how they are written, the answer is No. It is NOT safe to make that assumption.
Is there a solution for locking a file appropriately in Java 6?
Only if all of the applications (Java & other) cooperate; e.g. by using FileLock or the equivalent.
If you can't make that assumption, there is no solution using portable Java. Indeed, on most (if not all) common OS platforms, there is no solution at all, AFAIK ... because the platform itself doesn't support mandatory file locking independent of the application.
From the Javadoc of java.nio.channels.FileLock under Platform Dependencies:
The native file-locking facilities of some systems are merely advisory, meaning that programs must cooperatively observe a known locking protocol in order to guarantee data integrity. On other systems native file locks are mandatory, meaning that if one program locks a region of a file then other programs are actually prevented from accessing that region in a way that would violate the lock. On yet other systems, whether native file locks are advisory or mandatory is configurable on a per-file basis. To ensure consistent and correct behavior across platforms, it is strongly recommended that the locks provided by this API be used as if they were advisory locks.
As you discovered from your testing, other non-Java code running on your version of Windows does not have to honor your exclusive lock.
Your only solution is to read the file into memory as fast as you can, take your time processing the information, then write the file to disk as fast as you can.
It is stated that other processes can not access the file if we have an exclusive lock.
It is stated where? Not in the Javadoc.
However, the below answer on another [SO question][2] states other processes have to check for the filelock too in order for our process to be safe.
That is correct.
So I tested my code and tried to change, a file which I have the lock already, with Windows Text Editor and I was safe from harm but not when I test with Notepad++.
You're already doing something invalid by testing on the one platform where file locks affect ordinary opens, but the only conclusion to be drawn from this is that it isn't safe. Notepad++ keeps the file open, and so encounters your locks, but Windows Text Editor doesn't, and so doesn't see the locks either, until you try to save.
Is there a solution for locking a file appropriately in Java 6?
Not unless the applications you're locking against also use file locks.

Is there any way or tool that I can use to verify whether my API is thread safe in Java?

I make a tool and provide an API for external world, but I am not sure whether it is thread safe. Because users may want t use it in multiple-thread environment. Is there any way or tool that I can use to verify whether my API is thread safe in Java?
No. There is no such tool. Proving that a complex program is thread safe is very hard.
You have to analyze your program very carefully to ensure that is thread safe. Consider buying "Java concurrency in practice" (very good explanation of concurrency in java).
Stress tests, or static analysis tools like PMD and FindBugs can uncover some concurrency bugs in your code. So these can show if your code is not thread-safe. However they can never prove if it is thread-safe.
The most effective method is a thorough code review by developer(s) experienced in concurrency.
You can always stress-test it with tools like jmeter.
But the main problem with threads is that they're mostly unpredictable, so even with stress-tests etc. you can't be 100% sure that it will be totally thread safe.
Resources :
Wikipedia - Thread-safety
This is a variant (or so called "reduction") of the Halting Problem. Therefore it is provably unsolvable. for all non-trivial cases. (Yes, that's an edit)
That means you can find errors by any usual means (statistics, logic) but you can never completely prove that there are none.
I suppose those people saying proving an arbitrary multithreaded program is thread-safe is impossible are, in a way, correct. An arbitrary multithreaded program, coded without following strict guidelines, simply will have threading bugs, and you can't validly prove something that isn't true.
The trick is not to write an arbitrary program, but one with threading logic simple enough to possibly be correct. This then can be unambiguously validated by a tool.
The best such tool I'm aware of is CheckThread. It works on the basis of either annotations, or xml config files. If you mark a method as '#ThreadSafe' and it isn't, then you get a compile-time error. This is checked by looking at the byte code for thread-unsafe operations, e.g. reads/write sequences on unsynchronised data fields.
It also handles those APIs that require methods to be called on specific threads, e.g. Swing.
It doesn't actually handle deadlocks, but those can be statically eliminated without even requiring annotation, by using a tool such as Jlint. You just need to follow some minimal standards like ensuring locks are acquired according to a DAG, not willy-nilly.
You cannot and never will be able to automatically proof that a program is threadsafe anymore that you can prove that a program is correct (unless you think you solved the halting program, which you didn't).
So, no, you cannot verify that an API is threadsafe.
However in quite some case you can prove that it is broken, which is great!
You may also be interested in automatic deadlock detection, which in quite some case simply "just work". I'm shipping a Java program on hundreds of desktops with such a deadlock detector installed and it is a wonderful tool. For example:
http://www.javaspecialists.eu/archive/Issue130.html
You can also stress test your application in various ways.
Bogus multi-threaded programs tend to not work very well when a high load is present on the system.
Here's a question I asked about how to create easily create a high CPU load on a Un*x system, for example:
Bash: easy way to put a configurable load on a system?

java io read and write lock

suppose I have a file that might gets written by one thread/process Writer and read by another thread/process Reader.
Writer updates the file every x time interval, and Reader reads it every y time interval,
if they happen to read and write to the file at the same time, will there be any issues? would the read block until writes finishes? or would the read fails? and vice versa?
What's the best practice here?
You'll need to devise your own locking protocol to implement in the applications. Specifics depend on the underlying operating system, but in general, nothing will stop one process from reading a file even when another process is writing to it.
Java has a FileLock class that can be used to coordinate access to a file. However, you'll need to read the caveats carefully, especially those relating to the system-dependence of this feature. Testing the feature on the target operating system is extremely important.
A key concept of Java's FileLock is that it is only "advisory". Your process should be able to detect that another process holds a lock on a file, but your process can ignore it and do what it likes with the file, no restrictions.
The question is ambiguous whether multiple process will use the file, or merely separate threads within a single Java process. That's a big difference. If the problem requires only thread safety within a single process, a ReentrantReadWriteLock can provide a robust, high performance solution, without any platform-specific pitfalls.
Best practice is to not use a file for communication between processes. File are not designed for this purposes. Instead you should use messaging which IS designed for communication between processes. You can use files as well to audit what has been sent/received,
If you use files alone, you could come up with a solution which is good enough, but I don't believe you will have a solution which could be considered best practice.

Lock across several jvm?

this is a bit related to this question.
I'm using make to extract some information concerning some C programs. I'm wrapping the compilation using a bash script that runs my java program and then gcc. Basically, i'm doing:
make CC=~/my_script.sh
I would like to use several jobs (-j option with make). It's running several processes according to the dependency rules.
If i understood well, I would have as many instances of the jvm as jobs, right ?
The thing is that i'm using sqlite-jdb to collect some info. So the problem is how to avoid several processes trying to modify the db at the same time ?
It seems that the sqlite lock is jvm-dependant (i mean one lock can be "see" only inside the locking jvm), and that this is the same for RandomAccessFile.lock().
Do you have any idea how to do that ? (creating a tmp file and then looking if it exists or not seems to be one possibility but may be expensive. A locking table in the dB ? )
thanks
java.nio.channels.FileLock allows OS-level cross-process file locking.
However, using make to start a bash scripts that runs several JVMs in parallel before calling gcc sounds altogether too Rube-Goldbergian and brittle to me.
there are several solutions for this.
if your lock should be within the same machine, you can use a server socket to implement it (The process that manages to bind to the port first owns the lock, other processes waits for the port to become available).
if you need a lock that span across multiple machines you can use a memcached lock. this will require a memcached server running. I can paste some code if you are interested in this solution.
you can get Java library to connect to memcached here.
You may try Terracotta for sharing objects between various JVM instances. It may appear as a too heavy solution for your needs, but at least worth considering.

Categories