I'm trying to write to a file, using a Java FileLock, to prohibit all other processes and threads from reading from it or writing to it until I'm finished with it. Given this question and the answers to it, it seems to me like this is the perfect tool for what I want--a mutex for file access.
However, I am very concerned about this text from the JavaDocs:
File locks are held on behalf of the entire Java virtual machine. They
are not suitable for controlling access to a file by multiple threads
within the same virtual machine.
Can someone either alleviate my fears or point me in the right direction? It sounds like FileLock won't work at all to keep a different thread out of the file, even if another thread has already obtained it. If this is the case, is there another canonical Java method to do this that will protect from other threads?
The FileLock is a process level lock and will thus not protect the file from concurrent access from multiple threads within the process that has the lock.
You need to use a combination of the FileLock to protect from concurrent access from other processes and some other synchronization mechanism (like a synchronized method for accessing the file) within your process to protect from concurrent access by your own threads.
I would implement this as follows:
interface FileOperator {
public void operate(File file);
}
class FileProxy {
private static final ConcurrentHashMap<URI, FileProxy> map =
new ConcurrentHashMap<>();
private final Semaphore mutex = new Semaphore(1, true);
private final File file;
private final URI key;
private FileProxy(File file) {
this.file = file;
this.key = file.toURI();
}
public static void operate(URI uri, FileOperator operator) {
FileProxy curProxy = map.get(uri);
if(curProxy == null) {
FileProxy newProxy = new FileProxy(new File(uri));
FileProxy curProxy = map.putIfAbsent(newProxy.key, newProxy);
if(curProxy == null) {
curProxy = newProxy; // FileProxy was not in the map
}
}
try {
curProxy.mutex.acquire();
operator.operate(curProxy.file);
} finally {
curProxy.mutex.release();
}
}
}
The threads that are using a file implement FileOperator or something similar. Files are hidden behind a FileProxy that maintains a static ConcurrentHashMap of key (URI, or absolute path, or some other file invariant) value (FileProxy) pairs. Each FileProxy maintains a Semaphore that acts as a mutex - this is initialized with one permit. When the static operate method is called, a new FileProxy is created from the URI if none exists; the FileOperator is then added to the FileProxy queue; acquire is called on the mutex to ensure that only one thread can operate on the file at a time; and finally the FileOperator does its thing.
In this implementation, FileProxy objects are never removed from the ConcurrentHashMap - if this is a problem then a solution is to wrap the FileProxy objects in a WeakReference or SoftReference so that they can be garbage collected, and then call map.replace if reference.get() == null to ensure that only one thread replaces the GC'd reference.
Related
I use in project Striped locks from GUAVA for synchronizing input data for every unique identifier of process, it provides right order of data for next processing this data.
Striped<Lock> locks = Striped.lazyWeakLock(10);
public void put(String id, String data) {
Lock lock = locks.get(id);
lock.lock()
try {
// process data
} finally {
lock.unlock()
}
}
This code works good, provides right order of data.
But I want to get global lock (exclusive access) to current object for modification some data. So i want to stop getting locks for all unique id until I modify data.
Possible solutions:
I can use ReasWriteLock before getting striped lock. (read - for "locks.get(id)", write - for exclusive access) - Not good solution, extra locking in normal state.
I can use one lock for all, but it will slow down application
Global lock
Example:
if (locked) {
global.lock()
global.unlock()
Lock lock = locks.get(id);
...
} else {
Lock lock = locks.get(id);
if (locked){
...
}
lock.lock();
}
void extraLock(){
global.lock();
locked = true;
// modify data
locked = false;
global.unlock()
}
I think solution with global lock looks not good(flags, lock, double check and etc)
Question:
Can you advice any solution that can help me to solve this problem?
Or Can you tell some idea?
Thanks
I have an application where I want to ensure that a method is called at most once concurrently, say when updating user balance in a database.
I am thinking of using the following locking mechanism: (showing Scala code below, but should be similar with Java Lambdas):
object Foo{
val dbLocked = new java.util.concurrent.atomic.AtomicBoolean(false)
def usingAtoimcDB[T](f: => T):T = {
if (dbLocked.get) throw new Exception("db is locked")
dbLocked.set(true)
try f
finally dbLocked.set(false)
}
}
Is this safe to use when usingAtoimcDB may be called concurrently?
EDIT: The corrected code below, as pointed in this answer:
def usingAtoimcDB[T](f: => T):T = {
if(dbLocked.compareAndSet(false, true)) {
//db is now locked
try f
finally dbLocked.set(false)
} else {
//db is already locked
throw new Exception("db is locked")
}
}
EDIT 2:
Using a spinloop. Is this also ok?
def usingAtoimcDB[T](f: => T):T = {
while (!dbLocked.compareAndSet(false, true)) {Thread.sleep(1)}
try f
finally dbLocked.set(false)
}
EDIT3: Based on the answers and comments below, I am also considering using queues.
Inadvisable. You are requesting that the same pieco of code running in the same application instance on tha same server is the single point to do that transaction. There also is no provision to let this code stand-out. When you are retired, someone may start a second application instance or whatever.
Whereas a database commit/rollback is a quite simple and sure mechanism.
When you cannot write an integration (unit) test to ensure this sole point, then do not do it.
If you do it:
Revoke rights to the table modifications for the normal database user
Add a new database use who has sufficient right granted
And still: do not do it.
The code you posted above is not thread-safe, because you are not using an atomic check-and-set operation. Two threads can both be executing the if (dbLocked.get) statement at the same time and both get false as the answer, and then both will do dbLocked.set(true) and call f.
If you really want to use AtomicBoolean, then you must use compareAndSet as #leshkin already showed - this is an atomic operation that does the check and set in one go without the possibility of another thread doing the same thing at the same time, so that it is thread-safe.
You are using an AtomicBoolean as a lock here. There are classes in the standard Java library which are better suited (and specifically made) for this purpose; have a look at the package java.util.concurrent.locks.
You could for example use class ReentrantReadWriteLock, which combines two locks for reading and writing. The write lock is exclusive (when it's locked, nobody else can read or write); the read lock is shared (when it's locked, nobody can write, but others can read at the same time). This allows for there to be multiple readers concurrently, but only one writer at a time, possibly improving efficiency (it's not necessary to make reading an exclusive operation).
Example:
import java.util.concurrent.locks._
object Foo {
private val lock: ReadWriteLock = new ReentrantReadWriteLock
def doWriteOperation[T](f: => T): T = {
// Locks the write lock
lock.writeLock.lock()
try {
f
} finally {
lock.writeLock.unlock()
}
}
def doReadOperation[T](f: => T): T = {
// Locks the read lock
lock.readLock.lock()
try {
f
} finally {
lock.readLock.unlock()
}
}
}
Yes, it should work as espected. I would slightly modify your function using compareAndSet call.
compareAndSet method has the advantage to be an atomic operation - there are no race conditions and the value will be changed atomically.
def usingAtoimcDB[T](f: => T):T = {
if(dbLocked.compareAndSet(false, true)) {
//db is now locked
try f
finally dbLocked.set(false)
} else {
//db is already locked
throw new Exception("db is locked")
}
}
I have a write method that is supposed to safely write data to a file.
// The current file I am writing to.
FileOutputStream file = null;
...
// Synchronized version.
private void write(byte[] bytes) {
if (file != null && file.getChannel() != null) {
try {
boolean written = false;
do {
try {
// Lock it!
FileLock lock = file.getChannel().lock();
try {
// Write the bytes.
file.write(bytes);
written = true;
} finally {
// Release the lock.
lock.release();
}
} catch (OverlappingFileLockException ofle) {
try {
// Wait a bit
Thread.sleep(0);
} catch (InterruptedException ex) {
throw new InterruptedIOException("Interrupted waiting for a file lock.");
}
}
} while (!written);
} catch (IOException ex) {
log.warn("Failed to lock " + fileName, ex);
}
} else {
log.warn("Failing - " + (file == null ? "file" : "channel") + " is null!!");
}
}
It has worked fine for me for a while now, although I know there are some wrinkles in it.
I have recently changed a project that uses this code to build and run under Java 5 (from Java 6) and now it looks like it is deadlocked awaiting a lock on the file. It is a multithreaded app and it is quite possible for several threads to attempt to write to the same file.
The debugger tells me that the hung threads are waiting for the FileLock lock = file.getChannel().lock() call to return.
Some research brought up this interesting little nugget which mentions:
File locks are held on behalf of the entire Java virtual machine. They are not suitable for controlling access to a file by multiple threads within the same virtual machine.
So am I doing it wrong? If so what is the right way? If I am doing it right how come I hit a deadlock?
Added: Forgot to mention - each thread holds its own copy of this object so there should not be any synchronisation issues within the code. I felt safe to rely on the FileChannel.lock() method to ensure writes do not interleave.
Added too: I have indeed solved the issue using various synchronized mechanisms. I do, however, have outstanding questions:
Why is FileLock lock = file.getChannel().lock(); not suitable ...?
Why did my issues only appear when switching back to Java-5 when everything worked fine with Java-6?
FileLock is only for interprocess locking, javadoc reads:
"File locks are held on behalf of the entire Java virtual machine.
They are not suitable for controlling access to a file by multiple
threads within the same virtual machine."
To lock between java threads (same JVM) you need to use some shared lock. I would suggest within the file writing class to use a synchronized block (which according to these articles is likely to perform best):
final Object lock = new Object();
public void write(...){
synchronized(lock){
// do writing
}
}
Another approach is to use a ReentrantLock and then use the proven idiom of
final ReentrantLock lock = new ReentrantLock();
public void write(...){
try {
lock.lock()
// do the writing
} finally {
// forget this and you're screwed
lock.unlock();
}
}
you may need to implement critical section concept on the actual code using the file, rather than the hashmap. You can either create a synchronized block or separate the file access code into a separate procedure and make that method synchronized.
Essentially, only one thread executes a synchronized block at a time. It gives you the exclusive access you need.
Another way of doing it is, to use a serial thread Executor, depending on your functional requirements.
You may want to look at this thread:
Howto synchronize file access in a shared folder using Java (OR: ReadWriteLock on network level)
A multi-threaded piece of code accesses a resource (eg: a filesystem) asynchronously.
To achieve this, I'll use condition variables. Suppose the FileSystem is an interface like:
class FileSystem {
// sends a read request to the fileSystem
read(String fileName) {
// ...
// upon completion, execute a callback
callback(returnCode, buffer);
}
}
I have now an application accessing the FileSystem. Suppose I can issue multiple reads through a readFile() method.
The operation should write data to the byte buffer passed to it.
// constructor
public Test() {
FileSystem disk = ...
boolean readReady = ...
Lock lock = ...
Condition responseReady = lock.newCondition();
}
// the read file method in quesiton
public void readFile(String file) {
try {
lock.lock(); // lets imagine this operation needs a lock
// this operation may take a while to complete;
// but the method should return immediately
disk.read(file);
while (!readReady) { // <<< THIS
responseReady.awaitUninterruptibly();
}
}
finally {
lock.unlock();
}
}
public void callback(int returnCode, byte[] buffer) {
// other code snipped...
readReady = true; // <<< AND THIS
responseReady.signal();
}
Is this the correct way to use condition variables? Will readFile() return immediately?
(I know there is some sillyness in using locks for reads, but writing to a file is also an option.)
There's a lot missing from your question (i.e. no specific mention of Threads) but I will try to answer anyway.
Neither the lock nor the conditional variables give you background capabilities -- they just are used for a thread to wait for signals from other threads. Although you don't mention it, the disk.read(file) method could spawn a thread to do the IO and then return immediately but the caller is going to sit in the readReady loop anyway which seems pointless. If the caller has to wait then it could perform the IO itself.
A better pattern could be to use something like the Java 5 Executors service:
ExecutorService pool = Executors.newFixedThreadPool(int numThreads);
You can then call pool.submit(Callable) which will submit the job to be performed in the background in another thread (when the pool next has one available). Submit returns a Future which the caller can use to investigate if the background task has finished. It can return a result object as well. The concurrent classes take care of the locking and conditional signal/wait logic for you.
Hope this helps.
p.s. Also, you should make readReady be volatile since it is not synchronized.
What is the best way to share a file between two "writer" services in the same application?
Edit:
Sorry I should have given more details I guess.
I have a Service that saves entries into a buffer. When the buffer gets full it writes all the entries to the file (and so on). Another Service running will come at some point and read the file (essentially copy/compress it) and then empty it.
Here is a general idea of what you can do:
public class FileManager
{
private final FileWriter writer = new FileWriter("SomeFile.txt");
private final object sync = new object();
public void writeBuffer(string buffer)
{
synchronized(sync)
{
writer.write(buffer.getBytes());
}
}
public void copyAndCompress()
{
synchronized(sync)
{
// copy and/or compress
}
}
}
You will have to do some extra work to get it all to work safe, but this is just a basic example to give you an idea of how it looks.
A common method for locking is to create a second file in the same location as the main file. The second file may contain locking data or be blank. The benefit to having locking data (such as a process ID) is that you can easily detect a stale lockfile, which is an inevitability you must plan for. Although PID might not be the best locking data in your case.
example:
Service1:
creates myfile.lock
creates/opens myfile
Service2:
Notices that myfile.lock is present and pauses/blocks/waits
When myfile.lock goes away, it creates it and then opens myfile.
It would also be advantageous for you to double-check that the file contains your locking information (identification specific to your service) right after creating it - just in case two or more services are waiting and create a lock at the exact same time. The last one succeeds and so all other services should notice that their locking data is no longer in the file. Also - pause a few milliseconds before checking its contents.