The Java manual says:
The locks held on a particular file by a single Java virtual machine do not overlap. The overlaps method may be used to test whether a candidate lock range overlaps an existing lock.
I guess that if I lock a file in a tomcat web application I can't be sure that every call to this application is done by a different JVM, can I? So how can I lock files within my tomcat application in a reliable way?
I was actually having the exact opposite problem -- my servlet was causing files to get locked that wouldn't clear when the context was reloaded. Of course the reason was because I was opening up InputStreams/BufferedReaders and not closing them. For you, opening up the files in this fashion might be a pretty low-tech solution to your problem as it should result in a lock at the O/S level, which is probably what you want.
Related
I am using Java's file locking API on a Linux server machine and try to lock a file on a remote Linux NFS system. There are some issues that have popped and logs show that 2 different cluster nodes running the same Java webserver app are able to both acquire a lock on the same NFS file.
Reading online about how Java handles locks and Linux file locking (we usually deploy on Windows servers so there is very little Linux experience), what we do should work. We are essentially issuing advisory locks but since both cluster nodes run the same code they are cooperating processes and they check for lock acquisition before starting to do any read-write ops. However this does not seem to be the case and both systems are able to successfully acquire a lock on the file, concurrently.
Are those known issues? Many comments/articles online declare NFS file locking as unstable and should be avoided. Why is that? How would network connectivity issues (e.g. sudden communication drops) influence this behavior? Linux kernel version should be quite recent.
#StephenC so after some more testing, when calling RandomAccessFile.getChannel().tryLock() from a java main method it works fine over nfs4 but when the same code runs within Tomcat (8.5.68) multi-locking occurs.
OK. So I think I understand the root of your problem now. From what you have said, it sounds to me like you have are trying to use FileLock to stop one thread of your Tomcat JVM from locking a section of a file while another Tomcat thread has it locked.
That's not going to work.
The lock that you are using is a FileLock. A key paragraph of the javadoc states this:
File locks are held on behalf of the entire Java virtual machine. They are not suitable for controlling access to a file by multiple threads within the same virtual machine.
In this case, "not suitable" means "doesn't work".
If you drill down to the Linux manual page for flock (2) (which is used by Java to implement these locks), you will see that the semantics are defined in terms of multiple processes, not multiple threads. For example:
LOCK_EX Place an exclusive lock. Only one process may hold a shared lock for a given file at a given time.
and
A call to flock() may block if an incompatible lock is held by another process.
So, in summary, it is still not Java's fault. You are trying to use FileLock in a way that Java doesn't support ... and could not support, given how Linux (and indeed POSIX) flock is specified.
(IMO, all of the stuff about NFS is a red herring. The above problem is not caused by NFS. The reason that it shows up on an NFS file system, is that NFS operations take longer and therefore the time window for overlapping operations on the same file is much larger. And if your customer's use-case is hammering their NFS ...)
(But if I am wrong and NFS is implicated, then your "main vs Tomcat" observation is inexplicable. The JVM will not be doing file locking differently in those two cases: it will be using the same OpenJDK code in both cases. Furthermore, the JVM won't even be aware that it is talking to an NFS file system. You can take a look at the OpenJDK codebase if you don't believe me. It's not that hard ...)
See also:
How to lock file for another threads
Is FileLock in java safe across multiple threads within the same process or between different processes or both?
and so on.
I found the root cause of this issue. It seems that when two different threads of the same JVM create a RandomAccessFile object on the same file, calling RandomAccessFile.close from one thread, releases the lock the other thread has.
The documentation of RandomAccessFile.close says
Closes this random access file stream and releases any system resources associated with the stream.
I'm not sure if this means that the system resources are released on the JVM level. I would expect that each RandomAccessFile object would get its own stream and release only that one but it seems that is not the case (or at least the lock on that file gets released. Note that this behavior has not been observed on Windows systems.
Many times I met the statement that the application should always explicitly close all the resources that it opened.
My approach to programming is rather pragmatic and I don't like to blindly follow any convention that I don't clearly see benefits of. Hence my question.
Let's assume that:
I have a small application
It opens a few resources (e.g. files, database connections, remote streams) and processes it
It works a few minutes and then it exits
Let's say it's in Java (if the language is relevant)
Do I really have to care about closing all the resources that I opened? I guess all the resources I opened will be closed/released when the application/virtual machine exits. Am I right?
If that's true, are there any convincing reasons to care about closing resources in such small, short working application?
UPDATE:
The question is purely hypothetical, but the argument for not caring about that is that I may be just hacking together some quick script and don't want to write any unnecessary code not directly related to the problem at hand: closing resources, doing all this verbose try-catch-finally stuff, handling exceptions that I don't care about etc.
The point of the question is whether there are any practical consequences of not doing it.
I guess all the resources I opened will be closed/released when the application/Virtual machine exits.
What happens with a resource which was not regurarly released is out of your control. It may do no harm, or it may do some. It is also highly platform-dependent, so testing on just one won't help.
why should I care about closing these resources in such small, short working application?
The size of the application shouldn't matter. First, applications usually grow; second, if you don't practice doing it the right way, you won't know how to do it when it matters.
If you not close the resources ,that may leads to application servers being frequently restarted when resource exhaustion occurs.because operating systems and server
applications generally have an upper-bound limit for resources
According to docs
The typical Java application manipulates several types of resources such as
files,streams, sockets,and database connections.Such resources must be
handled with great care, because they acquire system resources for their
operations.Thus, you need to ensure that they get freed even in case of errors.
Indeed, incorrect resource management is a common source of failures in
production applications,with the usual pitfalls being database connections
and file descriptors remaining opened after an exception has occurred
somewhere else in the code.This leads to application servers being frequently
restarted when resource exhaustion occurs,because operating systems and server
applications generally have an upper-bound limit for resources.
try-with-resources Statement introduced in java 7 for the programmers who hates close statements.
Short answer - Yes. For one, it's TERRIBLE coding practice just as it is in every other area of life to not clean up after yourself. For another, you can't predict whether the operating system is going to recognize that the java environment no longer needs the resources and you could end up having locks on files/etc that can't be released without a forced restart.
Always clean up whatever resources you open!
Update regarding your update to the original question - it takes 5 seconds to add a try/catch block to close any open resources, and can prevent you having to spend 5 minutes restarting your computer. Doing it right always saves time in the end. My dad always told me the true lazy person does things right the first time so they don't have to come back and do it again. I just say don't be lazy and do it right. The 5 seconds it takes to write a catch block will never slow down the writing process significantly... the 5 seconds you save by not writing it could slow down your debugging immensely.
I'm trying to develop an application that just before quit has to run a new daemon process to execute the main method of a class.
I require that after the main application quits the daemon process must still be in execution.
It is a Java Stored Procedure running on Oracle DB so I can't use Runtime.exec because I can't locate the java class from the Operating System Shell because it's defined in database structures instead of file system files.
In particular the desired behavior should be that during a remote database session I should be able to
call the first java method that runs the daemon process and quits leaving the daemon process in execution state
and then (having the daemon process up and the session control, because the last call terminated) consequentially
call a method that communicates with the daemon process (that finally quits at the end of the communication)
Is this possible?
Thanks
Update
My exact need is to create and load (reaching the best performances) a big text file into the database supposing that the host doesn't have file transfer services from a Java JDK6 client application connecting to Oracle 11gR1 DB using JDBC-11G oci driver.
I already developed a working solution by calling a procedure that stores into a file the LOB(large database object) given as input, but such a method uses too many intermediate structures that I want to avoid.
So I thought about creating a ServerSocket on the DB with a first call and later connect to it and establish the data transfer with a direct and fast communication.
The problem I encountered comes out because the java procedure that creates the ServerSocket can't quit and leave an executing Thread/Process listening on that Socket and the client, to be sure that the ServerSocket has been created, can't run a separate Thread to handle the rest of the job.
Hope to be clear
I'd be surprised if this was possible. In effect you'd be able to saturate the DB Server machine with an indefinite number of daemon processes.
If such a thing is possible the technique is likely to be Oracle-specific.
Perhaps you could achieve your desired effect using database triggers, or other such event driven Database capabilities.
I'd recommend explaining the exact problem you are trying to solve, why do you need a daemon? My instict is that trying to manage your daemon's life is going to get horribly complex. You may well need to deal with problems such as preventing two instances being launched, unexpected termination of the daemon, taking daemon down when maintenance is needed. This sort of stuff can get really messy.
If, for example, you want to run some Java code every hour then almost certanly there are simpler ways to achieve that effect. Operating systems and databases tend to have nice methods for initiating work at desired times. So having a stored procedure called when you need it is probably a capability already present in your environment. Hence all you need to do is put your desired code in the stored procedure. No need for you to hand craft the shceduling, initiation and management. One quite significant advantage of this approach is that you end up using a tehcnique that other folks in your environment already understand.
Writing the kind of code you're considering is very intersting and great fun, but in commercial environments is often a waste of effort.
Make another jar for your other Main class and within your main application call the jar using the Runtime.getRuntime().exec() method which should run an external program (another JVM) running your other Main class.
The way you start subprocesses in Java is Runtime.exec() (or its more convenient wrapper, ProcessBuilder). If that doesn't work, you're SOL unless you can use native code to implement equivalent functionality (ask another question here to learn how to start subprocesses at the C++ level) but that would be at least as error-prone as using the standard methods.
I'd be startled if an application server like Oracle allowed you access to either the functionality of starting subprocesses or of loading native code; both can cause tremendous mischief so untrusted code is barred from them. Looking over your edit, your best approach is going to be to rethink how you tackle your real problem, e.g., by using NIO to manage the sockets in a more efficient fashion (and try to not create extra files on disk; you'll just have to put in extra elaborate code to clean them up…)
I have a system that caches the tiny/simple results of an on-startup SOAP call
I need instances to be able to reload their cache on startup (in case the SOAP service is dead) and ALSO handle the possibility of multiple instances using this cache file
I chose to use java.util.prefs but Java's builtin automatic sync thread is intermittently failing (1% of time using default JVM 30s backing store sync) dumping the following the exception:
Jan 8, 2010 12:30:07 PM java.util.prefs.FileSystemPreferences syncWorld
WARNING: Couldn't flush user prefs: java.util.prefs.BackingStoreException: Couldn't get file lock.
I suspected this bug but this was fixed in 1.5(tiger-b40) and our java 5 on this box is "1.5.0_16-b02".
I now suspect that it might be because we have multiple JVMs sharing this Backing Store, although this doesn't seem to happen on our other machines.
Can anyone confirm this?
What are the risks, if any?
If my approach is flawed what should I be using as an alternative?
"I now suspect that it might be because we have multiple JVMs sharing this Backing Store"
This could absolutely be the case!
If two JVMs attempt to lock the file at the same then this is what you'll see.
The exact details will depend on the type of lock, operating system and file system.
You might want to try wrapping the operation that causes this in a try/catch block, then retry the operation if it fails.
I ran into the same issue with jetty. I found the following fixed the issue.
Add a .systemPrefs to your JRE directory and provide access to the user who is running the process which is complaining.
Once that is done, go to the Jetty directory and open the start.ini file
-Djava.util.prefs.userRoot={user's home directory}
-Djava.util.prefs.systemRoot={user's home directory}
Once finished adding those lines I restarted jetty and found that the errors were gone.
Instead of using Preferences, just use any serializable Map and make a very simple cache class which serializes and deserializes it to a randomly-generated temporary filename (the filename being generated on first initialisation). Since it is only a cache, you can just catch any exceptions and reset the cache back to its initial state when an exception happens, so it will re-fetch from the original data source (the SOAP service in your case). So there's no need to worry about serialVersionUID or any of that compatibility stuff, if you don't want to.
What happens when you concurrently open two (or more) FileOutputStreams on the same file?
The Java API says this:
Some platforms, in particular, allow a file to be opened for writing by only one FileOutputStream (or other file-writing object) at a time.
I'm guessing Windows isn't such a platform, because I have two threads that read some big file (each one a different one) then write it to the same output file. No exception is thrown, the file is created and seems to contain chunks from both input files.
Side questions:
Is this true for Unix, too?
And since I want the behaviour to be the same (actually I want one thread to write correctly and the other to be warned of the conflict), how can I determine that the file is already opened for writing?
There's not a reliable, cross-platform way to be passively notified when a file has another writer—i.e., raise an exception if a file is already open for writing. There are a couple of techniques that help you actively check for this, however.
If multiple processes (which can be a mix of Java and non-Java) might be using the file, use a FileLock. A key to using file locks successfully is to remember that they are only "advisory". The lock is guaranteed to be visible if you check for it, but it won't stop you from doing things to the file if you forget. All processes that access the file should be designed to use the locking protocol.
If a single Java process is working with the file, you can use the concurrency tools built into Java to do it safely. You need a map visible to all threads that associates each file name with its corresponding lock instance. The answers to a related question can be adapted easily to do this with File objects or canonical paths to files. The lock object could be a FileOutputStream, some wrapper around the stream, or a ReentrantReadWriteLock.
I would be wary of letting the OS determine file status for you (since this is OS-dependent). If you've got a shared resource I would restrict access to it using a Re-entrant lock
Using this lock means one thread can get the resource (file) and write to it. The next thread can check for this lock being held by another thread, and/or block indefinitely until the first thread releases it.
Windows (I think) would restrict two processes writing to the same file. I don't believe Unix would do the same.
If the 2 threads you are talking about are in the same JVM, then you could have a boolean variable somewhere that is accessed by both threads.
Unix allows concurrent writers to the same file.
You shouldn't be attempting to write to the same file more than once. If you are you have a design flaw.