I've been trying to use FileLock to get exclusive access to a file in order to:
delete it
rename it
write to it
Because on Windows (at least) it seems that you cannot delete, rename, or write to a file that is already in use. The code I've written looks something like this:
import java.io.File;
import java.io.FileNotFoundException;
import java.io.IOException;
import java.io.RandomAccessFile;
import java.nio.channels.FileChannel;
import java.nio.channels.FileLock;
public abstract class LockedFileOperation {
public void execute(File file) throws IOException {
if (!file.exists()) {
throw new FileNotFoundException(file.getAbsolutePath());
}
FileChannel channel = new RandomAccessFile(file, "rw").getChannel();
try {
// Get an exclusive lock on the whole file
FileLock lock = channel.lock();
try {
doWithLockedFile(file);
} finally {
lock.release();
}
} finally {
channel.close();
}
}
public abstract void doWithLockedFile(File file) throws IOException;
}
Here are some unit tests that demonstrate the problem. You'll need to have Apache commons-io on your classpath to run the 3rd test.
import java.io.File;
import java.io.IOException;
import junit.framework.TestCase;
public class LockedFileOperationTest extends TestCase {
private File testFile;
#Override
protected void setUp() throws Exception {
String tmpDir = System.getProperty("java.io.tmpdir");
testFile = new File(tmpDir, "test.tmp");
if (!testFile.exists() && !testFile.createNewFile()) {
throw new IOException("Failed to create test file: " + testFile);
}
}
public void testRename() throws IOException {
new LockedFileOperation() {
#Override
public void doWithLockedFile(File file) throws IOException {
if (!file.renameTo(new File("C:/Temp/foo"))) {
fail();
}
}
}.execute(testFile);
}
public void testDelete() throws IOException {
new LockedFileOperation() {
#Override
public void doWithLockedFile(File file) throws IOException {
if (!file.delete()) {
fail();
}
}
}.execute(testFile);
}
public void testWrite() throws IOException {
new LockedFileOperation() {
#Override
public void doWithLockedFile(File file) throws IOException {
org.apache.commons.io.FileUtils.writeStringToFile(file, "file content");
}
}.execute(testFile);
}
}
None of the tests pass. The first 2 fail, and the last throws this exception:
java.io.IOException: The process cannot access the file because another process has locked a portion of the file
at java.io.FileOutputStream.writeBytes(Native Method)
at java.io.FileOutputStream.write(FileOutputStream.java:247)
at org.apache.commons.io.IOUtils.write(IOUtils.java:784)
at org.apache.commons.io.IOUtils.write(IOUtils.java:808)
at org.apache.commons.io.FileUtils.writeStringToFile(FileUtils.java:1251)
at org.apache.commons.io.FileUtils.writeStringToFile(FileUtils.java:1265)
It seems like the lock() method places a lock on the file which then prevents me from renaming/deleting/writing it. My assumption was that locking the file would give me exclusive access to the file, so I could then rename/delete/write it without worrying about whether any other process is also accessing it.
Either I'm misunderstanding FileLock or it's not an appropriate solution for my problem.
The message about another process just means that some process on your system has the file open. It does not actually check that that process happens to be the same as the one attempting to delete/rename the file. In this case, the same program has the file opened. You have opened it to get the lock. The lock here has little to no value, especially if you are doing this for delete or rename operations.
To do what you want, you would need to lock the directory entry. This is not available in Java and may not be available in Windows. These (delete and insert) operations are atomic. That means that the operating system takes care of locking the directory and other file system structures for you. If another process (or your own) has the file open then these operations will fail. If you are trying to lock the file exclusively (directory entry) and another process (or your own) has the file open, then the lock will fail. There is no difference, but attempting to do the lock just complicates, and in this case, makes the operation impossible (that is, the files are always opened before you attempt to do the operation).
Now writing to the file is a valid lock operation. Lock the file or portion of the file that you want to write to and then it will work. On Windows, this lock mechanism is mandatory so another open/file descriptor will not be able to write to any portion that is under the lock.
EDIT
According to the JavaDoc on FileChannel.lock, it is the same as calling FileChannel.lock(0L, Long.MAXVALUE, false). This is an exclusive lock on a region from the first byte to the last.
Second, according to JavaDoc on FileLock
Whether or not a lock actually prevents another program from accessing the content of the locked region is system-dependent and therefore unspecified. The native file-locking facilities of some systems are merely advisory, meaning that programs must cooperatively observe a known locking protocol in order to guarantee data integrity. On other systems native file locks are mandatory, meaning that if one program locks a region of a file then other programs are actually prevented from accessing that region in a way that would violate the lock. On yet other systems, whether native file locks are advisory or mandatory is configurable on a per-file basis. To ensure consistent and correct behavior across platforms, it is strongly recommended that the locks provided by this API be used as if they were advisory locks.
EDIT
For the testWrite method. The JavaDoc on the commons I/O static method is sparse but says "Writes a String to a file creating the file if it does not exist.." and being as this method takes a File instead of an opened stream, it likely opens the file internally. Probably it is not opening the file with shared access and also opening for append access. This means that the existing open and lock (your open to get the channel from which to get the lock) are blocking that use. To understand even more, you would need to get the source for that method and look at what it is doing.
EDIT
Sorry, I stand corrected. I checked the Windows API and file locking is mandatory on Windows. This is why the write fails. The first open (your new RandomAccessFile) and lock has the file locked. The open to write the string succeeds but the write fails because another open (file descriptor) has the full extent of the file under mandatory exclusive lock - that is, no other file descriptor can write to the file until the lock is released.
Note that locking is associated with the file descriptor NOT process or thread.
The lock you've for is locking a region inside a file, but not the file itself, so while region is locked you can't delete or rename the file.
You may want to look at the Commons Transaction project.
The delete and rename operations are performed by the operating system and are atomic (on most operating systems), so no locking is required.
To write a string to file, it would be simpler to write to a temporary file first (e.g. foo.tmp) and then rename it once it is ready.
Java file locks are specified only to protect against other locks, and nothing else. How they behave on specific platforms, i.e. any extra semantics, is platform-specific.
You should release the file with method release() before doing any action like rename or delete or ....
Related
i want to write to a File, that is already in use by an other process.
My Question is: is there a way to avoid the FilenotfoundException, and to write to the file although it is in use?
Thank you!
From the Oracle docs, do the following check:
boolean canWrite() - Returns true if a file is writable by the current application; false otherwise.
boolean canRead() - Returns true if a file is readable by the current application; false otherwise.
If you are asking if you can write to the file while it is in use by another application, you might be able to force it, but it depends on the behaviour of the other application.
If the other process acquired a lock on the file your application want to use, you have to ensure the application waits for a specific amount of time until the lock gets released. The locking mechanism is purely depends up on the other process that uses the file.
You may try something like this
FileInputStream fin = new FileInputStream("test.txt");
FileChannel channel = fin.getChannel();
FileLock lock = channel.tryLock();
if (lock != null) {
try {
// read the file
} finally {
lock.release();
}
} else {
// some other process has locked the file for some reason
}
I have implemented my own class to read pcap files. (Binary files, i.e. tcpdump, wireshark)
public class PcapReader implements Iterator<PcapPacket> {
private InputStream is;
public PcapReader (File file) throws FileNotFoundException, IOException {
is = this(new DataInputStream(
new BufferedInputStream(
new FileInputStream(file))));
}
#Override
public boolean hasNext () {
try {
return (is.available() > 0);
} catch (IOException e) {
return false;
}
}
//pseudo code!
#Override
public PcapPacket next () {
is.read(header);
is.read(body);
return new PcapPacket(header, body);
}
//more code here
}
Then I use it like this:
PcapReader reader = new PcapReader(file);
while (reader.hasNext()) {
PcapPacket pcapPacket = reader.next();
//process packet
}
The file under test has 190 Mb. And I also use JVisualVM to profile.
hasNext() is called 1.7 million times and time is 7.7 seconds
next() is called same number of times and time is 3.6 seconds
My main question is why hasNext() is so time consuming in absolute value and also twice greater than next?
When you call is.available(), in your hasNext() method, it goes down to FileInputStream.available() implementation. This is a native method, as one may see from FileInputStream source code.
In the end, this is indeed a time-consumming operation, as the Operating System implementation of the file operations will have to check ahead if more data is available to be read. So, it will actually do a read operation without updating the file pointer (or updating it back to the original position), just to check if there is a "next" byte.
I'm sure, that internal (native) implementation of available() method is not something like just returning some return availableSize;, but more complicated. Stream counts available data using OS API; especially, for example, for log files, which are written due Stream reads them.
I have implemented my own class to read pcap files.
Because you're not using jNetPcap, or because you are using jNetPcap but need something that can read from a File?
If the latter, you probably want to use a pattern other than one that has a "more data is available" method and a separate "so read that data" method; something that reads the data and either returns a "packet available"/"end of file"/"error" indication or throws an exception for one or both of the latter conditions (DataInputStream appears to throw exceptions for both I/O errors and EOF, so it might make sense to do the same for your class).
Yeah, that means it can't be an Iterator, but maybe Iterators weren't originally intended to represent records in a sequential file (besides, if you really want it to be an Iterator, what are you going to do about the remove method?).
And if you can avoid needing to read from a File, you could then use jNetPcap's own routines for reading capture files, which, in libpcap 1.1.0 and later, can also read some pcap-ng files.
I'm trying to delete a file that another thread within my program has previously worked with.
I'm unable to delete the file but I'm not sure how to figure out which thread may be using the file.
So how do I find out which thread is locking the file in java?
I don't have a straight answer (and I don't think there's one either, this is controlled at OS-level (native), not at JVM-level) and I also don't really see the value of the answer (you still can't close the file programmatically once you found out which thread it is), but I think you don't know yet that the inability to delete is usually caused when the file is still open. This may happen when you do not explicitly call Closeable#close() on the InputStream, OutputStream, Reader or Writer which is constructed around the File in question.
Basic demo:
public static void main(String[] args) throws Exception {
File file = new File("c:/test.txt"); // Precreate this test file first.
FileOutputStream output = new FileOutputStream(file); // This opens the file!
System.out.println(file.delete()); // false
output.close(); // This explicitly closes the file!
System.out.println(file.delete()); // true
}
In other words, ensure that throughout your entire Java IO stuff the code is properly closing the resources after use. The normal idiom is to do this in the try-with-resources statement, so that you can be certain that the resources will be freed up anyway, even in case of an IOException. E.g.
try (OutputStream output = new FileOutputStream(file)) {
// ...
}
Do it for any InputStream, OutputStream, Reader and Writer, etc whatever implements AutoCloseable, which you're opening yourself (using the new keyword).
This is technically not needed on certain implementations, such as ByteArrayOutputStream, but for the sake of clarity, just adhere the close-in-finally idiom everywhere to avoid misconceptions and refactoring-bugs.
In case you're not on Java 7 or newer yet, then use the below try-finally idiom instead.
OutputStream output = null;
try {
output = new FileOutputStream(file);
// ...
} finally {
if (output != null) try { output.close(); } catch (IOException logOrIgnore) {}
}
Hope this helps to nail down the root cause of your particular problem.
About this question, I also try to find out this answer, and ask this question and find answer:
Every time when JVM thread lock a file exclusively, also JVM lock
some Jave object, for example, I find in my case:
sun.nio.fs.NativeBuffer
sun.nio.ch.Util$BufferCache
So you need just find this locked Java object and analyzed them and
you find what thread locked your file.
I not sure that it work if file just open (without locked exclusively), but I'm sure that is work if file be locked exclusively by Thread (using java.nio.channels.FileLock, java.nio.channels.FileChannel and so on)
More info see this question
How to determine whether a file is using?
In java you can lock Files and checking for shared access.
You can use a file lock to restrict
access to a file from multiple
processes
public class Locking {
public static void main(String arsg[])
throws IOException {
RandomAccessFile raf =
new RandomAccessFile("junk.dat", "rw");
FileChannel channel = raf.getChannel();
FileLock lock = channel.lock();
try {
System.out.println("Got lock!!!");
System.out.println("Press ENTER to continue");
System.in.read(new byte[10]);
} finally {
lock.release();
}
}
}
You also can check whether a lock exists by calling
// Try acquiring the lock without blocking. This method returns
// null or throws an exception if the file is already locked.
try {
lock = channel.tryLock();
} catch (OverlappingFileLockException e) {
// File is already locked in this thread or virtual machine
}
I don't think that Java can tell you whether another process is using a file. Depending on what you're trying to do, you might get an IOException when you try to manipulate it. Otherwise, if you're no Linux, you might want to look at lsof.
I think that what you're asking about is whether the file you're attempting to use is locked. You can use the FileLock class to attempt to lock the file you're interested in. This class is intended to map to the native file-locking facility on the file system being used. If you can lock the file, it's safe to assume that no other process holds a lock on the file.
Here is an easy Solution for Windows Users:
http://www.dr-hoiby.com/WhoLockMe/
Tiny Tool but useful...
This is perhaps similar to previous posts, but I want to be specific about the use of locking on a network, rather than locally. I want to write a file to a shared location, so it may well go on a network (certainly a Windows network, maybe Mac). I want to prevent other people from reading any part of this file whilst it it being written. This will not be a highly concurrent process, and the files will be typically less than 10MB.
I've read the FileLock documentation and File documentation and am left somewhat confused, as to what is safe and what is not. I want to lock the entire file, rather than portions of it.
Can I use FileChannel.tryLock(), and it is safe on a network, or does it depend on the type of network? Will it work on a standard Windows network (if there is such a thing).
If this does not work, is the best thing to create a zero byte file or directory as a lock file, and then write out the main file. Why does that File.createNewFile() documentation say don't use this for file locking? I appreciate this is subject to race conditions, and is not ideal.
This can't be reliably done on a network file system. As long as your application is the only application that accesses the file, it's best to implement some kind of cooperative locking process (perhaps writing a lock file to the network filesystem when you open the file). The reason that is not recommended, however, is that if your process crashes or the network goes down or any other number of issues happen, your application gets into a nasty, dirty state.
You can have a empty file which is lying on the server you want to write to.
When you want to write to the server you can catch the token. Only when you have the token you should write to any file which is lying on the server.
When you are ready with you file operations or an exception was thrown you have to release the token.
The helper class can look like
private FileLock lock;
private File tokenFile;
public SLTokenLock(String serverDirectory) {
String tokenFilePath = serverDirectory + File.separator + TOKEN_FILE;
tokenFile = new File(tokenFilePath);
}
public void catchCommitToken() throws TokenException {
RandomAccessFile raf;
try {
raf = new RandomAccessFile(tokenFile, "rw"); //$NON-NLS-1$
FileChannel channel = raf.getChannel();
lock = channel.tryLock();
if (lock == null) {
throw new TokenException(CANT_CATCH_TOKEN);
}
} catch (Exception e) {
throw new TokenException(CANT_CATCH_TOKEN, e);
}
}
public void releaseCommitToken() throws TokenException {
try {
if (lock != null && lock.isValid()) {
lock.release();
}
} catch (Exception e) {
throw new TokenException(CANT_RELEASE_TOKEN, e);
}
}
Your operations then should look like
try {
token.catchCommitToken();
// WRITE or READ to files inside the directory
} finally {
token.releaseCommitToken();
}
I found this bug report which describes why the note about file locking was added to the File.createNewFile documentation.
http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=4676183
It states:
If you mark the file as deleteOnExit before invoking createNewFile but the file already exists, you run the risk of deleting a file you didn't create and dropping someone elses lock! On the other hand, if you mark the file after creating it, you lose atomicity: if the program exits before the file is marked, it won't get deleted and the lock will be "wedged".
So it looks like the main reason locking is discouraged with File.createNewFile() is that you can end up with orphaned lock files if the JVM unexpectedly terminates before you have a chance to delete it. If you can deal with orphaned lock files then it could be used as a simple locking mechanism. However, I wouldn't recommend the method suggested in the comments of the bug report as it has race conditions around read/writing the timestamp value and reclaiming the expired lock.
Rather than implementing a locking strategy which will, in all likelihood, rely on readers to adhere to your convention but will not force them to, perhaps you can write the file out to a hidden or obscurely named file where it will be effectively invisible to readers. When the write operation is complete, rename the file to the expected public name.
The downside is that hiding and/or renaming without additional IO may require you to use native OS commands, but the procedure to do so should be fairly simple and deterministic.