The story:
A few days ago I was thinking about inter-process communication based on file exchange. Say process A creates several files during its work and process B reads these files afterwards. To ensure that all files were correctly written, it would be convenient to create a special file, which existence will signal that all operations were done.
Simple workflow:
process A creates file "file1.txt"
process A creates file "file2.txt"
process A creates file "processA.ready"
Process B is waiting until file "processA.ready" appears and then reads file1 and file2.
Doubts:
File operations are performed by the operating system, specifically by the file subsystem. Since implementations can differ in Unix, Windows or MacOS, I'm uncertain about the reliability of file exchange inter-process communication. Even if OS will guarantee this consistency, there are things like JIT compiler in Java, which can reorder program instructions.
Questions:
1. Are there any real specifications on file operations in operating systems?
2. Is JIT really allowed to reorder file operation program instructions for a single program thread?
3. Is file exchange still a relevant option for inter-process communication nowadays or it is unconditionally better to choose TCP/HTTP/etc?
You don’t need to know OS details in this case. Java IO API is documented to guess whether file was saved or not.
JVM can’t reorder native calls. It is not written in JMM explicitly but it is implied that it can’t do it. JVM can’t guess what is impact of native call and reordering of those call can be quite generous.
There are some disadvantages of using files as a way of communication:
It uses IO which is slow
It is difficult to separate processes between different machines in case you would need it (there are ways using samba for example but is quite platform-dependant)
You could use File watcher (WatchService) in Java to receive a signal when your .ready file appears.
Reordering could apply but it shouldn't hurt your application logic in this case - refer the following link:
https://assylias.wordpress.com/2013/02/01/java-memory-model-and-reordering/
I don't know the size of your data but I feel it would still be better to use an Message Queue (MQ) solution in this case. Using a File IO is a relatively slow operation which could slow down the system.
Used file exchange based approach on one of my projects. It's based on renaming file extensions when a process is done so other process can retrieve it by file name expression checking.
FTP process downloads a file and put its name '.downloaded'
Main task processor searched directory for the files '*.downloaded'.
Before starting, job updates file name as '.processing'.
When finished then updates to '.done'.
In case of error, it creates a new supplemantary file with '.error' extension and put last processed line and exception trace there. On retries, if this file exists then read it and resume from correct position.
Locator process searches for '.done' and according to its config move to backup folder or delete
This approach is working fine with a huge load in a mobile operator network.
Consideration point is to using unique names for files is important. Because moving file's behaviour changes according to operating system.
e.g. Windows gives error when there is same file at destination, however unix ovrwrites it.
Related
I have looked at examples that tell best practices for file write/create operations but have not seen an example that takes into consideration my requirements. I have to create a class which reads the contents of 1 file, does some data transformation, and then write the transformed contents to a different file then sends the file to a web service. Both files ultimately can be quite large like up to 20 MB and also it is unpredictable when these files will be created because they are generated by the user. Therefore it could be like 2 minutes between the time when this process occurs or it could be several all in the same second. The system is not like crazy in the sense that it could be like hundreds of these operations in the same second but it could be several.
My instinct says to solve it by:
Creating a separate thread when the process begins.
Read the first file.
Do the data transformation.
Write the contents to the new file.
Send the file to the service.
Delete the created file.
Am I missing something? Is there a best practice to tackle this kind of issue?
The first question you should ask is weather you need to write the file to the disk in the first place. Even if you are supposed to send a file to a consumer at the end of your processing phase, you could keep the file contents in memory and send that. The consumer doesn't care weather the file is stored on disk or not, since it only receives an array of bytes with the file contents.
The only scenario in which it would make sense to store the file on disk would be if you would communicate between your processes via disk files (i.e. your producer writes a file to disk, sends some notification to your consumer and afterwards your consumer reads the file from disk - for example based on a file name it receives from the notification).
Regarding I/O best practices, make sure you use buffers to read (and potentially write) files. This could greatly reduce the memory overhead (since you would end up keeping only a chunk instead of the whole 20 MB file in memory at a given moment).
Regarding adding multiple threads, you should test weather that improves your application performance or not. If your application is already I/O intensive, adding multiple threads will result in adding even more contention on your I/O streams, which would result in a performance degradation.
Without the full details of the situation, a problem like this may be better solved with existing software such as Apache NiFi:
An easy to use, powerful, and reliable system to process and distribute data.
It's very good at picking up files, transforming them, and putting them somewhere else (and sending emails, and generating analytics, and...). NiFi is a very powerful tool, but may be overkill if you're needs are just a couple of files given the additional set-up.
Given the description you have given, I think you should perform the operations for each file on one thread; i.e. on thread will download the file, process it and then upload the results.
If you need parallelism, then implement the download / process / upload as a Runnable and submit the tasks to an ExecutorService with a bounded thread pool. And tune the size of the thread pool. (That's easy if you expose the thread pool size as a config property.)
Why this way?
It is simple. Minimal synchronization is required.
One of the three subtasks is likely to be your performance bottleneck. So by combining all three into a single task, you avoid the situation where the non-bottleneck tasks get too far ahead. And if you get too far ahead on some of the subtasks you risk running out of (local) disk space.
I'm going to contradict what Alex Rolea said about buffering. Yes, it may help. But in on a modern (e.g. Linux) operating system on a typical modern machine, memory <-> disk I/O is unlikely to be the main bottleneck. It is more likely that the bottleneck will be network I/O or server-side I/O performance (especially if the server is serving other clients at the same time.)
So, I would not prematurely tune the buffering. Get the system working, benchmark it, profile / analyze it, and based on those results figure out where the real bottlenecks are and how best to address them.
Part of the solution may be to not use disk at all. (I know you think you need to, but unless your server and its protocols are really strange, you should be able to stream the data to the server out of memory on the client side.)
Short version: Why should File.createNewFile() not be used for file locking? Or more specifically: Are there issues if it is used to lock an applications data directory?
Details:
I would like to protect my applications data directory using a lock file: If the file lock exists, the directory is locked and the application exits with an error message. If it does not exist it will be created and the application continues. On exit the file will be deleted.
The lock will not be created that often (i.e. performance is not an issue) and I have no problem with manually deleting the lock file in case of some error (i.e. failing to delete the file is not an issue).
The code looks something like this:
File lockFile = new File("lock");
boolean lockCreated = lockFile.createNewFile();
if (lockCreated)
{
// do stuff
lockFile.delete();
}
else
{
System.err.println("Lockfile exists => please retry later");
// alternative: Wait and retry e.g. 5 times
}
Now I'm a bit confused about the Javadoc of createNewFile():
Atomically creates a new, empty file named by this abstract pathname if and only if a file with this name does not yet exist. The check for the existence of the file and the creation of the file if it does not exist are a single operation that is atomic with respect to all other filesystem activities that might affect the file.
Note: this method should not be used for file-locking, as the resulting protocol cannot be made to work reliably. The FileLock facility should be used instead.
What are the potential problems mentioned in the note, considering the existence check and file creation are atomic?
This forum post from December 2007 indicates there are "significant platform differences" according to the Javadoc of File.delete() (although I cannot find such a statement since at least Java SE 1.4.2). But even if there would be such differences: Could they really cause the locking to fail (i.e. two processes think the data directory is usable at the same time)?
Note: I do not want any of the following:
Lock a file so that no other process can access and/or modify it (most information I found seems to discuss this issue).
Make sure no other process can remove the lock.
Synchronize multiple threads of the same JVM (although I think my solution should be able to handle that too).
The Javadoc of Files.createFile(…), part of java.nio.file available since Java 7, repeats the promise of atomicity but does not mention anything about file based locking.
My reasoning:
Either the newer method (from java.nio.file.Files) is affected by the same (or similar) problems as the older one (from java.io.File) and the Javadoc is simply missing this information…
… or the newer method actually behaves more predictably and correct.
Given the error handling and specification in java.nio.file has generally been improved compared to the File class (existing ever since JDK 1.2), I assume the second alternative is correct.
My conclusion: Using Files.createFile(…) is fine for this use case.
The short answer: reliable file based locking in Java is not practical.
The long answer: The issue with file based locking, in any OS, always comes down to what kind of storage system the file comes from. Almost all network accessed file systems (NFS, SAMBA, etc) have very unreliable (or at least unpredictable) synchronizations on file creates or deletes that make a general Java-ish approach inadvisable. In certain OSes, using local file systems, you can sometimes get what you desire. But you need to understand the underlying file system and its characteristics and proceed with care.
Short version: Why should File.createNewFile() not be used for file locking? Or more specifically: Are there issues if it is used to lock an applications data directory?
Details:
I would like to protect my applications data directory using a lock file: If the file lock exists, the directory is locked and the application exits with an error message. If it does not exist it will be created and the application continues. On exit the file will be deleted.
The lock will not be created that often (i.e. performance is not an issue) and I have no problem with manually deleting the lock file in case of some error (i.e. failing to delete the file is not an issue).
The code looks something like this:
File lockFile = new File("lock");
boolean lockCreated = lockFile.createNewFile();
if (lockCreated)
{
// do stuff
lockFile.delete();
}
else
{
System.err.println("Lockfile exists => please retry later");
// alternative: Wait and retry e.g. 5 times
}
Now I'm a bit confused about the Javadoc of createNewFile():
Atomically creates a new, empty file named by this abstract pathname if and only if a file with this name does not yet exist. The check for the existence of the file and the creation of the file if it does not exist are a single operation that is atomic with respect to all other filesystem activities that might affect the file.
Note: this method should not be used for file-locking, as the resulting protocol cannot be made to work reliably. The FileLock facility should be used instead.
What are the potential problems mentioned in the note, considering the existence check and file creation are atomic?
This forum post from December 2007 indicates there are "significant platform differences" according to the Javadoc of File.delete() (although I cannot find such a statement since at least Java SE 1.4.2). But even if there would be such differences: Could they really cause the locking to fail (i.e. two processes think the data directory is usable at the same time)?
Note: I do not want any of the following:
Lock a file so that no other process can access and/or modify it (most information I found seems to discuss this issue).
Make sure no other process can remove the lock.
Synchronize multiple threads of the same JVM (although I think my solution should be able to handle that too).
The Javadoc of Files.createFile(…), part of java.nio.file available since Java 7, repeats the promise of atomicity but does not mention anything about file based locking.
My reasoning:
Either the newer method (from java.nio.file.Files) is affected by the same (or similar) problems as the older one (from java.io.File) and the Javadoc is simply missing this information…
… or the newer method actually behaves more predictably and correct.
Given the error handling and specification in java.nio.file has generally been improved compared to the File class (existing ever since JDK 1.2), I assume the second alternative is correct.
My conclusion: Using Files.createFile(…) is fine for this use case.
The short answer: reliable file based locking in Java is not practical.
The long answer: The issue with file based locking, in any OS, always comes down to what kind of storage system the file comes from. Almost all network accessed file systems (NFS, SAMBA, etc) have very unreliable (or at least unpredictable) synchronizations on file creates or deletes that make a general Java-ish approach inadvisable. In certain OSes, using local file systems, you can sometimes get what you desire. But you need to understand the underlying file system and its characteristics and proceed with care.
I am using a file and need to update value in java when file is modified.
So, I am thinking to check modified time using lastModified of File class, and if modified read the file and update single property from the file.
My doubt is, is lastModified as heavy as reading single property from the file/reading whole file. Because my test results are showing almost same results.
So is it better to read file and update property from the file everytime or checking lastModified is better option in long run.
Note: This operation is performed every one minute.
Or is there any better option than polling lastModified to check if file has changed. I am using java 6.
Because you are using Java 6, checking the modified date or file contents is your only option (there's another answer that discusses using the newer java.nio.file functionality, and if you have the option of moving to Java 7, you should really, really consider that).
To answer your original question:
You didn't specify the location of the file (i.e. is it on a local disk or a server somewhere else) - I'll respond assuming local disk, but if the file is on a different machine, network latencies and netbios/dfs/whatever-network-file-system-you-use will exacerbate the differences.
Checking modified date on a file involves reading the meta data from disk. Checking the contents of the file require reading the file contents from disk (if the file is small, this will be one read operation. If the file is larger, it could be multiple read operations).
Reading the content of the file will probably involve read/write lock checking. Generally speaking, checking the modified date on the file will not require read/write lock checking (depending on the file system, there may still be consistency locks occurring on the meta data disk page, but those are generally lighter weight than file locks).
If the file changes frequently (i.e. you actually expect it to change every minute), then checking the modified date is just overhead - you are going to read the file contents in most cases anyway. If the file doesn't change frequently, then there would definitely be an advantage to modified date checking if the file was large (and you had to read the entire file to get at your information).
If the file is small, and doesn't change frequently, then it's pretty much a wash. In most cases, the file contents and the file meta data are already going to be paged into RAM - so both operations are a relatively efficient check of contents in RAM.
I personally would do the modified date check just b/c it logically makes sense (and it protects you from the performance hit if the file size ever grows above one disk page) - but if the file changes frequently, then I'd just read the file contents. But really, either way is fine.
And that brings us to the unsolicited advice: my guess is that the performance on this operation isn't a big deal in the greater scheme of things. Even if it took 1000X longer than it does now, it probably still wouldn't impact your application's primary purpose/performance. So my real advice here is just write the code and move on - don't worry about it's performance unless this becomes a bottleneck for your application.
Quoting from The JAVA Tutorials
To implement this functionality, called file change notification, a program must be able to detect what is happening to the relevant directory on the file system. One way to do so is to poll the file system looking for changes, but this approach is inefficient. It does not scale to applications that have hundreds of open files or directories to monitor.
The java.nio.file package provides a file change notification API, called the Watch Service API. This API enables you to register a directory (or directories) with the watch service. When registering, you tell the service which types of events you are interested in: file creation, file deletion, or file modification. When the service detects an event of interest, it is forwarded to the registered process. The registered process has a thread (or a pool of threads) dedicated to watching for any events it has registered for. When an event comes in, it is handled as needed.`
Here are some links which provide some sample source on implementation of this service:-
Link 1
Link 2
Edit:- Thanks to Kevin Day for pointing out in comments, since you are using java 6 this might not work for you. Although there is an alternative available in Apache Commons IO . But have not worked with it, so you have to check it yourself :)
Is there any way in java to read a file's content, which is being updated by another handler before closing it?
That depends on the operating systems.
Traditionally, POSIX-y operating systems (Linux, Solaris, ...) have absolutely no problem with having a file open for both reading and writing, even by separate processes (they even support deleting a file while it's being read from and/or written to).
In Windows, the more common approach is to open files exclusively (contrary to common believe, Windows does support non-exclusive file access, it's just rarely used by applications).
Java has no way* of specifying what way you want to access a file, so the platform default is used (shared access on Linux/Solaris, exclusive access on Windows).
* This might be wrong for NIO and new NIO in Java 7, but I'm not a big NIO expert.
In theory its quite easy to do, however files are not designed to exchange data this way and depending on your requirements it can be quite tricky to get right. This is why there is no general solution for this.
e.g. if you want to read a file as another process writes to it, the reading thread will see an EOF even though the writer hasn't finished. You have to re-open the file and skip to where the file was last read and continue. The writing thread might roll the files it is writing meaning the reading has to detect this and handle it.
What specificity do you want to do?