I have an applet that retrieves a byte array from a backend server. This byte array contains a dynamic library (DLL or SO, depending on which OS the applet is running on), that must be written to disk and subsequently loaded from the disk by a call to System.load().
I need to ensure that the file is not tampered with after it's been written to disk and before it's loaded by the OS through the call to System.load(). I obtain an exclusive lock on the file while it's written to disk, but my testing shows that I must release this lock before the call to System.load(), or it'll fail to load the library.
Is there some way I can keep the lock on the file while I load it?
Sample code:
File f = File.createTempFile("tmp", "");
RandomAccessFile raf = new RandomAccessFile(f, "rwd");
FileChannel channel = raf.getChannel();
FileLock lock = channel.lock(0, Long.MAX_VALUE, false);
// This would be where I write the DLL/SO from a byte array...
raf.write((int)65); // 'A'
raf.write((int)66); // 'B'
raf.write((int)67); // 'C'
System.out.println("Wrote dynamic library to file...");
// Close and release lock
raf.close();
System.out.println("File closed. Lock released.");
// This call fails if the file is still locked.
System.load(f.getAbsolutePath());
Any help is greatly appreciated. The solution (if there is any) must not be native to any OS, but work on all platforms supported by Java. It is also a requirement that the solution be compatible with Java 1.4.
In Java 7 you can implement in-memory file system (see java.nio.file.spi.FileSystemProvider), so library content will be completely in memory thus making attacker's life much harder.
Another possible approach is to sign the library and let OS do security checks after reading file from disk; might be not very portable though.
Most important - is it really the biggest security issue you're facing? The time between 2 calls will be some micro- (ok, maybe milli-) seconds - one must hack really deep into filesystem to do bad things. Wouldn't it be easier to alter file while it is transferred over network? Or don't you think an attacker that advanced may... let's say hack JVM and substitute content while library is written to disk? Nothing is bulletproof, maybe this is the risk you can accept?
Just out of interest - what exactly is the error you're getting?
Related
Linux machine, Java standalone application
I am having the following situation:
I have:
consecutive file write(which creates the destination file and writes some content to it) and file move.
I also have a power outage problem, which instantly cuts off the power of computer during these operations.
As a result, I am getting that the file was created, and it was moved as well, but the file content is empty.
The question is what under the hood can be causing this exact outcome? Considering the time sensitivity, may be hard drive is disabled before the processor and RAM during the cut out, but in that case, how is it possible that the file is created and moved after, but the write before moving is not successful?
I tried catching and logging the exception and debug information but the problem is power outage disables the logging abilities(I/O) as well.
try {
FileUtils.writeStringToFile(file, JsonUtils.toJson(object));
} finally {
if (file.exists()) {
FileUtils.moveFileToDirectory(file, new File(path), true);
}
}
Linux file systems don't necessarily write things to disk immediately, or in exactly the order that you wrote them. That includes both file content and file / directory metadata.
So if you get a power failure at the wrong time, you may find that the file data and metadata is inconsistent.
Normally this doesn't matter. (If the power fails and you don't have a UPS, the applications go away without getting a chance to finish what they were doing.)
However, if it does matter, you can do the following: to force the file to "sync" before you move it:
FileOutputStream fos = ...
// write to file
fs.getFD().sync();
fs.close();
// now move it
You need to read the javadoc for sync() carefully to understand what the method actually does.
You also need to read the javadoc for the method you are using to move the file regarding atomicity.
(This is a hypothetical question since it's very broad, and workarounds exist for specific cases.)
Is it possible to atomically write a byte[] to a file (as FileOutputStream or FileWriter?
If writing fails, then it's unacceptable that part of the array is written. For example, if the array is 1,000,000 bytes and the disk is full after 500,000 bytes, then no bytes should be written to the file, or the changes should somehow be rolled back. This should even be the case if a medium is physically disconnected mid-write.
Assume that the maximum size of the array is known.
Atomic writes to files are not possible. Operating systems don't support it, and since they don't, programming language libraries can't do it either.
The best you are going to get with a files in a conventional file system is atomic file renaming; i.e.
write new file into same file system as the old one
use FileDescriptor.sync() to ensure that new file is written
rename the new file over the old one; e.g. using
java.nio.file.Files.move(Path source, Path target,
CopyOption... options)
with CopyOptions ATOMIC_MOVE. According to the javadocs, this may not be supported, but if it isn't supported you should get an exception.
But note that the atomicity is implemented in the OS, and if the OS cannot give strong enough guarantees, you are out of luck.
(One issue is what might happen in the event of a hard disk error. If the disk dies completely, then atomicity is moot. But if the OS is still able to read data from the disk after the failure, then the outcome may depend on the OS'es ability to repair a possibly inconsistent file system.)
I implement a file manipulation functionality, and I paid attention that Java provides multiple techniques to copy and move files. Below you can find code snippets, briefly describing these approaches:
Approach #1:
File from = new File(src.getPath());
File to = new File(dst.getPath());
from.renameTo(to);
Approach #2:
FileChannel inChannel = new FileInputStream(src).getChannel();
FileChannel outChannel = new FileOutputStream(dst).getChannel();
inChannel.transferTo(0, inChannel.size(), outChannel);
Approach #3:
InputStream in = getContentResolver().openInputStream(selectedImageUri);
OutputStream out = new FileOutputStream("/sdcard/wallpapers/" + wall);
byte[] buffer = new byte[1024];
int read;
while ((read = in.read(buffer)) != -1) {
out.write(buffer, 0, read);
}
Approach #4:
import static java.nio.file.StandardCopyOption.*;
Files.copy(source, target, REPLACE_EXISTING);
All these approaches work, but I can't grasp when should I use each of them? What are the pros and cons of each of these methods, especially from the performance and the reliability points of view? Is there any specific scenario when I have to prefer one technique over another?
It is already discussed enough here and the following is from here
Your first approach is File rename that has nothing to do with File copy
java.io.File class doesn’t have any shortcut method to copy file from source to destination.
1. Using Stream: This is the conventional way of file copy in java, here we create two Files, source and destination. Then we create InputStream from source and write it to destination file using OutputStream.
2. Using java.nio.channels.FileChannel: Java NIO classes were introduced in Java 1.4 and FileChannel can be used to copy file in java. According to transferFrom() method javadoc, this way of copy file is supposed to be faster than using Streams to copy files.
3. Using Apache Commons IO: Apache Commons IO FileUtils.copyFile(File srcFile, File destFile) can be used to copy file in java. If you are already using Apache Commons IO in your project, it makes sense to use this for code simplicity. Internally it uses Java NIO FileChannel, so you can avoid this wrapper method if you are not already using it for other functions.
4. Java 7 Files class: If you are working on Java 7, you can use Files class copy() method to copy file in java. It uses File System providers to copy the files.
Now to see which one of these methods is more efficient we will copy a large file[1 GB] using each one of them in a simple program. To avoid any performance speedups from caching we are going to use four different source files and four different destination files.{Refer code in link}
Time taken by FileStreams Copy = 127572360
Time taken by FileChannels Copy = 10449963
Time taken by Java7 Files Copy = 10808333
Time taken by Apache Commons IO Copy = 17971677
From the output it’s clear that Stream Copy is the best way to copy File in Java. FileChannels is the best way to copy large files. If you work with even larger files you will notice a much bigger speed difference
We can divide your four approaches into two types:
Use a built-in standard library method (such as File.renameTo() and Files.move()).
Do the work ourselves - by copying bytes from source to target.
First, note that File doesn't have a copy method, so you only have one option for built-in, standard library method when you're talking about copy.
Also note that "do the work ourselves" when renaming is going to be very bad - you're going to copy the entire file, then delete the old file. Not a good or efficient approach. In most cases, renaming/moving within the same filesystem requires just changing file metadata without actually touching the content, so it's really a lot better to use a standard library.
So you have two cases:
Renaming
The options are really using either File.renameTo() or Files.move(). No point in using streams and copying data.
File is an outdated class. It shouldn't really be used anymore. There is an excellent explanation why, which sums up to the fact that File doesn't give you any information when any of its standard methods fail, whereas Files provides you with very accurate exceptions when that happens.
Copying
You have two choices - either use Files.copy() or one of the "do it yourself" approaches.
By far, if what you are copying are actual files, your choice should be Files.copy(). There is no need to re-invent the wheel. It does exactly what you want, is well documented, you're not likely to introduce bugs accidentally. And yes, it's very efficient.
Files.copy() relies on underlying "providers" for its operation. What it means is that there are specialized vendor (or operating system) specific classes that do the operation that is the most efficient for that filesystem. Whether it's a Linux filesystem or a Windows one, the copy will be optimized for it. There are even providers for specialized cases, such as zip files, so you can copy files inside a zip, jar or war file using Files.copy() - which is a lot more complicated if you try the "do it yourself" approach.
Besides, Files.copy() checks lots of things that you might forget when you write "your own" copy. For example, did you remember to check that the file that you are reading from and the file you are writing to are not the same file? It could cause serious trouble. Files.copy() does it. It checks permissions, it checks if the target of the copy is a directory, and so on. So it's very reliable.
So why do you have the option to do "your own"? Because well, Java is a general-purpose language. You have the option to read from a file, the option to write to a file, so you can write your own "copy" method. That doesn't mean you should.
Note that in your "approach #3", the "source" file is not actually a file! It's produced from an Image URI, which means it could be a network source. When your source is not a file, but a stream or channel based on a socket, database BLOB, web server request etc., you can't really use Files.copy(). This is where you'd need to write your own.
Actually, Files also has options for copying from a file to an OutputStream or from an InputStream to a file, so if one side of the copy is a stream and the other a file, you can use that. It will be readable, safe, and throw meaningful exceptions.
So write your own copy:
when you need to move data from sources to targets which are not files,
when you need to filter or process the data somehow rather than copy as-is from source to target,
when you are using old versions of Java, prior to 1.7. In this case, channels would probably be better than streams.
I'm using memory mapped files in some Java code to quickly write to a 2G file. I'm mapping the entire file into memory. The issue I have with my solution is that if the file I'm writing to mysteriously disappears or the disk has some type of error, those errors aren't getting bubbled up to the Java code.
In fact, from the Java code, it looks as though my write completed successfully. Here's the unit test I created to simulate this type of failure:
File twoGigFile = new File("big.bin");
RandomAccessFile raf = new RandomAccessFile(twoGigFile, "rw");
raf.setLength(Integer.MAX_VALUE);
raf.seek(30000); // Totally arbitrary
raf.writeInt(42);
raf.writeInt(42);
MappedByteBuffer buf = raf.getChannel().map(MapMode.READ_WRITE, 0, Integer.MAX_VALUE);
buf.force();
buf.position(1000000); // Totally arbitrary
buf.putInt(0);
assertTrue(twoGigFile.delete());
buf.putInt(0);
raf.close();
This code runs without any errors at all. This is quite an issue for me. I can't seem to find anything out there that speaks about this type of issue. Does anyone know how to get memory mapped files to correctly throw exceptions? Or if there is another way to ensure that the data is actually written to the file?
I'm trying to avoid using a RandomAccessFile because they are much slower than memory mapped files. However, there might not be any other option.
You can't. To quote the JavaDoc:
All or part of a mapped byte buffer may become inaccessible at any time [...] An attempt to access an inaccessible region of a mapped byte buffer will not change the buffer's content and will cause an unspecified exception to be thrown either at the time of the access or at some later time.
And here's why: when you use a mapped buffer, you are changing memory, not the file. The fact that the memory happens to be backed by the file is irrelevant until the OS attempts to write the buffered blocks to disk, which is something that is managed entirely by the OS (ie, your application will not know it's happening).
If you expect the file to disappear underneath you, then you'll have to use an alternate mechanism to see if this happens. One possibility is to occasionally touch the file using a RandomAccessFile, and catch the error that it will throw. Depending on your OS, even this may not be sufficient: on Linux, for example, a file exists for those programs that have an open handle to it, even if it has been deleted externally.
Tim Bray's article "Saving Data Safely" left me with open questions. Today, it's over a month old and I haven't seen any follow-up on it, so I decided to address the topic here.
One point of the article is that FileDescriptor.sync() should be called to be on the safe side when using FileOutputStream. At first, I was very irritated, because I never have seen any Java code doing a sync during the 12 years I do Java. Especially since coping with files is a pretty basic thing. Also, the standard JavaDoc of FileOutputStream never hinted at syncing (Java 1.0 - 6). After some research, I figured ext4 may actually be the first mainstream file system requiring syncing. (Are there other file systems where explicit syncing is advised?)
I appreciate some general thoughts on the matter, but I also have some specific questions:
When will Android do the sync to the file system? This could be periodic and additionally based on life cycle events (e.g. an app's process goes to the background).
Does FileDescriptor.sync() take care of syncing the meta data? That is syncing the directory of the changed file. Compare to FileChannel.force().
Usually, one does not directly write into the FileOutputStream. Here's my solution (do you agree?):
FileOutputStream fileOut = ctx.openFileOutput(file, Context.MODE_PRIVATE);
BufferedOutputStream out = new BufferedOutputStream(fileOut);
try {
out.write(something);
out.flush();
fileOut.getFD().sync();
} finally {
out.close();
}
Android will do the sync when it needs to -- such as when the screen turns off, shutting down the device, etc. If you are just looking at "normal" operation, explicit sync by applications is never needed.
The problem comes when the user pulls the battery out of their device (or does a hard reset of the kernel), and you want to ensure you don't lose any data.
So the first thing to realize: the issue is when power is suddenly lost, so a clean shutdown can not happen, and the question of what is going to happen in persistent storage at that point.
If you are just writing a single independent new file, it doesn't really matter what you do. The user could have pulled the battery while you were in the middle of writing, right before you started writing, etc. If you don't sync, it just means there is some longer time from when you are done writing during which pulling the battery will lose the data.
The big concern here is when you want to update a file. In that case, when you next read the file you want to have either the previous contents, or the new contents. You don't want to get something half-way written, or lose the data.
This is often done by writing the data in a new file, and then switching to that from the old file. Prior to ext4 you knew that, once you had finished writing a file, further operations on other files would not go on disk until the ones on that file, so you could safely delete the previous file or otherwise do operations that depend on your new file being fully written.
However now if you write the new file, then delete the old one, and the battery is pulled, when you next boot you may see that the old file is deleted and new file created but the contents of the new file is not complete. By doing the sync, you ensure that the new file is completely written at that point so can do further changes (such as deleting the old file) that depend on that state.
fileOut.getFD().sync(); should be on the finally clause, before the close().
sync() is way more important than close() considering durability.
So, everytime you want to 'finish' working on a file you should sync() it before close()ing it.
posix does not guarantee that pending writes will be written to disk when you issue a close().