We Keep Coding

Read/Write Bytes to and From a File Using Only Java.IO

How can we write a byte array to a file (and read it back from that file) in Java?
Yes, we all know there are already lots of questions like that, but they get very messy and subjective due to the fact that there are so many ways to accomplish this task.
So let's reduce the scope of the question:
Domain:
Android / Java
What we want:
Fast (as possible)
Bug-free (in a rigidly meticulous way)
What we are not doing:
Third-party libraries
Any libraries that require Android API later than 23 (Marshmallow)
(So, that rules out Apache Commons, Google Guava, Java.nio, and leaves us with good ol' Java.io)
What we need:
Byte array is always exactly the same (content and size) after going through the write-then-read process
Write method only requires two arguments: File file, and byte[] data
Read method returns a byte[] and only requires one argument: File file
In my particular case, these methods are private (not a library) and are NOT responsible for the following, (but if you want to create a more universal solution that applies to a wider audience, go for it):
Thread-safety (file will not be accessed by more than one process at once)
File being null
File pointing to non-existent location
Lack of permissions at the file location
Byte array being too large
Byte array being null
Dealing with any "index," "length," or "append" arguments/capabilities
So... we're sort of in search of the definitive bullet-proof code that people in the future can assume is safe to use because your answer has lots of up-votes and there are no comments that say, "That might crash if..."
This is what I have so far:
Write Bytes To File:
private void writeBytesToFile(final File file, final byte[] data) {
try {
FileOutputStream fos = new FileOutputStream(file);
fos.write(data);
fos.close();
} catch (Exception e) {
Log.i("XXX", "BUG: " + e);
}
}
Read Bytes From File:
private byte[] readBytesFromFile(final File file) {
RandomAccessFile raf;
byte[] bytesToReturn = new byte[(int) file.length()];
try {
raf = new RandomAccessFile(file, "r");
raf.readFully(bytesToReturn);
} catch (Exception e) {
Log.i("XXX", "BUG: " + e);
}
return bytesToReturn;
}
From what I've read, the possible Exceptions are:
FileNotFoundException : Am I correct that this should not happen as long as the file path being supplied was derived using Android's own internal tools and/or if the app was tested properly?
IOException : I don't really know what could cause this... but I'm assuming that there's no way around it if it does.
So with that in mind... can these methods be improved or replaced, and if so, with what?

It looks like these are going to be core utility/library methods which must run on Android API 23 or later.
Concerning library methods, I find it best to make no assumptions on how applications will use these methods. In some cases the applications may want to receive checked IOExceptions (because data from a file must exist for the application to work), in other cases the applications may not even care if data is not available (because data from a file is only cache that is also available from a primary source).
When it comes to I/O operations, there is never a guarantee that operations will succeed (e.g. user dropping phone in the toilet). The library should reflect that and give the application a choice on how to handle errors.
To optimize I/O performance always assume the "happy path" and catch errors to figure out what went wrong. This is counter intuitive to normal programming but essential in dealing with storage I/O. For example, just checking if a file exists before reading from a file can make your application twice as slow - all these kind of I/O actions add up fast to slow your application down. Just assume the file exists and if you get an error, only then check if the file exists.
So given those ideas, the main functions could look like:
public static void writeFile(File f, byte[] data) throws FileNotFoundException, IOException {
try (FileOutputStream out = new FileOutputStream(f)) {
out.write(data);
}
}
public static int readFile(File f, byte[] data) throws FileNotFoundException, IOException {
try (FileInputStream in = new FileInputStream(f)) {
return in.read(data);
}
}
Notes about the implementation:
The methods can also throw runtime-exceptions like NullPointerExceptions - these methods are never going to be "bug free".
I do not think buffering is needed/wanted in the methods above since only one native call is done
(see also here).
The application now also has the option to read only the beginning of a file.
To make it easier for an application to read a file, an additional method can be added. But note that it is up to the library to detect any errors and report them to the application since the application itself can no longer detect those errors.
public static byte[] readFile(File f) throws FileNotFoundException, IOException {
int fsize = verifyFileSize(f);
byte[] data = new byte[fsize];
int read = readFile(f, data);
verifyAllDataRead(f, data, read);
return data;
}
private static int verifyFileSize(File f) throws IOException {
long fsize = f.length();
if (fsize > Integer.MAX_VALUE) {
throw new IOException("File size (" + fsize + " bytes) for " + f.getName() + " too large.");
}
return (int) fsize;
}
public static void verifyAllDataRead(File f, byte[] data, int read) throws IOException {
if (read != data.length) {
throw new IOException("Expected to read " + data.length
+ " bytes from file " + f.getName() + " but got only " + read + " bytes from file.");
}
}
This implementation adds another hidden point of failure: OutOfMemory at the point where the new data array is created.
To accommodate applications further, additional methods can be added to help with different scenario's. For example, let's say the application really does not want to deal with checked exceptions:
public static void writeFileData(File f, byte[] data) {
try {
writeFile(f, data);
} catch (Exception e) {
fileExceptionToRuntime(e);
}
}
public static byte[] readFileData(File f) {
try {
return readFile(f);
} catch (Exception e) {
fileExceptionToRuntime(e);
}
return null;
}
public static int readFileData(File f, byte[] data) {
try {
return readFile(f, data);
} catch (Exception e) {
fileExceptionToRuntime(e);
}
return -1;
}
private static void fileExceptionToRuntime(Exception e) {
if (e instanceof RuntimeException) { // e.g. NullPointerException
throw (RuntimeException)e;
}
RuntimeException re = new RuntimeException(e.toString());
re.setStackTrace(e.getStackTrace());
throw re;
}
The method fileExceptionToRuntime is a minimal implementation, but it shows the idea here.
The library could also help an application to troubleshoot when an error does occur. For example, a method canReadFile(File f) could check if a file exists and is readable and is not too large. The application could call such a function after a file-read fails and check for common reasons why a file cannot be read. The same can be done for writing to a file.

Although you can't use third party libraries, you can still read their code and learn from their experience. In Google Guava for example, you usually read a file into bytes like this:
FileInputStream reader = new FileInputStream("test.txt");
byte[] result = ByteStreams.toByteArray(reader);
The core implementation of this is toByteArrayInternal. Before calling this, you should check:
A not null file is passed (NullPointerException)
The file exists (FileNotFoundException)
After that, it is reduced to handling an InputStream and this where IOExceptions come from. When reading streams a lot of things out of the control of your application can go wrong (bad sectors and other hardware issues, mal-functioning drivers, OS access rights) and manifest themselves with an IOException.
I am copying here the implementation:
private static final int BUFFER_SIZE = 8192;
/** Max array length on JVM. */
private static final int MAX_ARRAY_LEN = Integer.MAX_VALUE - 8;
private static byte[] toByteArrayInternal(InputStream in, Queue<byte[]> bufs, int totalLen)
throws IOException {
// Starting with an 8k buffer, double the size of each successive buffer. Buffers are retained
// in a deque so that there's no copying between buffers while reading and so all of the bytes
// in each new allocated buffer are available for reading from the stream.
for (int bufSize = BUFFER_SIZE;
totalLen < MAX_ARRAY_LEN;
bufSize = IntMath.saturatedMultiply(bufSize, 2)) {
byte[] buf = new byte[Math.min(bufSize, MAX_ARRAY_LEN - totalLen)];
bufs.add(buf);
int off = 0;
while (off < buf.length) {
// always OK to fill buf; its size plus the rest of bufs is never more than MAX_ARRAY_LEN
int r = in.read(buf, off, buf.length - off);
if (r == -1) {
return combineBuffers(bufs, totalLen);
}
off += r;
totalLen += r;
}
}
// read MAX_ARRAY_LEN bytes without seeing end of stream
if (in.read() == -1) {
// oh, there's the end of the stream
return combineBuffers(bufs, MAX_ARRAY_LEN);
} else {
throw new OutOfMemoryError("input is too large to fit in a byte array");
}
}
As you can see most of the logic has to do with reading the file in chunks. This is to handle situations, where you don't know the size of the InputStream, before starting reading. In your case, you only need to read files and you should be able to know the length beforehand, so this complexity could be avoided.
The other check is OutOfMemoryException. In standard Java the limit is too big, however in Android, it will be a much smaller value. You should check, before trying to read the file that there is enough memory available.

Java Bytecode - ASM - Get Label Offset

I am trying to get the offsets of all labels in a method.
I tried using the following code:
private static ArrayList<Integer> GetLabelOffsets(MethodNode methodNode) {
ArrayList<Integer> labelOffsets = new ArrayList<>();
for (AbstractInsnNode instruction : methodNode.instructions.toArray()) {
if (instruction instanceof JumpInsnNode) {
JumpInsnNode jumpInsnNode = (JumpInsnNode) instruction;
labelOffsets.add(jumpInsnNode.label.getLabel().getOffset());.
}
}
return labelOffsets;
}
However the getOffset() method throws an Exception:
java.lang.IllegalStateException: Label offset position has not been resolved yet
How can I resolve these offset positions? Or what is the proper way to achieve this?
Edit
The MethodNode is an org.objectweb.asm.tree.MethodNode Object from the Java ASM library
Added more code at request:
public static HashMap<String, ClassNode> ParseJar(JarFile jar) {
HashMap<String, ClassNode> classes = new HashMap<>();
try {
Enumeration<?> enumeration = jar.entries();
while (enumeration.hasMoreElements()) {
JarEntry entry = (JarEntry) enumeration.nextElement();
if (entry.getName().endsWith(".class")) {
ClassReader classReader = new ClassReader(jar.getInputStream(entry));
ClassNode classNode = new ClassNode();
classReader.accept(classNode, ClassReader.SKIP_DEBUG | ClassReader.SKIP_FRAMES);
classes.put(classNode.name, classNode);
}
}
jar.close();
return classes;
} catch (Exception ex) {
return null;
}
}
public static void main(String[] args) {
JarFile jar = new JarFile(fileName);
HashMap<String, ClassNode> classes = JarUtils.ParseJar(jar);
for (ClassNode classNode : classes.values()) {
for (MethodNode methodNode : classNode.methods) {
ArrayList<Integer> offsets = GetLabelOffsets(methodNode);
// do more stuff with offsets
}
}
}

From the documentation of getOffset():
This method is intended for Attribute sub classes, and is normally not needed by class generators or adapters.
Since this offset is defined in terms of bytes, it wouldn’t be very helpful when processing a list of instructions, especially as ASM abstracts the different forms of instructions that may have different lengths in byte code.
The general idea is that this instruction list can be changed, so Labels represent logical positions and the offset will be calculated when writing a method’s resulting bytecode and the actual numbers are known.
Within the instruction list, there should be a corresponding LabelNode referencing the same Label as the instruction.

Liferay Concurrent FileEntry Upload

Problem Statement :
In liferay i have to import a zip file in to some folder in liferay cms, So far I had implemented serial unzipping of the zip file create it's folder and then it's files. The problem here is that the whole process takes a lot of time. So I had to use parallel approach in creating folders and creating files.
My Solution :
I have used a java java.util.concurrent.ExecutorService to create a Executors.newFixedThreadPool(NTHREDS) where NTHREDS is the number of threads to be run in parallel (say 5)
I read all the folder paths from the zip and placed , list of zip
entires (files) against folder path as a key in HashMap
Traversed all keys in the map and created folders serially
Now traversed the list of zip entries (files) from map and passed to a thread worker,one file for each worker, these workers are then sent to
ExecutorService to Execute
So far i didn't find any significant reduction in time of the whole process, am i moving in the correct direction? Does liferay support concurrent file addition? What am I doing wrong?
I will be much thankful for any help in this regard
below is my code
imports
...
...
public class TestImportZip {
private static final int NTHREDS = 5;
ExecutorService executor = null;
...
...
....
Map<String,Folder> folders = new HashMap<String,Folder>();
File zipsFile = null;
public TestImportZip(............,File zipFile, .){
.
.
this.zipsFile = zipFile;
this.executor = Executors.newFixedThreadPool(NTHREDS);
}
// From here the process starts
public void importZip() {
Map<String,List<ZipEntry>> foldersMap = new HashMap<String, List<ZipEntry>>();
try (ZipFile zipFile = new ZipFile(zipsFile)) {
zipFile.stream().forEach(entry -> {
String entryName = entry.getName();
if(entryName.contains("/")) {
String key = entryName.substring(0, entryName.lastIndexOf("/"));
List<ZipEntry> zipEntries = foldersMap.get(key);
if(zipEntries == null){
zipEntries = new ArrayList<>();
}
zipEntries.add(entry);
foldersMap.put(key,zipEntries);
}
});
createFolders(foldersMap.keySet());
createFiles(foldersMap);
} catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}
private void createFolders(Set<String> folderPathSets) {
// create folder and put the folder in map
.
.
.
folders.put(folderPath,folder);
}
private void createFiles(Map<String, List<ZipEntry>> foldersMap) {
.
.
.
//Traverse all the files from all the list in map and send them to worker
createFileWorker(folderPath,zipEntry);
}
private void createFileWorker(String folderPath,ZipEntry zipEntry) {
CreateEntriesWorker cfw = new CreateEntriesWorker(folderPath, zipEntry);
executor.execute(cfw);
}
class CreateEntriesWorker implements Runnable{
Folder folder = null;
ZipEntry entryToCreate = null;
public CreateEntriesWorker(String folderPath, ZipEntry zipEntry){
this.entryToCreate = zipEntry;
// get folder from already created folder map
this.folder = folders.get(folderPath);
}
public void run() {
if(this.folder != null) {
long startTime = System.currentTimeMillis();
try (ZipFile zipFile = new ZipFile(zipsFile)) {
InputStream inputStream = zipFile.getInputStream(entryToCreate);
try{
String name = entryToCreate.getName();
// created file entry here
}catch(Exception e){
}finally{
if(inputStream != null)
inputStream.close();
}
} catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}
}
}
}

Your simplified code does not contain any Liferay reference that I recognize. The description you provide gives a hint that you're trying to optimize some code, but don't get any better performance out of your try. This typically is a sign that you're trying to optimize the wrong aspect of the problem (or it's already quite optimized).
You'll need to determine the actual bottleneck of your operation in order to know if it's feasible to optimize. There's a common saying that "premature optimization is the root of all evil". What does it mean?
I'll completely make up numbers here - don't quote me on them: They're freely invented for illustration purposes. Let's say, that your operation of adding the contents of a Zip file to Liferay's repository is distributed to the following percentages of operational resources:
4% zip file decoding/decompressing
6% file I/O for zip operations and temporary files
10% database operation for storing the files
60% for extracting text-only from word, pdf, excel and other files stored within the zip file in order to index the document in the full-text index
20% overhead of the full-text indexing library for putting together the index.
Suppose you're optimizing the zip file decoding/decompressing - what overall improvement of numbers can you expect?
While my numbers are made up: If your optimizations do not have any result, I'd recommend to reverse them, measure where you need to optimize and go after that place (or accept it and upgrade your hardware if that place is out of reach).
Run those numbers for CPU, I/O, memory and other potential bottlenecks. Identify your actual bottleneck #1, fix it, measure again. You'll see that bottleneck #2 has gotten a promotion. Rinse repeat until you're happy

Java TreeMap<String, String> returns null even if the key exists

I have a class which extends TreeMap with one external method.
The external method "open" suppose to read lines from a given file in the following format "word:meaning" and add it to the TreeMap - put("word", "meaning").
So I read the file with RandomAccessFile and put the keys-values in the TreeMap and when I print the TreeMap I can see the proper keys and values, for example:
{AAAA=BBBB, CAB=yahoo!}
But for some reason when I do get("AAAA") I get null.
Any reason why it's happening and how to solve it?
Here is the code
public class InMemoryDictionary extends TreeMap<String, String> implements
PersistentDictionary {
private static final long serialVersionUID = 1L; // (because we're extending
// a serializable class)
private File dictFile;
public InMemoryDictionary(File dictFile) {
super();
this.dictFile = dictFile;
}
#Override
public void open() throws IOException {
clear();
RandomAccessFile file = new RandomAccessFile(dictFile, "rw");
file.seek(0);
String line;
while (null != (line = file.readLine())) {
int firstColon = line.indexOf(":");
put(line.substring(0, firstColon - 1),
line.substring(firstColon + 1, line.length() - 1));
}
file.close();
}
#Override
public void close() throws IOException {
dictFile.delete();
RandomAccessFile file = new RandomAccessFile(dictFile, "rw");
file.seek(0);
for (Map.Entry<String, String> entry : entrySet()) {
file.writeChars(entry.getKey() + ":" + entry.getValue() + "\n");
}
file.close();
}
}

the "question marks" from a previous version of your question are important. they indicate that the strings you thought you were seeing are not in fact the strings you are using. RandomAccessFile is a poor choice to read a text file. You are presumably reading a text file with a text encoding which is not single byte (utf-16 perhaps)? the resulting strings are mis-encoded since RandomAccessFile does an "ascii" character conversion. this is causing your get() call to fail.
first, figure out the character encoding of your file and open it with the appropriately configured InputStreamReader.
second, extending TreeMap is a very poor design. Use aggregation here, not extension.

removeEldestEntry overriding

How can I override removeEldestEntry method to saving eldest entry to file? Also how to limit the size of a file like I did it in LinkedHashMap. Here is code:
import java.util.*;
public class level1 {
private static final int max_cache = 50;
private Map cache = new LinkedHashMap(max_cache, .75F, true) {
protected boolean removeEldestEntry(Map.Entry eldest) {
return size() > max_cache;
}
};
public level1() {
for (int i = 1; i < 52; i++) {
String string = String.valueOf(i);
cache.put(string, string);
System.out.println("\rCache size = " + cache.size() +
"\tRecent value = " + i + " \tLast value = " +
cache.get(string) + "\tValues in cache=" +
cache.values());
}
I tried to use FileOutPutSTream :
private Map cache = new LinkedHashMap(max_cache, .75F, true) {
protected boolean removeEldestEntry(Map.Entry eldest) throws IOException {
boolean removed = super.removeEldestEntry(eldest);
if (removed) {
FileOutputStream fos = new FileOutputStream("t.tmp");
ObjectOutputStream oos = new ObjectOutputStream(fos);
oos.writeObject(eldest.getValue());
oos.close();
}
return removed;
}
But I have gained an error
Error(15,27): removeEldestEntry(java.util.Map.Entry) in cannot override removeEldestEntry(java.util.Map.Entry) in java.util.LinkedHashMap; overridden method does not throw java.io.IOException
Without IOExecptio compiler asks to handle IOexception and Filenotfoundexception.
Maybe another way exists? Pls show me example code, I am new in java and just trying to understand the basic principles of 2 level caching. Thx

You first need to make sure your method properly overrides the parent. You can make some small changes to the signature, such as only throwing a more specific checked exception that is a sub-class of a checked exception declared in the parent. In this case, the parent does not declare any checked exception so you can not refine that further and may not throw any checked exceptions. So you will have to handle the IOException locally. There are several ways you can do that, convert it to a RuntimeException of some kind and/or log it.
If you are concerned about the file size, you probably do not want to keep just the last removed entry but many of them - so you should open the file for append.
You need to return true from the method to actually remove the eldest and you need to decide if the element should be removed.
When working with files you should use try/finally to ensure that you close the resource even if there is an exception. This can get a little ugly - sometimes it's nice to have a utility method to do the close so you don't need the extra try/catch.
Generally you should also use some buffering for file I/O which greatly improves performance; in this case use wrap the file stream in a java.io.BufferedOutputStream and provide that to the ObjectOutputStream.
Here is something that may do what you want:
private static final int MAX_ENTRIES_ALLOWED = 100;
private static final long MAX_FILE_SIZE = 1L * 1024 * 1024; // 1 MB
protected boolean removeEldestEntry(Map.Entry eldest) {
if (size() <= MAX_ENTRIES_ALLOWED) {
return false;
}
File objFile = new File("t.tmp");
if (objFile.length() > MAX_FILE_SIZE) {
// Do something here to manage the file size, such as renaming the file
// You won't be able to easily remove an object from the file without a more
// advanced file structure since you are writing arbitrary sized serialized
// objects. You would need to do some kind of tagging of each entry or include
// a record length before each one. Then you would have to scan and rebuild
// a new file. You cannot easily just delete bytes earlier in the file without
// even more advanced structures (like having an index, fixed size records and
// free space lists, or even a database).
}
FileOutputStream fos = null;
try {
fos = new FileOutputStream(objFile, true); // Open for append
ObjectOutputStream oos = new ObjectOutputStream(new BufferedOutputStream(fos));
oos.writeObject(eldest.getValue());
oos.close(); // Close the object stream to flush remaining generated data (if any).
return true;
} catch (IOException e) {
// Log error here or....
throw new RuntimeException(e.getMessage(), e); // Convert to RuntimeException
} finally {
if (fos != null) {
try {
fos.close();
} catch (IOException e2) {
// Log failure - no need to throw though
}
}
}
}

You can't change the method signature when overriding a method. So you need to handle the exception in the overridden method instead of throwing it.
This contains a good explanation on how to use try and catch: http://download.oracle.com/javase/tutorial/essential/exceptions/try.html