We Keep Coding

How to read a File character-by-character in reverse without running out-of-memory?

The Story
I've been having a problem lately...
I have to read a file in reverse character by character without running out of memory.
I can't read it line-by-line and reverse it with StringBuilder because it's a one-line file that takes up to a gig (GB) of I/O space.
Hence it would take up too much of the JVM's (and the System's) Memory.
I've decided to just read it character by character from end-to-start (back-to-front) so that I could process as much as I can without consuming too much memory.
What I've Tried
I know how to read a file in one go:
(MappedByteBuffer+FileChannel+Charset which gave me OutOfMemoryExceptions)
and read a file character-by-character with UTF-8 character support
(FileInputStream+InputStreamReader).
The problem is that FileInputStream's #read() only calls #read0() which is a native method!
Because of that I have no idea about the underlying code...
Which is why I'm here today (or at least until this is done)!

This will do it (but as written it is not very efficient).
just skip to the last location read less one and read and print the character.
then reset the location to the mark, adjust size and continue.
File f = new File("Some File name");
int size = (int) f.length();
int bsize = 1;
byte[] buf = new byte[bsize];
try (BufferedInputStream b =
new BufferedInputStream(new FileInputStream(f))) {
while (size > 0) {
b.mark(size);
b.skip(size - bsize);
int k = b.read(buf);
System.out.print((char) buf[0]);
size -= k;
b.reset();
}
} catch (IOException ioe) {
ioe.printStackTrace();
}
This could be improved by increasing the buffer size and making equivalent adjustments in the mark and skip arguments.
Updated Version
I wasn't fully satisfied with my answer so I made it more general. Some variables could have served double duty but using meaningful names helps clarify how they are used.
Mark must be used so reset can be used. However, it only needs to be set once and is set to position 0 outside of the main loop. I do not know if marking closer to the read point is more efficient or not.
skipCnt - initally set to fileLength it is the number of bytes to skip before reading. If the number of bytes remaining is greater than the buffer size, then the skip count will be skipCnt - bsize. Else it will be 0.
remainingBytes - a running total of how many bytes are still to be read. It is updated by subtracting the current readCnt.
readCnt - how many bytes to read. If remainingBytes is greater than bsize then set to bsize, else set to remainingBytes
The while loop continuously reads the file starting near the end and then prints the just read information in reverse order. All variables are updated and the process repeats until the remainingBytes reaches 0.
File f = new File("some file");
int bsize = 16;
int fileSize = (int)f.length();
int remainingBytes = fileSize;
int skipCnt = fileSize;
byte[] buf = new byte[bsize];
try (BufferedInputStream b =
new BufferedInputStream(new FileInputStream(f))) {
b.mark(0);
while(remainingBytes > 0) {
skipCnt = skipCnt > bsize ? skipCnt - bsize : 0;
b.skip(skipCnt);
int readCnt = remainingBytes > bsize ? bsize : remainingBytes;
b.read(buf,0,readCnt);
for (int i = readCnt-1; i >= 0; i--) {
System.out.print((char) buf[i]);
}
remainingBytes -= readCnt;
b.reset();
}
} catch (IOException ioe) {
ioe.printStackTrace();
}

This doesn't support multi byte UTF-8 characters
Using a RandomAccessFile you can easily read a file in chunks from the end to the beginning, and reverse each of the chunks.
Here's a simple example:
import java.io.FileWriter;
import java.io.IOException;
import java.io.RandomAccessFile;
import java.util.stream.IntStream;
class Test {
private static final int BUF_SIZE = 10;
private static final int FILE_LINE_COUNT = 105;
public static void main(String[] args) throws Exception {
// create a large file
try (FileWriter fw = new FileWriter("largeFile.txt")) {
IntStream.range(1, FILE_LINE_COUNT).mapToObj(Integer::toString).forEach(s -> {
try {
fw.write(s + "\n");
} catch (IOException e) {
throw new RuntimeException(e);
}
});
}
// reverse the file
try (RandomAccessFile raf = new RandomAccessFile("largeFile.txt", "r")) {
long size = raf.length();
byte[] buf = new byte[BUF_SIZE];
for (long i = size - BUF_SIZE; i > -BUF_SIZE; i -= BUF_SIZE) {
long offset = Math.max(0, i);
long readSize = Math.min(i + BUF_SIZE, BUF_SIZE);
raf.seek(offset);
raf.read(buf, 0, (int) readSize);
for (int j = (int) readSize - 1; j >= 0; j--) {
System.out.print((char) buf[j]);
}
}
}
}
}
This uses a very small file and very small chunks so that you can test it easily. Increase those constants to see it work on a larger scale.
The input file contains newlines to make it easy to read the output, but the reversal doesn't depend on the file "having lines".

How to properly validate PNG header in java?

I'm trying to write a function which will check if PNG file is not corrupted. Here I found a function, also provided below, which stores file bytes into its own byte array. I know that the first eight bytes of a PNG file always contain the same decimal values 137 80 78 71 13 10 26 10 (hex: 89 50 4e 47 0d 0a 1a 0a).
When I print the context of the byte array which starts with -1-40-1-3201674707370011007207200-1-370-124022222232235333565555686666681088888810101010101 ... and then convert it to decimal values I don't see the magic number in the beginning. Please, what have I misunderstood? I would like to read the image and compare its header to either decimal or hex values.
public static void main(String[] args)
{
File file = new File("src/resources/dog.png");
readContentIntoByteArray(file);
}
private static byte[] readContentIntoByteArray(File file)
{
FileInputStream fileInputStream = null;
byte[] bFile = new byte[(int) file.length()];
try
{
//convert file into array of bytes
fileInputStream = new FileInputStream(file);
fileInputStream.read(bFile);
fileInputStream.close();
for (int i = 0; i < bFile.length; i++)
{
System.out.print((char) bFile[i]);
}
}
catch (Exception e)
{
e.printStackTrace();
}
return bFile;
}

You are printing the actual bytes as characters to the terminal, not the representation of those bytes in decimal or hex, as #Andreas says.
You can check the header with:
byte[] data = readContentIntoByteArray(file);
byte[] expected = new byte[] {-119, 80, 78, 71, 13, 10, 26, 10};
for (int i = 0; i < expected.length; i++) {
if (expected[i] != data[i]) {
System.out.println("mismatch at " + i);
}
}

Read Data File address (Java)

I have this code for reading data and works fine but I want to change the start point that the data is read from - My DataFile.txt is "abcdefghi"
and the output is
1)97
2)98
3)99
4)100
I want to start at the second byte so the output would be
1)98
2)99
3)100
4)etc
Code:
import java.io.*;
public class ReadFileDemo3 {
public static void main(String[] args)throws IOException {
MASTER MASTER = new MASTER();
MASTER.PART1();
}
}
class MASTER {
void PART1() throws IOException{
System.out.println("OK START THIS PROGRAM");
File file = new File("D://DataFile.txt");
BufferedInputStream HH = null;
int B = 0;
HH = new BufferedInputStream(new FileInputStream(file));
for (int i=0; i<4; i++) {
B = B + 1;
System.out.println(B+")"+HH.read());
}
}
}

You can simple ignore the first n bytes as follows.
HH = new BufferedInputStream(new FileInputStream(file));
int B = 0;
int n = 1; // number of bytes to ignore
while(HH.available()>0) {
// read the byte and convert the integer to character
char c = (char)HH.read();
if(B<n){
continue;
}
B++;
System.out.println(B + ")" + (int)c);
}
Edit: If you want to access a random location in file, then you need to use RandomAccessFile. See this for detailed examples.
Related SO post:
How do I fetch specific bytes from a file knowing the offset and length?
How to read a file from a certain offset

Get the int values from a .wav file in Java

I seem to have reached a dead-end. I try to display the first int values of a 32 bits per sample, 2 channels .wav file with the following code :
public static void main(String[] args) {
File inFile = new File("C:/.../1.wav");
FileInputStream fstream = null;
try {
fstream = new FileInputStream(inFile);
} catch (FileNotFoundException e1) {
e1.printStackTrace();
}
BufferedInputStream in = new BufferedInputStream(fstream);
byte[] bytes = null;
try {
bytes = IOUtils.toByteArray(in);
} catch (IOException e) {
e.printStackTrace();
}
ByteBuffer wrapped = ByteBuffer.wrap(bytes);
int num = wrapped.getInt();
System.out.println(num);
}
But i'm not sure what the number displayed means. I get "1380533830", first of all if I understood correctly is the int value is the not from 0 to 2^32-1 but from -2^31 to 2^31 - 1.
If that's the case it should be the first int from the first channel, but when I do an audioread in matlab I get a completely different value : "-1376256". I tried to see if "1380533830" was a value somewhere in my audioread but it wasn't so I don't know what I did.

String invalid length after writing to StringBuilder and ByteArrayOutputStream from FileInputStream, issue with "null characters"

The goal is to read a file name from a file, which is a max of 100 bytes, and the actual name is the file name filled with "null-bytes".
Here is what it looks like in GNU nano
Where .PKGINFO is the valid file name, and the ^# represent "null bytes".
I tried here with StringBuilder
package falken;
import java.io.*;
public class Testing {
public Testing() {
try {
FileInputStream tarIn = new FileInputStream("/home/gala/falken_test/test.tar");
final int byteOffset = 0;
final int readBytesLength = 100;
StringBuilder stringBuilder = new StringBuilder();
for ( int bytesRead = 1, n, total = 0 ; (n = tarIn.read()) != -1 && total < readBytesLength ; bytesRead++ ) {
if (bytesRead > byteOffset) {
stringBuilder.append((char) n);
total++;
}
}
String out = stringBuilder.toString();
System.out.println(">" + out + "<");
System.out.println(out.length());
} catch (Exception e) {
/*
This is a pokemon catch not used in final code
*/
e.printStackTrace();
}
}
}
But it gives an invalid String length of 100, while the output on IntelliJ shows the correct string passed withing the >< signs.
>.PKGINFO<
100
Process finished with exit code 0
But when i paste it here on StackOverflow I get the correct string with unknown "null-characters", whose size is actually 100.
>.PKGINFO <
What regex can i use to get rid of the characters after the valid file name?
The file I am reading is ASCII encoded.
I also tried ByteArrayOutputStream, with the same result
package falken;
import java.io.*;
import java.nio.charset.StandardCharsets;
public class Testing {
public Testing() {
try {
FileInputStream tarIn = new FileInputStream("/home/gala/falken_test/test.tar");
final int byteOffset = 0;
final int readBytesLength = 100;
ByteArrayOutputStream byteArrayOutputStream = new ByteArrayOutputStream();
for ( int bytesRead = 1, n, total = 0 ; (n = tarIn.read()) != -1 && total < readBytesLength ; bytesRead++ ) {
if (bytesRead > byteOffset) {
byteArrayOutputStream.write(n);
total++;
}
}
String out = byteArrayOutputStream.toString();
System.out.println(">" + out + "<");
System.out.println(out.length());
} catch (Exception e) {
/*
This is a pokemon catch not used in final code
*/
e.printStackTrace();
}
}
}
What could be the issue here?

Well, it seems to be reading null characters as actual characters, spaces in fact. If it's possible, see if you can read the filename, then, cut out the null characters. In your case, you need a data.trim(); and a data2 = data.substring(0,(data.length()-1))

You need to stop appending to the string buffer once you read the first null character from the file.
You seem to want to read a tar archive, have a look at the following code which should get you started.
byte[] buffer = new byte[500]; // POSIX tar header is 500 bytes
FileInputStream is = new FileInputStream("test.tar");
int read = is.read(buffer);
// check number of bytes read; don't bother if not at least the whole
// header has been read
if (read == buffer.length) {
// search for first null byte; this is the end of the name
int offset = 0;
while (offset < 100 && buffer[offset] != 0) {
offset++;
}
// create string from byte buffer using ASCII as the encoding (other
// encodings are not supported by tar)
String name = new String(buffer, 0, offset,
StandardCharsets.US_ASCII);
System.out.println("'" + name + "'");
}
is.close();
You really shouldn't use trim() on the filename, this will break whenever you encounter a filename with leading or trailing blanks.