We Keep Coding

How to read a File character-by-character in reverse without running out-of-memory?

The Story
I've been having a problem lately...
I have to read a file in reverse character by character without running out of memory.
I can't read it line-by-line and reverse it with StringBuilder because it's a one-line file that takes up to a gig (GB) of I/O space.
Hence it would take up too much of the JVM's (and the System's) Memory.
I've decided to just read it character by character from end-to-start (back-to-front) so that I could process as much as I can without consuming too much memory.
What I've Tried
I know how to read a file in one go:
(MappedByteBuffer+FileChannel+Charset which gave me OutOfMemoryExceptions)
and read a file character-by-character with UTF-8 character support
(FileInputStream+InputStreamReader).
The problem is that FileInputStream's #read() only calls #read0() which is a native method!
Because of that I have no idea about the underlying code...
Which is why I'm here today (or at least until this is done)!

This will do it (but as written it is not very efficient).
just skip to the last location read less one and read and print the character.
then reset the location to the mark, adjust size and continue.
File f = new File("Some File name");
int size = (int) f.length();
int bsize = 1;
byte[] buf = new byte[bsize];
try (BufferedInputStream b =
new BufferedInputStream(new FileInputStream(f))) {
while (size > 0) {
b.mark(size);
b.skip(size - bsize);
int k = b.read(buf);
System.out.print((char) buf[0]);
size -= k;
b.reset();
}
} catch (IOException ioe) {
ioe.printStackTrace();
}
This could be improved by increasing the buffer size and making equivalent adjustments in the mark and skip arguments.
Updated Version
I wasn't fully satisfied with my answer so I made it more general. Some variables could have served double duty but using meaningful names helps clarify how they are used.
Mark must be used so reset can be used. However, it only needs to be set once and is set to position 0 outside of the main loop. I do not know if marking closer to the read point is more efficient or not.
skipCnt - initally set to fileLength it is the number of bytes to skip before reading. If the number of bytes remaining is greater than the buffer size, then the skip count will be skipCnt - bsize. Else it will be 0.
remainingBytes - a running total of how many bytes are still to be read. It is updated by subtracting the current readCnt.
readCnt - how many bytes to read. If remainingBytes is greater than bsize then set to bsize, else set to remainingBytes
The while loop continuously reads the file starting near the end and then prints the just read information in reverse order. All variables are updated and the process repeats until the remainingBytes reaches 0.
File f = new File("some file");
int bsize = 16;
int fileSize = (int)f.length();
int remainingBytes = fileSize;
int skipCnt = fileSize;
byte[] buf = new byte[bsize];
try (BufferedInputStream b =
new BufferedInputStream(new FileInputStream(f))) {
b.mark(0);
while(remainingBytes > 0) {
skipCnt = skipCnt > bsize ? skipCnt - bsize : 0;
b.skip(skipCnt);
int readCnt = remainingBytes > bsize ? bsize : remainingBytes;
b.read(buf,0,readCnt);
for (int i = readCnt-1; i >= 0; i--) {
System.out.print((char) buf[i]);
}
remainingBytes -= readCnt;
b.reset();
}
} catch (IOException ioe) {
ioe.printStackTrace();
}

This doesn't support multi byte UTF-8 characters
Using a RandomAccessFile you can easily read a file in chunks from the end to the beginning, and reverse each of the chunks.
Here's a simple example:
import java.io.FileWriter;
import java.io.IOException;
import java.io.RandomAccessFile;
import java.util.stream.IntStream;
class Test {
private static final int BUF_SIZE = 10;
private static final int FILE_LINE_COUNT = 105;
public static void main(String[] args) throws Exception {
// create a large file
try (FileWriter fw = new FileWriter("largeFile.txt")) {
IntStream.range(1, FILE_LINE_COUNT).mapToObj(Integer::toString).forEach(s -> {
try {
fw.write(s + "\n");
} catch (IOException e) {
throw new RuntimeException(e);
}
});
}
// reverse the file
try (RandomAccessFile raf = new RandomAccessFile("largeFile.txt", "r")) {
long size = raf.length();
byte[] buf = new byte[BUF_SIZE];
for (long i = size - BUF_SIZE; i > -BUF_SIZE; i -= BUF_SIZE) {
long offset = Math.max(0, i);
long readSize = Math.min(i + BUF_SIZE, BUF_SIZE);
raf.seek(offset);
raf.read(buf, 0, (int) readSize);
for (int j = (int) readSize - 1; j >= 0; j--) {
System.out.print((char) buf[j]);
}
}
}
}
}
This uses a very small file and very small chunks so that you can test it easily. Increase those constants to see it work on a larger scale.
The input file contains newlines to make it easy to read the output, but the reversal doesn't depend on the file "having lines".

Java - Start Audio Playback at X Position

Edit: I am using a .wav file
I'm trying to figure out how to start audio at a certain position (for example: 10 seconds into audio file rather than at the start). Reading the documentation for SourceDataLine had me believe this may be achieved using the offset during:
line.write(byte[] b, int offset, int length)
but every time I've tried any value other than 0 (the default I believe), I get java.lang.IndexOutOfBoundsException, which maybe it hasn't read x byte position yet so cannot write x byte position? I'm unsure and left scratching my head.
I figured this would be a common enough request but can't seem to find anything online related to this, only pausing and resuming audio. I'm probably not searching properly.
In case it matters, here is how I'm currently doing my audio:
AudioInputStream stream = AudioSystem.getAudioInputStream("...file...");
AudioFormat format = stream.getFormat();
SourceDataLine.Info info = new DataLine.Info(SourceDataLine.class, format,((int)stream.getFrameLength()*format.getFrameSize()));
SourceDataLine line = (SourceDataLine)AudioSystem.getLine(info);
int bufferSize = line.getBufferSize();
byte inBuffer[] = new byte[bufferSize];
byte outBuffer[] = new byte[bufferSize];
int numRead, numWritten;
do {
numRead = audioStream.read(inBuffer, 0, bufferSize);
if(numRead <= 0) {
myAudio.flushStream();
} else {
myAudio.writeBytesToStream(inBuffer, numRead);
}
do {
numWritten = myAudio.readBytesFromStream(outBuffer, bufferSize);
if(numWritten > 0) {
line.write(outBuffer, 0, numWritten);
}
} while(numWritten > 0);
} while(numRead > 0);

The problem you are having probably stems from the fact that you are adjusting the offset without adjusting the length. If your array is 10 bytes long and you are starting reading 10 bytes from offset 5 instead of 0, you are reading 5 bytes past its end.
I'd recommend to first skip the appropriate number of bytes using skip(long) on the AudioInputStream and then write to the line.
AudioInputStream stream = AudioSystem.getAudioInputStream("...file...");
AudioFormat format = stream.getFormat();
// find out how many bytes you have to skip, this depends on bytes per frame (a.k.a. frameSize)
int secondsToSkip = 10;
long bytesToSkip = format.getFrameSize() * ((int)format.getFrameRate()) * secondsToSkip;
// now skip until the correct number of bytes have been skipped
int justSkipped = 0;
while (bytesToSkip > 0 && (justSkipped = stream.skip(bytesToSkip)) > 0) {
bytesToSkip -= justSkipped;
}
// then proceed with writing to your line like you have done before
[...]
Note that this only works, if the audio file is uncompressed. If you are dealing with something like .mp3, you first have to convert the stream to PCM (see https://stackoverflow.com/a/41850901/942774)

I've created an example which compiles and works. You can play a .wav file from any time point. It should also work for an mp3 file, but I haven't tested that. Invoke mp3ToWav() for that.
import javax.sound.sampled.*;
import java.io.File;
import java.io.IOException;
public class PlayWavAtTimePoint {
public static void main(String[] args) throws Exception {
String fileName = args[0];
int secondsToSkip = (Integer.parseInt(args[1]));
PlayWavAtTimePoint program = new PlayWavAtTimePoint();
AudioInputStream is = program.getAudioInputStream(fileName);
program.skipFromBeginning(is, secondsToSkip);
program.playSound(is);
}
private static void skipFromBeginning(AudioInputStream audioStream, int secondsToSkip) throws UnsupportedAudioFileException, IOException, LineUnavailableException {
AudioFormat format = audioStream.getFormat();
// find out how many bytes you have to skip, this depends on bytes per frame (a.k.a. frameSize)
long bytesToSkip = format.getFrameSize() * ((int)format.getFrameRate()) * secondsToSkip;
// now skip until the correct number of bytes have been skipped
long justSkipped = 0;
while (bytesToSkip > 0 && (justSkipped = audioStream.skip(bytesToSkip)) > 0) {
bytesToSkip -= justSkipped;
}
}
private static final int BUFFER_SIZE = 128000;
/**
* #param filename the name of the file that is going to be played
*/
public void playSound(String filename) throws IOException, UnsupportedAudioFileException, LineUnavailableException {
AudioInputStream audioStream = getAudioInputStream(filename);
playSound(audioStream);
}
private AudioInputStream getAudioInputStream(String filename) throws UnsupportedAudioFileException, IOException {
return AudioSystem.getAudioInputStream(new File(filename));
}
public void playSound(AudioInputStream audioStream) throws LineUnavailableException, IOException {
AudioFormat audioFormat = audioStream.getFormat();
DataLine.Info info = new DataLine.Info(SourceDataLine.class, audioFormat);
SourceDataLine audioOutput = (SourceDataLine) AudioSystem.getLine(info);
audioOutput.open(audioFormat);
audioOutput.start();
//This seems to be reading the whole file into a buffer before playing ... not efficient.
//Why not stream it?
int nBytesRead = 0;
byte[] abData = new byte[BUFFER_SIZE];
while (nBytesRead != -1) {
nBytesRead = audioStream.read(abData, 0, abData.length);
if (nBytesRead >= 0) {
audioOutput.write(abData, 0, nBytesRead);
}
}
audioOutput.drain();
audioOutput.close();
}
/**
* Invoke this function to convert to a playable file.
*/
public static void mp3ToWav(File mp3Data) throws UnsupportedAudioFileException, IOException {
// open stream
AudioInputStream mp3Stream = AudioSystem.getAudioInputStream(mp3Data);
AudioFormat sourceFormat = mp3Stream.getFormat();
// create audio format object for the desired stream/audio format
// this is *not* the same as the file format (wav)
AudioFormat convertFormat = new AudioFormat(AudioFormat.Encoding.PCM_SIGNED,
sourceFormat.getSampleRate(), 16,
sourceFormat.getChannels(),
sourceFormat.getChannels() * 2,
sourceFormat.getSampleRate(),
false);
// create stream that delivers the desired format
AudioInputStream converted = AudioSystem.getAudioInputStream(convertFormat, mp3Stream);
// write stream into a file with file format wav
AudioSystem.write(converted, AudioFileFormat.Type.WAVE, new File("/tmp/out.wav"));
}
}

writing double[] as WAV file in Java

I'm trying to save a double[] array as .WAV file using this method:
public static void saveWav(String filename, double[] samples) {
// assumes 44,100 samples per second
// use 16-bit audio, 2 channels, signed PCM, little Endian
AudioFormat format = new AudioFormat(SAMPLE_RATE * 2, 16, 1, true, false);
byte[] data = new byte[2 * samples.length];
for (int i = 0; i < samples.length; i++) {
int temp = (short) (samples[i] * MAX_16_BIT);
data[2*i + 0] = (byte) temp;
data[2*i + 1] = (byte) (temp >> 8);
}
// now save the file
try {
ByteArrayInputStream bais = new ByteArrayInputStream(data);
AudioInputStream ais = new AudioInputStream(bais, format, samples.length);
if (filename.endsWith(".wav") || filename.endsWith(".WAV")) {
AudioSystem.write(ais, AudioFileFormat.Type.WAVE, new File(filename));
}
else if (filename.endsWith(".au") || filename.endsWith(".AU")) {
AudioSystem.write(ais, AudioFileFormat.Type.AU, new File(filename));
}
else {
throw new RuntimeException("File format not supported: " + filename);
}
}
catch (IOException e) {
System.out.println(e);
System.exit(1);
}
}
but when I reload the files I saved, for every song[i], the double value is different than the original. I use this method to read WAV files:
public static double[] read(String filename) {
byte[] data = readByte(filename);
int N = data.length;
double[] d = new double[N/2];
for (int i = 0; i < N/2; i++) {
d[i] = ((short) (((data[2*i+1] & 0xFF) << 8) + (data[2*i] & 0xFF))) / ((double) MAX_16_BIT);
}
return d;
}
private static byte[] readByte(String filename) {
byte[] data = null;
AudioInputStream ais = null;
try {
// try to read from file
File file = new File(filename);
if (file.exists()) {
ais = AudioSystem.getAudioInputStream(file);
data = new byte[ais.available()];
ais.read(data);
}
// try to read from URL
else {
URL url = StdAudio.class.getResource(filename);
ais = AudioSystem.getAudioInputStream(url);
data = new byte[ais.available()];
ais.read(data);
}
}
catch (IOException e) {
System.out.println(e.getMessage());
throw new RuntimeException("Could not read " + filename);
}
catch (UnsupportedAudioFileException e) {
System.out.println(e.getMessage());
throw new RuntimeException(filename + " in unsupported audio format");
}
return data;
}
I need both double[] arrays to have the exact same values, and that's not the case.
when I hear the song playing I can't tell the difference, but I still need those original values.
Any help appreciated.
Guy

A double requires 64-bits of storage and has a lot of precision. You can't just throw away 48 bits of data in the round trip and expect to get the exact same value back. It is analogous to starting with a high resolution image, converting it to a thumbnail and then expecting that you can magically recover the original high resolution image. In the real world, the human ear is not going to be able to distinguish between the two. The higher resolution is useful during computation and signal processing routines to reduce the accumulation of computational errors. That being said, if you want to store 64-bit you'll need to use something other than .WAV. The closest you'll get is 32-bit.

Graphing wav file in java

I have been searching for this but none seems to answer my question.
I have been trying to graph/plot a wav file by this:
int result = 0;
try {
result = audioInputStream.read(bytes);
} catch (Exception e) {
e.printStackTrace();
}
and then using the result to be a variable for a graph. I've been thinking if it is correct to change first the result to decibels. Also, am I right to use the result as a variable to be use in the graph? Or is there any way that has to be use in graphing the wav file?

The first thing you need to do is read the samples of the file, this will give you the min/max ranges of the waveform (sound wave)...
File file = new File("...");
AudioInputStream ais = null;
try {
ais = AudioSystem.getAudioInputStream(file);
int frameLength = (int) ais.getFrameLength();
int frameSize = (int) ais.getFormat().getFrameSize();
byte[] eightBitByteArray = new byte[frameLength * frameSize];
int result = ais.read(eightBitByteArray);
int channels = ais.getFormat().getChannels();
int[][] samples = new int[channels][frameLength];
int sampleIndex = 0;
for (int t = 0; t < eightBitByteArray.length;) {
for (int channel = 0; channel < channels; channel++) {
int low = (int) eightBitByteArray[t];
t++;
int high = (int) eightBitByteArray[t];
t++;
int sample = getSixteenBitSample(high, low);
samples[channel][sampleIndex] = sample;
}
sampleIndex++;
}
} catch (Exception exp) {
exp.printStackTrace();
} finally {
try {
ais.close();
} catch (Exception e) {
}
}
//...
protected int getSixteenBitSample(int high, int low) {
return (high << 8) + (low & 0x00ff);
}
Then you would need to determine the min/max values, the next example simply checks for channel 0, but you could use the same concept to check all the available channels...
int min = 0;
int max = 0;
for (int sample : samples[0]) {
max = Math.max(max, sample);
min = Math.min(min, sample);
}
FYI: It would be more efficient to populate this information when you read the file
Once you have this, you can model the samples...but that would depend on framework you intend to use...

Storing a large binary file

Are there any ways to store a large binary file like 50 MB in the ten files with 5 MB?
thanks
are there any special classes for doing this?

Use a FileInputStream to read the file and a FileOutputStream to write it.
Here a simple (incomplete) example (missing error handling, writes 1K chunks)
public static int split(File file, String name, int size) throws IOException {
FileInputStream input = new FileInputStream(file);
FileOutputStream output = null;
byte[] buffer = new byte[1024];
int count = 0;
boolean done = false;
while (!done) {
output = new FileOutputStream(String.format(name, count));
count += 1;
for (int written = 0; written < size; ) {
int len = input.read(buffer);
if (len == -1) {
done = true;
break;
}
output.write(buffer, 0, len);
written += len;
}
output.close();
}
input.close();
return count;
}
and called like
File input = new File("C:/data/in.gz");
String name = "C:/data/in.gz.part%02d"; // %02d will be replaced by segment number
split(input, name, 5000 * 1024));

Yes, there are. Basically just count the bytes which you write to file and if it hits a certain limit, then stop writing, reset the counter and continue writing to another file using a certain filename pattern so that you can correlate the files with each other. You can do that in a loop. You can learn here how to write to files in Java and for the remnant just apply the primary school maths.