We Keep Coding

reading multiple parquet files in java takes unresonable amount of memory

Reading 20 uncompressed parquet files with total size 3.2GB, takes more then 12GB in RAM, when reading them "concurrently".
"concurrently" means that I need to read the second file before closing the first file, not multithreading.
The data is time series, so my program needs to read all the files up to some time, and then proceed.
I expect Arrow to use the amount of memory that corresponds to a single batch multiplied by the amount of files, but in reality the memory used is much more then the entire files.
The files were created with pandas default config (using pyarrow), and reading them in java gives the correct values.
when reading each file to the fullest, and then closing the file, the amount of ram used is ok.
I have tried to switch between the netty, and unsafe memory jars but they have the same results.
-Darrow.memory.debug.allocator=true did not produce any error.
trying to limit the amount of direct memory (the excess memory is outside of the JVM) I have tried to replace NativeMemoryPool.getDefault() with
NativeMemoryPool.createListenable(DirectReservationListener.instance()) or NativeMemoryPool.createListenable(.. some custom listener ..)
but the result is exception:
Exception in thread "main" java.lang.RuntimeException: JNIEnv was not attached to current thread
at org.apache.arrow.dataset.jni.JniWrapper.nextRecordBatch(Native Method)
at org.apache.arrow.dataset.jni.NativeScanner$NativeReader.loadNextBatch(NativeScanner.java:134)
at ParquetExample.main(ParquetExample.java:47)
using -XX:MaxDirectMemorySize=1g, -Xmx4g anyways had no effect.
the runtime is using env varibale:
_JAVA_OPTIONS="--add-opens=java.base/java.nio=ALL-UNNAMED"
on JDK 17.0.2 with arrow 9.0.0
the code is extracted to this simple example, taken from the official documentation:
import org.apache.arrow.dataset.file.FileFormat;
import org.apache.arrow.dataset.file.FileSystemDatasetFactory;
import org.apache.arrow.dataset.jni.NativeMemoryPool;
import org.apache.arrow.dataset.scanner.ScanOptions;
import org.apache.arrow.dataset.scanner.Scanner;
import org.apache.arrow.dataset.source.Dataset;
import org.apache.arrow.dataset.source.DatasetFactory;
import org.apache.arrow.memory.BufferAllocator;
import org.apache.arrow.memory.RootAllocator;
import org.apache.arrow.vector.VectorSchemaRoot;
import org.apache.arrow.vector.ipc.ArrowReader;
import java.io.IOException;
import java.nio.file.Files;
import java.nio.file.Path;
import java.util.ArrayList;
import java.util.List;
public class ParquetExample {
static BufferAllocator allocator = new RootAllocator(128 * 1024 * 1024); // limit does not affect problem
public static ArrowReader read_parquet_file(Path filePath, NativeMemoryPool nativeMemoryPool) {
String uri = "file:" + filePath;
ScanOptions options = new ScanOptions(/*batchSize*/ 64 * 1024 * 1024);
try (
DatasetFactory datasetFactory = new FileSystemDatasetFactory(
allocator, nativeMemoryPool, FileFormat.PARQUET, uri);
Dataset dataset = datasetFactory.finish()
) {
Scanner scanner = dataset.newScan(options);
return scanner.scan().iterator().next().execute();
} catch (Exception e) {
throw new RuntimeException(e);
}
}
public static void main(String[] args) throws IOException {
List<VectorSchemaRoot> schemaRoots = new ArrayList<>();
for (Path filePath : [...] ) { // 20 files, total uncompressed size 3.2GB
ArrowReader arrowReader = read_parquet_file(file,
NativeMemoryPool.getDefault());
if (arrowReader.loadNextBatch()) { // single batch read
schemaRoots.add(arrowReader.getVectorSchemaRoot());
}
}
}
}
the question is - why Arrow using so much memory in a straight-forward example, and why replacing the NativeMemoryPool results in crash?
Thanks

How to wait till an audio clip is loaded?

I'm quite a newbie in JAVA and I am trying to read a clip. Here is my code :
import java.io.BufferedInputStream;
import java.io.File;
import java.io.FileInputStream;
import java.io.InputStream;
import javax.sound.sampled.AudioFormat;
import javax.sound.sampled.AudioInputStream;
import javax.sound.sampled.AudioSystem;
import javax.sound.sampled.Clip;
import javax.sound.sampled.DataLine;
import javax.sound.sampled.LineEvent;
import javax.sound.sampled.LineListener;
public class TestClipBis {
protected static AudioFormat audioFormat;
public static void main(String[] args) throws Exception {
// Specification of the sound to play
// No control. We assume that the sound can be played on audio system
//File soundFile = new File("chimes.wav");
File soundFile = new File("test.wav");
InputStream is = new FileInputStream(soundFile);
InputStream bufferedIn = new BufferedInputStream(is);
//AudioInputStream sound = AudioSystem.getAudioInputStream(soundFile);
AudioInputStream sound = AudioSystem.getAudioInputStream(bufferedIn);
audioFormat = sound.getFormat();
System.out.println(audioFormat);
// Loading the sound into the memory (a Clip)
DataLine.Info info = new DataLine.Info(Clip.class, sound.getFormat());
System.out.println(info);
//Clip clip = (Clip) AudioSystem.getClip();
Clip clip = (Clip) AudioSystem.getLine(info);
System.out.println("Sound frame lenght : "+sound.getFrameLength());
System.out.println("Clip FrameLength before opening : "+clip.getFrameLength());
System.out.println("Clip will open - "+info);
System.out.println("Info format : "+info.getLineClass());
// Check before this line that everything is in memory
// Yes, but how ?
clip.open(sound);
System.out.println("Clip is open");
System.out.println("Clip FrameLength after opening : "+clip.getFrameLength());
// Due to a bug in Java Sound,
// we explicitly out of the VM when the sounds stop
clip.addLineListener(new LineListener() {
public void update(LineEvent event) {
if (event.getType() == LineEvent.Type.STOP) {
System.out.println("Methode de sortie");
event.getLine().close();
System.exit(0);
}
}
});
// Playing the clip
clip.start();
System.out.println("IsActive : "+clip.isActive());
//clip.close();
}
}
My problem is how to be sure that the clip is loaded in memory before opening and playing it ? With the above code, when I open and play the sound file, I have few seconds of playing but never the same length, randomly, and never the full song.
Or should I use something else than a clip to play a song? But I want to "move" into the song and not only streaming it from the start to the end.
Edit:
Ok, I tried few things. First, I tried to see if the "ByteArrayOuptputStream" was written. I had a "println" in the loop and yes, all is written but it don't fix the problem.
Then, when I open the clip, I tried to add the parameters : audioformat, bytearray, startpoint, bufferlength. Nothing better.
Then, I noticed that when the sounds stop, the method to exit was used. So, I tried to "mute" that method (with comment signs). The result is different : it read the file but the sound is jerky. And when I check the CPU use, it's around 100%. Is it a first clue to guess what's the problem ?
I tried to make a loop that indicates the frameposition after the start : all the frames are read but the sound is still jerky.
I also tried the thead.sleep before and after the start method : nothing better.
So, here is the code I use. Many code parts are between "comment quotes" because they are try, unsuccessfull...
import java.io.ByteArrayInputStream;
import java.io.ByteArrayOutputStream;
import java.io.File;
import java.io.FileInputStream;
import java.nio.ByteBuffer;
import javax.sound.sampled.AudioFormat;
import javax.sound.sampled.AudioInputStream;
import javax.sound.sampled.AudioSystem;
import javax.sound.sampled.Clip;
import javax.sound.sampled.DataLine;
import javax.sound.sampled.LineEvent;
import javax.sound.sampled.LineListener;
public class TestBufferedClip {
protected static AudioFormat audioFormat;
public static ByteBuffer buffer;
public static void main(String[] args) throws Exception {
// Specification of the sound to play
// No control. We assume that the sound can be played on audio system
//File soundFile = new File("chimes.wav");
File soundFile = new File("test.wav");
FileInputStream fis = new FileInputStream(soundFile);
ByteArrayOutputStream baos = new ByteArrayOutputStream((int)soundFile.length());
byte[] buf = new byte[1024];
int n = 0;
int loop = 0;
while ((n=fis.read(buf))>=0) {
baos.write(buf);
buf=new byte[1024];
System.out.println("Loop = "+loop);
loop+=1;
}
byte[] ba = baos.toByteArray();
System.out.println("ByteArray size : "+ba.length);
ByteArrayInputStream bais = new ByteArrayInputStream(ba);
//AudioInputStream sound = AudioSystem.getAudioInputStream(soundFile);
AudioInputStream sound = AudioSystem.getAudioInputStream(bais);
audioFormat = sound.getFormat();
System.out.println(audioFormat);
// Loading the sound into the memory (a Clip)
DataLine.Info info = new DataLine.Info(Clip.class, sound.getFormat());
System.out.println("Info :"+info);
//Clip clip = (Clip) AudioSystem.getClip();
Clip clip = (Clip) AudioSystem.getLine(info);
System.out.println("Sound frame lenght : "+sound.getFrameLength());
System.out.println("Info format : "+info.getLineClass());
// Check before this line that everything is in memory
// Yes, but how ?
clip.open(audioFormat, ba, 0, ba.length);
//clip.open(sound);
System.out.println("Clip is open");
System.out.println("Clip FrameLength after opening : "+clip.getFrameLength());
// Due to a bug in Java Sound,
// we explicitly out of the VM when the sounds stop
/*
clip.addLineListener(new LineListener() {
public void update(LineEvent event) {
if (event.getType() == LineEvent.Type.STOP) {
System.out.println("Methode de sortie");
event.getLine().close();
System.exit(0);
}
}
});
*/
// Playing the clip
System.out.println("Before thread sleep");
try {
Thread.sleep(31000);
} catch (InterruptedException e){
e.printStackTrace();
}
System.out.println("After thread sleep");
clip.start();
System.out.println("IsActive : "+clip.isActive());
/*
while (clip.isActive()==true) {
System.out.println("Position = "+clip.getFramePosition());
}
*/
//clip.close();
}
}
#Phil Freihofner :
I thought about your solution to read and discards data until I reach my "start" point. You wrote "In order to start at a point within the audio file, using a SourceDataLine, you would have to read and discard data from the audio input line until you got to the desired starting spot". How do you do that ? When I use the "SourceDataLine" method, my start method is a loop with a line.write(bytes, 0, bytesreads); to point the sound on the speakers.
So, how do you just read and discard ? I didn't find any "read" method with the line.

javax.sound.sampled supports two objects for playing back audio. The Clip, as you are using, has to be loaded completely into memory before one can play it back. On the plus side, it is also easy to position the playback to start from within the Clip, either using microseconds or frame position.
I see no benefit from first loading the sound into a byte buffer. That is a redundant level of buffering for Clips. I'd only do it if you were trying to do DSP or something else that requires getting to the data, something beyond the Clip's built in ability to set a start point.
If you are able to preload the possible audio choices as Clips before they are selected, that might be the best solution, as long as you don't run out of RAM.
The other option for playback is a SourceDataLine. It reads and plays back the file on a per-buffer basis. Thus, it can start up quicker than an unloaded Clip (no need to load the entire file into memory first). However, once the Clip is preloaded, the Clip will play back without having to do repeated file loads.
In order to start at a point within the audio file, using a SourceDataLine, you would have to read and discard data from the audio input line until you got to the desired starting spot. You can do this by counting frames (the format will tell you the number of bytes per frame). This reading and discarding would disrupt timing a bit, but my experience has been that reading and discarding audio data is a couple of orders of magnitude faster than playback, since playback involves blocking queues to keep the output at the required frame rate.
Check the Java Sound Tutorials for more info, which includes links to the Clip and SourceDataLine APIs.
Here is an example of the loading of a Clip:
File soundFile = new File("test.wav");
AudioInputStream sound = AudioSystem.getAudioInputStream(soundFile);
DataLine.Info info = new DataLine.Info(Clip.class, sound.getFormat());
Clip clip = (Clip) AudioSystem.getLine(info);
clip.open(sound);
The data from test.wav should now be in RAM. All that other stuff you have using byte buffers and buffered lines is unnecessary.
When you are ready to play, use clip.setMicrosecondPosition(long milliseconds) to set your sound to start at the desired location, (not needed if you are starting from the beginning, unless you've already played the Clip, in which case the position will be where it was when you stopped). Then use clip.start() to commence playing.
IMPORTANT NOTE: playback will end prematurely if the program running it exits. One way to test this is to put a Thread.sleep(long milliseconds) command after the clip.start(), where the value of milliseconds is longer than the length of the clip. But that is not a real solution, just a diagnostic to prevent the program from closing the clip playback thread. You should be handling keeping the program running from the main threads, not the thread with the audio playback.

First read whole into a byte buffer. Do this by copying all content from file to ByteArrayOutputStream. This way you will have whole media content in memory. Now you can wrap array from ByteArrayOutputStream.toByteArray() into ByteArrayInputStream and provide that pure in-memory stream as audio input stream.

How to get exact size of zipped file before zipping?

I am using following standalone class to calculate size of zipped files before zipping.
I am using 0 level compression, but still i am getting a difference of few bytes.
Can you please help me out in this to get exact size?
Quick help will be appreciated.
import java.io.ByteArrayInputStream;
import java.io.ByteArrayOutputStream;
import java.io.File;
import java.io.FileInputStream;
import java.io.FileNotFoundException;
import java.io.FileOutputStream;
import java.io.IOException;
import java.io.InputStream;
import java.util.zip.CRC32;
import java.util.zip.ZipEntry;
import java.util.zip.ZipInputStream;
import java.util.zip.ZipOutputStream;
import org.apache.commons.io.FilenameUtils;
public class zipcode {
/**
* #param args
*/
public static void main(String[] args) {
// TODO Auto-generated method stub
try {
CRC32 crc = new CRC32();
byte[] b = new byte[1024];
File file = new File("/Users/Lab/Desktop/ABC.xlsx");
FileInputStream in = new FileInputStream(file);
crc.reset();
// out put file
ZipOutputStream out = new ZipOutputStream(new FileOutputStream("/Users/Lab/Desktop/ABC.zip"));
// name the file inside the zip file
ZipEntry entry = new ZipEntry("ABC.xlsx");
entry.setMethod(ZipEntry.DEFLATED);
entry.setCompressedSize(file.length());
entry.setSize(file.length());
entry.setCrc(crc.getValue());
out.setMethod(ZipOutputStream.DEFLATED);
out.setLevel(0);
//entry.setCompressedSize(in.available());
//entry.setSize(in.available());
//entry.setCrc(crc.getValue());
out.putNextEntry(entry);
// buffer size
int count;
while ((count = in.read(b)) > 0) {
System.out.println();
out.write(b, 0, count);
}
out.close();
in.close();
} catch (FileNotFoundException e) {
// TODO Auto-generated catch block
e.printStackTrace();
} catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}
}

Firstly, I'm not convinced by explanation for why you need to do this. There is something wrong with your system design or implementation if it is necessary to know the file size before you start uploading.
Having said that, the solution is basically to create the ZIP file on the server side so that you know its size before you start uploading it to the client:
Write the ZIP file to a temporary file and upload from that.
Write the ZIP file to an buffer in memory and upload from that.
If you don't have either the file space or the memory space on the server side, then:
Create "sink" outputStream that simply counts the bytes that are written to calculate the nominal file size.
Create / write the ZIP file to the sink, and capture the file size.
Open your connection for uploading.
Send the metadata including the file size.
Create / write the ZIP a second time, writing to the socket stream ... or whatever.
These 3 approaches will all allow you to create and send a compressed ZIP, if that is going to help.
If you insist on trying to do this on-the-fly in one pass, then you are going to need to read the ZIP file spec in forensic detail ... and do some messy arithmetic. Helping you is probably beyond the scope of a SO question.

I had to do this myself to write the zip results straight to AWS S3 which requires a file size. Unfortunately there is no way I found to compute the size of a compressed file without performing the computation on each block of data.
One method is to zip everything twice. The first time you throw out the data but add up the number of bytes:
long getSize(List<InputStream> files) throws IOException {
final AtomicLong counter = new AtomicLong(0L);
final OutputStream countingStream = new OutputStream() {
#Override
public void write(int b) throws IOException {
counter.incrementAndGet();
}
};
ZipOutputStream zoutcounter = new ZipOutputStream(countingStream);
// Loop through files or input streams here and do compression
// ...
zoutcounter.close();
return counter.get();
}
The alternative is to do the above creating an entry for each file but then don't write any actual data (don't call write()) so you can compute the total size of just the zip entry headers. This will only work if you turn off compression like this:
entry.setMethod(ZipEntry.STORED);
The size of the zip entries plus the size of each uncompressed file should give you an accurate final size, but only with compression turned off. You don't have to set the CRC values or any of those other fields when computing the zip file size as those entries always have the same size in the final entry header. It's only the name, comment and extra fields on the ZipEntry that vary in size. The other entries like the file size, CRC, etc. take up the same space in the final zip file whether or not they were set.
There is one more solution you can try. Guess the size conservatively and add a safety margin, then compress it aggressively. Pad the rest of the file until it equals your estimated size. Zip ignores padding. If you implement an output stream that wrappers your actual output stream but implements the close operation as a noop then you can pass that as the output stream for your ZipOutputStream. After you close your ZipOutputStream instance, write the padding to the actual output stream to equal your estimated number of bytes, then close it for real. The file will be larger than it could be but you save the computation of the accurate file size and the result will benefit from at least some compression.

Concatenation of two WAV files failed

I have this simple code to concatenate two wav files. Its pretty simple and the code runs without any errors. But there is a problem with the output file. The output file generated does not play, and surprisingly its size is only 44 bytes whereas my input files "a.wav" & "b.wav" are both more than 500Kb in size.
Here is my code:
import java.io.File;
import java.io.IOException;
import java.io.SequenceInputStream;
import javax.sound.sampled.AudioFileFormat;
import javax.sound.sampled.AudioInputStream;
import javax.sound.sampled.AudioSystem;
public class WavConcat {
public static void main(String[] args) {
String wFile1 = "./sounds/a.wav";
String wFile2 = "./sounds/b.wav";
try {
AudioInputStream clip1 = AudioSystem.getAudioInputStream(new File(wFile1));
AudioInputStream clip2 = AudioSystem.getAudioInputStream(new File(wFile2));
AudioInputStream appendedFiles =
new AudioInputStream(
new SequenceInputStream(clip1, clip2),
clip1.getFormat(),
clip1.getFrameLength() + clip2.getFrameLength());
AudioSystem.write(appendedFiles,
AudioFileFormat.Type.WAVE,new File("./sounds/ab.wav"));
} catch (Exception e) {
e.printStackTrace();
}
}
}

Try this kind of structure. This worked for me
List audioInputStreamList = new ArrayList();
String wFile1 = "./sounds/a.wav";
String wFile2 = "./sounds/b.wav";
AudioInputStream audioInputStream1 = AudioSystem.getAudioInputStream(new File(wFile1));
AudioInputStream audioInputStream2 = AudioSystem.getAudioInputStream(new File(wFile2));
audioInputStreamList.add(audioInputStream1);
audioInputStreamList.add(audioInputStream2);
AudioFormat audioFormat = audioInputStream1.getFormat(); // audioInputStream2 format also same
AudioInputStream udioInputStream = new SequenceAudioInputStream(audioFormat,audioInputStreamList);
AudioSystem.write(audioInputStream, AudioFileFormat.Type.WAVE,new File("./sounds/ab.wav"));
UPDATE
check for SequenceAudioInputStream

clip1.getFormat() returns-->
MPEG2L3 24000.0 Hz, unknown bits per sample, mono, unknown frame size, 41.666668 frames/second
clip2.getFormat() returns-->
MPEG2L3 24000.0 Hz, unknown bits per sample, mono, unknown frame size, 41.666668 frames/second
That is an odd format. I can imagine the 'unknown bits per sample' is causing a problem, but also the MPEG2L3, since JavaSound has no inbuilt encoder for MP3. It seems like they are not encoded properly. Try loading them in sound editing software and save them as a type of WAV or AU that Java Sound can understand 'out of the box'. Hopefully the editing software:
Can understand the broken MP3, and..
Will write a valid WAV or AU.
If you can convert them to 8 bit mono & 8KHz during the conversion, it might reduce the byte[] size by a factor of 6 to 1. 8KHz is considered good enough to understand speech, and for this use you need to serve the bytes of the combined sound out to the browser - so reducing it in size is crucial.

Join two WAV files from Java?

What's the simplest way to concatenate two WAV files in Java 1.6? (Equal frequency and all, nothing fancy.)
(This is probably sooo simple, but my Google-fu seems weak on this subject today.)

Here is the barebones code:
import java.io.File;
import java.io.IOException;
import java.io.SequenceInputStream;
import javax.sound.sampled.AudioFileFormat;
import javax.sound.sampled.AudioInputStream;
import javax.sound.sampled.AudioSystem;
public class WavAppender {
public static void main(String[] args) {
String wavFile1 = "D:\\wav1.wav";
String wavFile2 = "D:\\wav2.wav";
try {
AudioInputStream clip1 = AudioSystem.getAudioInputStream(new File(wavFile1));
AudioInputStream clip2 = AudioSystem.getAudioInputStream(new File(wavFile2));
AudioInputStream appendedFiles =
new AudioInputStream(
new SequenceInputStream(clip1, clip2),
clip1.getFormat(),
clip1.getFrameLength() + clip2.getFrameLength());
AudioSystem.write(appendedFiles,
AudioFileFormat.Type.WAVE,
new File("D:\\wavAppended.wav"));
} catch (Exception e) {
e.printStackTrace();
}
}
}

The WAV header should be not be too hard to parse, and if I read this header description correctly, you can just strip the first 44 bytes from the second WAV and simply append the bytes to the first one. After that, you should of course change some of the header fields of the first WAV so that they contain the correct new length.

I found this (AudioConcat) via the "Code Samples & Apps" link on here.

Your challenge though occurs if the two WAV files don't have the exact same format in the wave header.
If the wave formats on the two files aren't the same, you're going to have to find a way to transmogrify them so they match.
That may involve an MP3 transcode or other kinds of transcoding (if one of them is encoded with an MP3 codec).