We Keep Coding

Java TargetDataLine not picking up any audio?

I'm writing a function to capture an audio clip for ~ 7.5 seconds using a TargetDataLine. The code executes and renders an 'input.wav' file, but when I play it there is no sound.
My approach, as shown in the code at the bottom of this post, is to do the following things:
Create an AudioFormat and get the Info for a Target Data Line.
Create the Target Data Line by getting the line from AudioSystem.
Open and Start the TargetDataLine, which allocates system resources for recording.
Create an auxiliary Thread that will record audio by writing to a file.
Start the auxiliary Thread, pause the main Thread in the meantime, and then close out the Target Data Line in order to stop recording.
What I have tried so far:
Changing the AudioFormat. Initially, I was using the other AudioFormat constructor which takes the file type as well (where the first argument is AudioFormat.Encoding.PCM_SIGNED etc). I had a sample rate of 44100, 16 bits, 2 channels and small-Endian settings on the other format, which yielded the same result.
Changing the order of commands on my auxiliary and main Thread (i.e. performing TLine.open() or start() in alternate locations).
Checking that my auxiliary thread does actually start.
For reference I am using IntelliJ on a Mac OS Big Sur.
public static void captureAudio() {
try {
AudioFormat f = new AudioFormat(22050, 8, 1, false, false);
DataLine.Info secure = new DataLine.Info(TargetDataLine.class, f);
if (!AudioSystem.isLineSupported(secure)) {
System.err.println("Unsupported Line");
}
TargetDataLine tLine = (TargetDataLine)AudioSystem.getLine(secure);
System.out.println("Starting recording...");
tLine.open(f);
tLine.start();
File writeTo = new File("input.wav");
Thread t = new Thread(){
public void run() {
try {
AudioInputStream is = new AudioInputStream(tLine);
AudioSystem.write(is, AudioFileFormat.Type.WAVE, writeTo);
} catch(IOException e) {
System.err.println("Encountered system I/O error in recording:");
e.printStackTrace();
}
}
};
t.start();
Thread.sleep(7500);
tLine.stop();
tLine.close();
System.out.println("Recording has ended.");
} catch(Exception e) {
e.printStackTrace();
}
}
Update 1: Some new testing and results
My microphone and speakers are both working with other applications - recorded working audio with QuickTimePlayer.
I did a lot of testing around what my TargetDataLines are and what the deal is with them. I ran the following code:
public static void main(String[] args) {
AudioFormat f = new AudioFormat(48000, 16, 2, true, false);
//DataLine.Info inf = new DataLine.Info(SourceDataLine.class, f);
try {
TargetDataLine line = AudioSystem.getTargetDataLine(f);
DataLine.Info test = new DataLine.Info(TargetDataLine.class, f);
TargetDataLine other = (TargetDataLine)AudioSystem.getLine(test);
String output = line.equals(other) ? "Yes" : "No";
if (output.equals("No")) {
System.out.println(other.toString());
}
System.out.println(line.toString());
System.out.println("_______________________________");
for (Mixer.Info i : AudioSystem.getMixerInfo()) {
Line.Info[] tli = AudioSystem.getMixer(i).getTargetLineInfo();
if (tli.length != 0) {
Line comp = AudioSystem.getLine(tli[0]);
System.out.println(comp.toString() + ":" +i.getName());
if (comp.equals(line) || comp.equals(other)) {
System.out.println("The TargetDataLine is from " + i.getName());
}
}
}
} catch (Exception e) {
e.printStackTrace();
}
}
Long story short, the TargetDataLine I receive from doing
TargetDataLine line = AudioSystem.getTargetDataLine(f); and
TargetDataLine other = (TargetDataLine)AudioSystem.getLine(new DataLine.Info(TargetDataLine.class, f));
are different, and furthermore, don't match any of the TargetDataLines that are associated with my system's mixers.
The output of the above code was this (where there first lines are other and line respectively):
com.sun.media.sound.DirectAudioDevice$DirectTDL#cc34f4d
com.sun.media.sound.DirectAudioDevice$DirectTDL#17a7cec2
_______________________________
com.sun.media.sound.PortMixer$PortMixerPort#79fc0f2f:Port MacBook Pro Speakers
com.sun.media.sound.PortMixer$PortMixerPort#4d405ef7:Port ZoomAudioDevice
com.sun.media.sound.DirectAudioDevice$DirectTDL#3f91beef:Default Audio Device
com.sun.media.sound.DirectAudioDevice$DirectTDL#1a6c5a9e:MacBook Pro Microphone
com.sun.media.sound.DirectAudioDevice$DirectTDL#37bba400:ZoomAudioDevice
Upon this realization I manually loaded up all the TargetDataLines from my mixers and tried recording audio with each of them to see if I got any sound.
I used the following method to collect all the TargetDataLines:
public static ArrayList<Line.Info> allTDL() {
ArrayList<Line.Info> all = new ArrayList<>();
for (Mixer.Info i : AudioSystem.getMixerInfo()) {
Line.Info[] tli = AudioSystem.getMixer(i).getTargetLineInfo();
if (tli.length != 0) {
for (int f = 0; f < tli.length; f += 1) {
all.add(tli[f]);
}
}
}
return all;
}
My capture/record audio method remained the same, except for switching the format to AudioFormat f = new AudioFormat(48000, 16, 2, true, false);, changing the recording time to 5000 milliseconds, and writing the method header as public static void recordAudio(Line.Info inf) so I could load each TargetDataLine individually with it's info.
I then executed the following code to rotate TargetDataLines:
public static void main(String[] args) {
for (Line.Info inf : allTDL()) {
recordAudio(inf);
try {
Thread.sleep(5000);
} catch(Exception e) {
e.printStackTrace();
}
if (!soundless(loadAsBytes("input.wav"))) {
System.out.println("The recording with " + inf.toString() + " has sound!");
}
System.out.println("The last recording with " + inf.toString() + " was soundless.");
}
}
}
The output was as such:
Recording...
Was unable to cast com.sun.media.sound.PortMixer$PortMixerPort#506e1b77 to a TargetDataLine.
End recording.
The last recording with SPEAKER target port was soundless.
Recording...
Was unable to cast com.sun.media.sound.PortMixer$PortMixerPort#5e9f23b4 to a TargetDataLine.
End recording.
The last recording with ZoomAudioDevice target port was soundless.
Recording...
End recording.
The last recording with interface TargetDataLine supporting 8 audio formats, and buffers of at least 32 bytes was soundless.
Recording...
End recording.
The last recording with interface TargetDataLine supporting 8 audio formats, and buffers of at least 32 bytes was soundless.
Recording...
End recording.
The last recording with interface TargetDataLine supporting 14 audio formats, and buffers of at least 32 bytes was soundless.
TL;DR the audio came out soundless for every TargetDataLine.
For completeness, here are the soundless and loadAsBytes functions:
public static byte[] loadAsBytes(String name) {
assert name.contains(".wav");
ByteArrayOutputStream out = new ByteArrayOutputStream();
File retrieve = new File("src/"+ name);
try {
InputStream input = AudioSystem.getAudioInputStream(retrieve);
int read;
byte[] b = new byte[1024];
while ((read = input.read(b)) > 0) {
out.write(b, 0, read);
}
out.flush();
byte[] full = out.toByteArray();
return full;
} catch(UnsupportedAudioFileException e) {
System.err.println("The File " + name + " is unsupported on this system.");
e.printStackTrace();
} catch (IOException e) {
System.err.println("Input-Output Exception on retrieval of file " + name);
e.printStackTrace();
}
return null;
}
static boolean soundless(byte[] s) {
if (s == null) {
return true;
}
for (int i = 0; i < s.length; i += 1) {
if (s[i] != 0) {
return false;
}
}
return true;
}
I'm not really sure what the issue could be at this point save for an operating system quirk that doesn't allow Java to access audio lines, but I do not know how to fix that - looking at System Preferences there isn't any obvious way to allow access. I think it might have to be done with terminal commands but also not sure of precisely what commands I'd have to execute there.

I'm not seeing anything wrong in the code you are showing. I haven't tried testing it on my system though. (Linux, Eclipse)
It seems to me your code closely matches this tutorial. The author Nam Ha Minh is exceptionally conscienscious about answering questions. You might try his exact code example and consult with him if his version also fails for you.
But first, what is the size of the resulting .wav file? Does the file size match the amount of data expected for the duration you are recording? If so, are you sure you have data incoming from your microphone? Nam has another code example where recorded sound is progressively read and placed into memory. Basically, instead of using the AudioInputStream as a parameter to the AudioSystem.write method, you execute multiple read method calls on the AudioInputStream and inspect the incoming data directly. That might be helpful for trouble-shooting whether the problem is occurring on the incoming vs outgoing part of the process.
I'm not knowledgeable enough about formats to know if the Mac does things differently. I'm surprised you are setting the format to unsigned. For my limited purposes, I stick with "CD quality stereo" and signed PCM at all junctures.
EDIT: based on feedback, it seems that the problem is that the incoming line is not returning data. From looking at other, similar tutorials, it seems that several people have had the same problem on their Mac systems.
First thing to verify: does your microphone work with other applications?
As far as next steps, I would try verifying the chosen line. The lines that are exposed to java can be enumerated/inspected. The tutorial Accessing Audio System Resources has some basic information on how to do this. It looks like AudioSystem.getMixerInfo() will return a list of available mixers that can be inspected. Maybe AudioSystem.getTargetLineInfo() would be more to the point.
I suppose it is possible that the default Line or Port being used when you obtain a TargetDataLine isn't the one that is running the microphone. If a particular line or port turns out to be the one you need, then it can be specified explicitly via an overridden getTargetDataLine method.
I'm reading that there might be a security policy that needs to be handled. I don't fully understand the code, but if that were the issue, an Exception presumably would have been thrown. Perhaps there are new security measures coming from the MacOs, to prevent an external program from opening a mic line surreptitiously?
If you do get this solved, be sure and post the answer and mark it solved. This seems to be a live question for many people.

Java: Append zeros (silence) to a sound file

I want to implement a metronome app in Java for playing beats in complicated rhythm patterns. There are many kinds of beats, (drums and other percussion instruments) that's why using timers and threads may cause a not precise and optimal performance. I decided firstly to generate the sound by adding silence intervals for each instrument and then to mix them together for better performance (not sure if this is the best solution, but anyway). Now my problem is to add silence intervals to each beat.
public AudioStream create(String file, String rhythm) {
InputStream in = null;
AudioStream audioStream = null;
try {
in = new FileInputStream(path + file);
audioStream = new AudioStream(in);
} catch (FileNotFoundException e1) {
e1.printStackTrace();
} catch (IOException e) {
e.printStackTrace();
}
}
This function should create and return the sample for one instrument. So how can I add silence milliseconds to the file and then return the final sample?

Thread not releasing memory

I'm trying to play a sound effect in my program using threads, I searched the web and as I understand when a thread reaches the end of the run function it will become free for the GC to collect.
However when I call for the function many times one after another the task manager shows a high increase at memory usage and it never went back down, I waited for 2 minutes for the GC but there was no effect.
Here is the code that I use for playing sound effect:
public static void playSfx(final String path) {
new Thread(new Runnable() {
public void run() {
try {
AudioInputStream audioInputStream = AudioSystem.getAudioInputStream(new File(path));
final int BUFFER_SIZE = 128000;
SourceDataLine sourceLine = null;
AudioFormat audioFormat = audioInputStream.getFormat();
DataLine.Info info = new DataLine.Info(SourceDataLine.class, audioFormat);
sourceLine = (SourceDataLine) AudioSystem.getLine(info);
sourceLine.open(audioFormat);
if (sourceLine == null)
return;
sourceLine.start();
int nBytesRead = 0;
byte[] abData = new byte[BUFFER_SIZE];
while (nBytesRead != -1) {
try {
nBytesRead = audioInputStream.read(abData, 0, abData.length);
} catch (IOException e) {
e.printStackTrace();
}
if (nBytesRead >= 0) {
sourceLine.write(abData, 0, nBytesRead);
}
}
sourceLine.drain();
sourceLine.close();
audioInputStream.close();
} catch (IOException e) {
e.printStackTrace();
} catch (Exception e) {
e.printStackTrace();
System.exit(1);
}
}
}).start();
}
What should I do to reclaim the lost memory?

I can't reproduce this. For me this will peak at about 60k memory usage, which
considering the language in question is perfectly ok imo. Do you have any small
code which reliably reproduces the problem?
public static void main(String[] args) {
for(int i = 0; i < 1000; i++) {
playSfx("somesortsound.wav");
try {
Thread.sleep(100);
} catch (Exception e) {
}
}
}
In addition, what are your expectations when it comes to memory usage? Predicting GC
behaviour is not easy, and you are not guaranteed to see memory freed just because
you wait a while. Especially so if you are not allocating a lot.
I would also get rid of the try catch in your read loop. You don't handle any
exception there anyway, and you catch it outside of the loop as well. If the
stream causes an IO exception chances are you want to abort the loop anyway, right?

Running that method from the following class with compatible files and without any exceptions; I did not see that behaviour.
public class PlaySound {
public static void main(String[] args) throws Exception {
String filePath = "C:"+File.separator+"bach.wav";
playSfx(filePath);
playSfx(filePath);
playSfx(filePath);
while(true){
Thread.sleep(1000);
}
}
}
It could be a memory leak. You are not managing the resources with a finally block (or 'try-with-resources'), coupling this with return statements could result in the internals of the AudioInputStream driver maintaining a reference to the file.
I would use a try-with-resources block.
try(AudioInputStream audioInputStream = AudioSystem.getAudioInputStream(new File(path));){
//your code
}
Also, I wouldn't use the task manager to reliably determine the garbage collectors behaviour, there are many components that determine the size of a Java process in memory.
To gain an understanding of what's going on; use VisualVM. (This is located in the bin directory of your jdk). You can see when the garbage collector is active, as well as the size of the heap compared to its current maximum size. You can also 'Request' (does not guarantee) garbage collection, which will give you an indication of the amount of memory required for 'survivors', and the amount that can be freed.
Admittedly 2 minutes is a long time, but it is worth mentioning garbage collection usually occurs as a failure to allocate memory to a particular generation, but this is dependant on the particular GC algorithm being used. In other words, if it didn't reach an upper limit then it is possible it wouldn't have performed a garbage collection.
This article is potentially worth a read/glance; it details various different garbage collectors and how they are expected to function.

Playing wavs in Java

So, I'm working on a project for class wherein we have to have a game with background music. I'm trying to play a .wav file as background music, but since I can't use clips (too short for a music file) I have to play with the AudioStream.
In my first implementation, the game would hang until the song finished, so I threw it into its own thread to try and alleviate that. Currently, the game plays very slowly while the song plays. I'm not sure what I need to do to make this thread play nice with my animator thread, because we we're never formally taught threads. Below is my background music player class, please someone tell me what I've done wrong that makes it hog all the system resources.
public class BGMusicPlayer implements Runnable {
File file;
AudioInputStream in;
SourceDataLine line;
int frameSize;
byte[] buffer = new byte [32 * 1024];
Thread player;
boolean playing = false;
boolean fileNotOver = true;
public BGMusicPlayer (File inputFile){
try{
file = inputFile;
in = AudioSystem.getAudioInputStream (inputFile);
AudioFormat format = in.getFormat();
frameSize = format.getFrameSize();
DataLine.Info info =new DataLine.Info (SourceDataLine.class, format);
line = (SourceDataLine) AudioSystem.getLine (info);
line.open();
player = new Thread (this);
player.start();
}
catch(Exception e){
System.out.println("That is not a valid file. No music for you.");
}
}
public void run() {
int readPoint = 0;
int bytesRead = 0;
player.setPriority(Thread.MIN_PRIORITY);
while (fileNotOver) {
if (playing) {
try {
bytesRead = in.read (buffer,
readPoint,
buffer.length - readPoint);
} catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
if (bytesRead == -1) {
fileNotOver = false;
break;
}
int leftover = bytesRead % frameSize;
// send to line
line.write (buffer, readPoint, bytesRead-leftover);
// save the leftover bytes
System.arraycopy (buffer, bytesRead,
buffer, 0,
leftover);
readPoint = leftover;
try {
Thread.sleep(20);
} catch (InterruptedException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}
}
}
public void start() {
playing = true;
if(!player.isAlive())
player.start();
line.start();
}
public void stop() {
playing = false;
line.stop();
}
}

You are pretty close, but there are a couple of unusual things that maybe are contributing to the performance problem.
First off, if you are just playing back a .wav, there shouldn't really be a need to deal with any "readpoint" but a value of 0, and there shouldn't really be a need for a "leftover" computation. When you do the write, it should simply be the same number of bytes that were read in (return value of the read() method).
I'm also unclear why you are doing the ArrayCopy. Can you lose that?
Setting the Thread to low priority, and putting a Sleep--I guess you were hoping those would slow down the audio processing to allow more of your game to process? I've never seen this done before and it is really unusual if it is truly needed. I really recommend getting rid of these as well.
I'm curious where your audio file is coming from. Your not streaming it over the web, are you?
By the way, the way you get your input from a File and place it into an InputStream very likely won't work with Java7. A lot of folks are reporting a bug with that. It turns out it is more correct and efficient to generate a URL from the File, and then get the AudioInputStream using the URL as the argument rather than the file. The error that can come up is a "Mark/Reset" error. (A search on that will show its come up a number of times here.)

Recording multiplexed Audio/Video to a file using JMF

I have a project that uses JMF, and records for a short time (a few seconds to a couple of minutes) both the web camera, and audio inputs, and then writes the results to a file.
The problem with my project is that this file is never produced properly, and cannot be played back.
While I've found numerous examples of how to do multiplexed transmission of audio and video over RTP, or conversion of an input file from one format to another , I haven't seen a working example yet that captures audio and video, and writes it to a file.
Does anyone have an example of functioning code to do this?

I've found the reason why I was not able to generate a file from two separate capture devices under JMF, and it relates to ordering of the start commands. In particular, things like Processors will take a datasource, or merging datasource, assign and synchronize the time base(s) and start/stop the sources for you, so the extra work I was trying to do starting the datasources manually is utterly redundant, and throws a wrench in the works.
This was a lot of painful trial and error, and I would suggest you read every line of code, understand the sequencing, and understand what has been included, and what has been left out and why before trying to implement this yourself. JMF is quite the bear if you're not careful.
Oh, and remember to catch exceptions. I had to omit that code due to length restrictions.
Here's my final solution:
public void doRecordingDemo() {
// Get the default media capture device for audio and video
DataSource[] sources = new DataSource[2];
sources[0] = Manager.createDataSource(audioDevice.getLocator());
sources[1] = Manager.createDataSource(videoDevice.getLocator());
// Merge the audio and video streams
DataSource source = Manager.createMergingDataSource(sources);
// Create a processor to convert from raw format to a file format
// Notice that we are NOT starting the datasources, but letting the
// processor take care of this for us.
Processor processor = Manager.createProcessor(source);
// Need a configured processor for this next step
processor.configure();
waitForState(processor, Processor.Configured);
// Modify this to suit your needs, but pay attention to what formats can go in what containers
processor.setContentDescriptor(new FileTypeDescriptor(FileTypeDescriptor.QUICKTIME));
// Use the processor to convert the audio and video into reasonable formats and sizes
// There are probably better ways to do this, but you should NOT make any assumptions
// about what formats are supported, and instead use a generic method of checking the
// available formats and sizes. You have been warned!
for (TrackControl control : processor.getTrackControls()) {
if (control.getFormat() instanceof VideoFormat || control.getFormat() instanceof AudioFormat) {
if (control.getFormat() instanceof AudioFormat) {
// In general, this is safe for audio, but do not make assumptions for video.
// Things get a little wonky for video because of how complex the options are.
control.setFormat(new AudioFormat(AudioFormat.GSM));
}
if (control.getFormat() instanceof VideoFormat) {
VideoFormat desiredVideoFormat = null;
Dimension targetDimension = new Dimension(352, 288);
// Search sequentially through this array of formats
VideoFormat[] desiredFormats = new VideoFormat[] {new H263Format(), new JPEGFormat(), new RGBFormat(), new YUVFormat()};
for (VideoFormat checkFormat : desiredFormats) {
// Search the video formats looking for a match.
List<VideoFormat> candidates = new LinkedList<VideoFormat>();
for (Format format : control.getSupportedFormats()) {
if (format.isSameEncoding(checkFormat)) {
candidates.add((VideoFormat) format);
}
}
if (!candidates.isEmpty()) {
// Get the first candidate for now since we have at least a format match
desiredVideoFormat = candidates.get(0);
for (VideoFormat format : candidates) {
if (targetDimension.equals(format.getSize())) {
// Found exactly what we're looking for
desiredVideoFormat = format;
break;
}
}
}
if (desiredVideoFormat != null) {
// If we found a match, stop searching formats
break;
}
}
if (desiredVideoFormat != null) {
// It's entirely possible (but not likely) that we got here without a format
// selected, so this null check is unfortunately necessary.
control.setFormat(desiredVideoFormat);
}
}
control.setEnabled(true);
System.out.println("Enabled track: " + control + " (" + control.getFormat() + ")");
}
}
// To get the output from a processor, we need it to be realized.
processor.realize();
waitForState(processor, Processor.Realized);
// Get the data output so we can output it to a file.
DataSource dataOutput = processor.getDataOutput();
// Create a file to receive the media
File answerFile = new File("recording.mov");
MediaLocator dest = new MediaLocator(answerFile.toURI().toURL());
// Create a data sink to write to the disk
DataSink answerSink = Manager.createDataSink(dataOutput, dest);
// Start the processor spinning
processor.start();
// Open the file
answerSink.open();
// Start writing data
answerSink.start();
// SUCCESS! We are now recording
Thread.sleep(10000); // Wait for 10 seconds so we record 10 seconds of video
try {
// Stop the processor. This will also stop and close the datasources
processor.stop();
processor.close();
try {
// Let the buffer run dry. Event Listeners never seem to get called,
// so this seems to be the most effective way.
Thread.sleep(1000);
} catch (InterruptedException ex) {
Logger.getLogger(getClass().getName()).log(Level.SEVERE, null, ex);
}
try {
// Stop recording to the file.
answerSink.stop();
} catch (IOException ex) {
Logger.getLogger(getClass().getName()).log(Level.SEVERE, null, ex);
}
} finally {
try {
// Whatever else we do, close the file if we can to avoid leaking.
answerSink.close();
} catch (Exception ex) {
Logger.getLogger(getClass().getName()).log(Level.SEVERE, null, ex);
}
try {
// Deallocate the native processor resources.
processor.deallocate();
} catch (Exception ex) {
Logger.getLogger(getClass().getName()).log(Level.SEVERE, null, ex);
}
}
}
// My little utility function to wait for a given state.
private void waitForState(Player player, int state) {
// Fast abort
if (player.getState() == state) {
return;
}
long startTime = new Date().getTime();
long timeout = 10 * 1000;
final Object waitListener = new Object();
ControllerListener cl = new ControllerListener() {
#Override
public void controllerUpdate(ControllerEvent ce) {
synchronized (waitListener) {
waitListener.notifyAll();
}
}
};
try {
player.addControllerListener(cl);
// Make sure we wake up every 500ms to check for timeouts and in case we miss a signal
synchronized (waitListener) {
while (player.getState() != state && new Date().getTime() - startTime < timeout) {
try {
waitListener.wait(500);
} catch (InterruptedException ex) {
Logger.getLogger(getClass().getName()).log(Level.SEVERE, null, ex);
}
}
}
} finally {
// No matter what else happens, we want to remove this
player.removeControllerListener(cl);
}
}