Let's say I have a number of sounds (imagine piano/guitar notes).
I want to play each sound after a given interval, for example 200 miliseconds.
But I want to let the previous sound "ring out".
Although the approach below works ok for longer delays (700-1000 ms) it's not too precise.
For short delays, sometimes the sounds "bunch up" and play in rapid succession.
What I tried (sort of pseudo code):
for (Clip clip: clipList){
clip.start();
Thread.sleep(500);
}
My guess this has something to do with thread scheduling by JVM/OS...
Any ideas?
EDIT:
As advised in the comments I tried with a timer too:
final Timer timer = new Timer();
TimerTask timerTask = new TimerTask() {
#Override
public void run() {
try{
Clip clip = clipList.remove();
clip.start();
}catch (NoSuchElementException e) {
timer.cancel();
}
}
};
timer.schedule(timerTask, 0, delay);
I still get the same behaviour when I have 7-8 sounds and 500 ms delay.
Having the notes "ring out" is a great thing, and something that has driven my researches and trials with Java Sound as well. I'm not sure I can help specifically, but I can share an experience or two.
I made a clip player that allows one to retrigger the clip before it has finished playing (allowing the first to play out while another is started), and at different sample rates. This is done by giving the clip multiple cursors, and having them increment through the internally-stored sample data independently. (The playbacks that are at higher or lower speeds use linear interpolation to derive audio values when the cursor lands in between two samples.) The output is funneled into a single SourceDataLine.
Curiously, when this clip player is invoked multiple times, as a stand-alone program, there is a bunching up that occurs that is similar to what you describe. However, I also wrote an audio mixer that is capable of playing back both a number of these clips AND a sourceDataLine .wav file AND some live FM synth sounds, mixing them all into a SINGLE SourceDataLine output, and the timing on this is pretty darn good!
It really baffles me, as the clip part (and the triggers to the clips) are virtually the same code. One key difference might be that the AudioMixer I wrote is set to run continuously, so some of possible timing problems might be coming from when code is run from bytecode versus from memory. The HotSpot compiler will run code a few times as bytecode before committing to placing it in memory where it will run more quickly, and this would account for some timing problems.
I found a good article on timing issues you might want to check out:
http://quod.lib.umich.edu/cgi/p/pod/dod-idx?c=icmc;idno=bbp2372.2007.131
Basically, java doesn't offer "real time" guarantees, and this is the big challenge to achieving low latency performance. Sources of variance include: the issue of when HotSpot or whatever decides to run from bytecode or from memory, garbage collection, vm thread switching.
Having clips that "play out" would be a key component to doing branching music, allowing one to zig instead of zag at a moment in a game, for example, and the music or sfx would remain "seamless." That is part of what I'm envisioning. Of course, it would also be nice if there was a DAW where one could take a track and designate it be saved as multiple audio tiles (that overlap/playout), rather than making one export each tile individually. But that is getting ahead of the game...Was this a general direction you are also thinking about? Or do you have another application in mind? (Just curious.)
Related
How can you ensure the transition between two pieces of audio is seamless?
In a JavaFX application, I am using the javafx.scene.media.MediaPlayer to play an intro-piece, which is proceeded by the main/looping-piece. The media is played just fine, but the issue is the transition and loop.
Here is what I am currently doing:
private static void foo(final Media intro, final Media loop) {
final MediaPlayer introPlayer = new MediaPlayer(intro);
introPlayer.play();
final MediaPlayer loopPlayer = new MediaPlayer(loop);
loopPlayer.pause(); // An attempt to load the media so it will be ready to be played.
introPlayer.setOnEndOfMedia(loopPlayer::play());
loopPlayer.setOnEndOfMedia(() -> loopPlayer.seek(Duration.ZERO));
//loopPlayer.setCycleCount(Integer.MAX_VALUE); // Similar to the above line, but there is still a delay between loops.
}
The MediaPlayer::pause does help some, but there is a very noticeable delay between the end of the intro media and the start of the looping media. Furthermore, there is another noticeable delay between the end of the looping media and the repeat.
I additionally tried using javafx.scene.media.AudioClip, since it supposedly has less overhead than a javafx.scene.media.MediaPlayer. I wrote my own listener to tell when the track ended (and to immediately thereafter start the looping piece), but I still saw a similar delay.
Here were some similar posts I found, but did not provide a solution to the problem:
JavaFX MediaPlayer playing background music loop with small intro music
This one is definitely relevant (coincidentally, it is almost the
anniversary of that post), but I am already using a .wav formatted
media file and still experience a delay.
JavaFX AudioClip.play()
Which is similar to what I tried with the Audioclip, exept I used a
scheduled executor to time when to replay the audio. (Where I still
experienced a delay).
And, as a final note, I have tested my audio in Audacity, where they transitioned and looped seamlessly.
What are some recommended solutions for these types of problems?
Edit:
Added an addendum to the code-block, mentioning MediaPlayer::setCycleCount(Integer)
I realize it's been a while since you posted. Have you found an answer? I am wondering if you loaded loopPlayer before playing introPlayer, if that would help.
If MediaPlayer's "listener" is just kind of sluggish, maybe switching to using Java's SourceDataLine with a LineListener used to trigger the looping cue would work more seamlessly? (I would use a Clip for the looping playback.)
Last suggestion, I have an audio library AudioCue that could work for this. The library includes an AudioCueListener that can trigger an event (such as starting another AudioCue playing) when a cue ends. But limitations of the library would require that you hold your music in memory, and that the source files be .wav's.
The AudioClip Javadocs state that an AudioClip represents a segment of audio that can be played with minimal latency and are usable immediately. However, it also states
Media objects are however better suited for long-playing sounds. This is primarily because AudioClip stores in memory the raw, uncompressed audio data for the entire sound, which can be quite large for long audio clips. A MediaPlayer will only have enough decompressed audio data pre-rolled in memory to play for a short amount of time so it is much more memory efficient for long clips, especially if they are compressed.
Depending on the length of the looping media, an AudioClip might be better suited for you. Instead of needing a ScheduledExecutorService to replay the audio, you can use AudioClip.setCycleCount(AudioClip.INDEFINITE) to loop forever.
Using this info, I believe your best bet is to use a MediaPlayer for the intro and then utilize MediaPlayer#setOnEndOfMedia to call the looping AudioClip; possibly having a small delay between the intro and looping transition but seamless after that.
I'm looking to write a simple MIDI-driven audio sequencer using Javasound.
I have multiple samples (one for each MIDI pitch) that are loaded into memory as a (globally accessible) Map<MidiPitch,AudioInputStream>.
A custom subclass of javax.sound.midi.Receiver responds to incoming MIDI events as follows:
If the event is a note-on, a Clip is obtained and played as follows:
Clip clip = AudioSystem.getClip();
clip.open(lookupAIS(pitch));
clip.start();
The clip is then added to a globally accessible Map<MidiPitch,List<Clip>>, representing started clips, i.e. clips for which start() has been called as above, but for which a note-off event has not yet been received.
If the event is a note-off, the corresponding list of started clips is obtained from the above map. The clip at the head of the list is removed, and stop() and close() are called on it.
The above Receiveris connected to MidiSystem.getSequencer() in the usual way, then the following called:
sequencer.setSequence(MidiSystem.getSequence(new File(myMidFile)))
sequencer.open()
sequencer.start()
Thread.sleep(aLongTime())
sequencer.stop()
sequencer.close()
The above works when the driving MIDI sequence is at a slow tempo, but at higher tempos, notes simply hang (even for sequences containing a very small number of notes).
My understanding is that clip.start() is run within a separate thread behind the scenes by the Javasound API.
Can anyone suggest why this might be happening? Is it perhaps a synchronization issue?
EDIT: By 'hang', I mean that some notes are stuck, despite the fact that log output reports that the corresponding 'stop' method has been called.
EDIT2: It looks as if the hanging first happens when a given note is played for the second time. This happens even if the MIDI sequence is monophonic, i.e. the previous note has stopped.
Your method of loading the Clip for each play is going to be a considerable source of variable latency. Every time you call this, the file is read anew and will not start playing until it the entire file has finished loading.
I recommend pre-loading all the clips and holding them in memory. When the note-on is called, set the clip cursor to zero and then play:
clip[mapIndex].setFramePosition(0);
clip[mapIndex].start();
These clips should have already been opened. I'm putting them in an array and using "mapIndex" as a plausible way of selecting the correct clip that might work with the mapping you've already set up.
You probably won't need to "stop" or "close" the clips until the entire sequence has finished, unless the clips are rather long and are designed to be stopped while in progress, or if they are being played as loops.
This should improve things considerably. I can't say if it will fix everything. The cpu is probably doing some thread multiplexing, and it is plausible that occasionally, in your current code, the clip.close is being called on one thread before the clip has finished loading on the other.
as an excercise I'm trying to create a metronome with Java using Thread.sleep as a timer and JMF for sounds. It's working out quite well but for some reason JMF seems to be only playing sounds at a maximum of 207 beats per minute.
From my Metronome-class:
public void play() {
soundPlayer.play();
waitPulse();
play();
}
From my SoundPlayer-class:
public void play() {
new Thread(new ThreadPlayer()).start();
}
private class ThreadPlayer implements Runnable {
public void run() {
System.out.println("Click");
player.setMediaTime(new Time(0));
player.start();
}
}
I've made SoundPlayer.play() work as a thread to test if it would make a difference, but it's not. I can easily change tempos up to about 207bpm but even if I set my timer to 1000bpm the sounds aren't played any faster than about 207bpm.
I've put the System.out.println("Click"); inside my ThreadPlayer.run() to check if my loop is working correctly – it is.
The issue seems to be with my implementation of JMF. I'm pretty sure there is a simple solution, can anybody help me out?
Thank you so much for your help! :)
The answer about Thread.sleep() being unreliable is correct: you can't count on it to return in exactly the amount of time you specify. In fact, I'm rather surprised your metronome is usable at all, especially when your system is under load. Read the docs for Thread.sleep() for more details. Max Beikirch's answer about MIDI is a good suggestion: MIDI handles timing very well.
But you ask how to do this with audio. The trick is to open an audio stream and fill it with silence between the metronome clicks and insert the metronome clicks where you want. When you do that, your soundcard plays back the samples (whether they contain a click or silence) at a constant rate. The key here is to keep the audio stream open and never close it. The clock, then, is the audio hardware, not your system clock -- a subtle but important distinction.
So, let's say you are generating 16 bit mono samples at 44100 Hz. Here is a function that will create a click sound at the requested rate. Keep in mind that this click sound is bad for speakers (and your ears) so if you actually use it, play it at a low volume. (Also, this code is untested -- it's just to demonstrate the concept)
int interval = 44100; // 1 beat per second, by default
int count = 0;
void setBPM( float bpm ) {
interval = ( bpm / 60 ) * 44100 ;
}
void generateMetronomeSamples( short[] s ) {
for( int i=0; i<s.length; ++i ) {
s = 0;
++count;
if( count == 0 ) {
s = Short.MAX_VALUE;
}
if( count == interval ) {
count = 0;
}
}
}
Once you set the tempo with setBPM, you can play back the samples generated by calling the the generateMetronomeSamples() function repeatedly, and streaming that output to your speakers using JavaSound. (see JSResources.org for a good tutorial)
Once you have that working, you could replace the harsh click with a sound you get from a WAV or AIFF or a short tone or whatever.
Take your time and take a look at MIDI! - http://www.ibm.com/developerworks/library/it/it-0801art38/ or http://docs.oracle.com/javase/tutorial/sound/TOC.html . It's the best solution for everything related to computer made sound.
My assumption would be, and maybe someone else can jump in here, is that thread running times are up to the whims of the thread scheduler. You can't guaranty how long it will take for the JVM to get back to that thread. Also, seeing as the JVM is running as a process on the machine, and is subject to the OS's process scheduler, you are looking at at least two levels of unpredictability.
Just like Jamie Duby said, just because you tell the Thread to sleep 1 millisecond, doesn't mean that it will get called back at exactly one millisecond. The ONLY guarantee is that AT LEAST one millisecond has passed since you called Thread.sleep();. In reality, the processor cannot process the code fast enough to play a beep sound every millisecond, which is why you see the delay. If you want a dramatic example, make a home-made timer class and try making it count one millisecond for a full minute, you will see that the timer is off by quite a bit.
The person who really deserves the answer credit here is Max Beikrich, Midi is the only way you are going to be able to produce the output you are looking for.
I have far more experience as a musician than a programmer but I just finished a metronome application I started a while back, I had put the project on hold for a while because I couldn't figure out why I was having the same problem you are. Yes Thread.sleep() can be unreliable but I'm managed to make a good metronome using that method.
I saw you mentioned trying an ExecutorService I don't think using concurrent classes will solve your problem. My guess is its a system resource issue, I'm pretty certain MIDIs are the way to go with a metronome. I force my students to practice with a metronome and I've used many, I haven't ever been too concerned with the audio quality of the ticks, timing is far more important and MIDIs are going to be much faster than any other audio file. I used the javax.sound.midi library from the Sound API. I suspect that will solve your problem.
You may notice your ticks are uneven when it works properly, this is due to the Thread.sleep() method not being very reliable. How I solved that problem was by doing all my calculations in nanoseconds using the System.nanoTime() method instead of the System.currentTimeMillis() method, just don't forget to convert back to milliseconds before passing the sleep time into the Thread.sleep() method.
I don't want to post the code for my metronome here in case you want to figure it out on your own but if you'd like to see it just send me an e-mail kevin.bigler3#gmail.com and I'd be happy to send it to you. Good luck.
I am seeing some strange behaviour with Clip instances in Java.
The purpose of the class I am working on is to keep count of the number of Clip instances containing the same sound sample (indexed by URI.) When the application requests to play a clip and there are already three or more clips from the same source already playing, the following steps are performed:
Sort the currently-playing clips by a weighted sum of PAN and framePosition.
Select the clip with the highest value as the one to be stopped and re-started.
Re-start the clip (the following method):
void restart(Clip clip, float gain, float pan) {
clip.stop();
clip.flush();
pan = Math.max(-1f, Math.min(pan, 1f));
((FloatControl) clip.getControl(FloatControl.Type.MASTER_GAIN))
.setValue(gain);
((FloatControl) clip.getControl(FloatControl.Type.PAN))
.setValue(pan);
clip.setFramePosition(0);
clip.start();
}
Strange behaviour occurs if this method is called many times in quick succession (e.g. 20 times within 1ms):
The clip plays
The clip fires a START event to signal that it has started playing
The clip never fires a STOP event.
Subsequent calls to stop and start have no effect (but do not throw exceptions.)
getFramePosition always returns 0, even when the clip is audible (for the last time.)
Any idea what could be causing this?
I don't think it's a threading issue (at least not in my code.) Only one thread is calling the public methods of my class (and they are all synchronized anyway)
Might be related to this bug.
The calls to DataLine.start and DataLine.stop are already synchronized on the DataLine's mixer inside AbstractDataLine.
I strongly suspect that somewehere down the call stack (below implStart()/implStop() of whatever DataLine incarnation you got, very likely inside the native nStart/nstop) at least one asynchronous call is made, thus resulting in the race condition you observe.
It would be impossible for you to work around this kind of problem using synchronized or any other Java construct without more intimate knowledge of the native implementation that is called.
A viable, immediate workaround could be to close the old clip and open a new instance instead of rewinding the old one. Not optimal, but it may well do the trick pending a deeper investigation.
In order to be able to perform the aforementioned deeper investigation, one would have to know what platform you are on, as well as a confirmation of the actual (implementation) class names of your Clip and Mixer instances.
UPDATE
In parallel, please use introspection to set com.sun.media.sound.Printer.trace = true (or provide your own implementation of com.sun.media.sound.Printer in the CLASSPATH.)
Essentially DirectClip.open() spawns a thread which accesses several volatile variables (of particular interest being doIO) in a non-threadsafe manner, which could potentially result in the main playback loop hanging.
You can confirm (or infirm) this (in conjunction with the Printer traces) by forcing a thread dump at the time of the apparent hang, and inspecting the playback thread state/stack trace (or use a debugger.)
If doIO etc. access turns out not to be the problem then continuing to dig at the native implementations is still the thing to do; if doIO etc. access does turn out to be the problem then again, there is no easy fix (you can try to use introspection to grab DirectClip.thread and signal it periodically in case it stalls because of doIO -- again, to be confirmed.)
I have a problem playing a sound correctly in Java using the Clip interface.
Playing the sound works using:
clip = (Clip)mixer.getLine(dataLineInfo);
clip.open(audioFormat, byteData, 0, byteData.length);
clip.start();
However there is a memory leak if clips aren't closed.
I have tried adding a line listener before starting the clip, and use the following code:
public void update(LineEvent e) {
if (e.getType() == LineEvent.Type.STOP) {
e.getLine().close();
However, this causes the sound to degrage.
Adding a 1 second sleep in the method makes things work again on my machine - but I would prefer a more elegant solution - I don't think waiting in a listener method is good practice and other machines may take longer.
It is bizarre that the stop event is sent some time arbitrary time prior to the sound stopping.
Does anyone have any ideas on a better way to solve this?
(Related to this, this and this but none are solutions for me)
This appears to be the symptoms described in bug 4434125.
The suggested solution in that bug comments is to call clip.drain() (in an appropriate thread, because it is a blocking call), and then closing the clip when drain completes (since it will block until the data line's internal buffer is emptied (as per Javadocs for the drain method).