We Keep Coding

MediaCodec AV Sync when decoding

All of the questions regarding syncing audio and video, when decoding using MediaCodec, suggests that we should use an "AV Sync" mechanism to sync the video and audio using their timestamps.
Here is what I do to achieve this:
I have 2 threads, one for decoding video and one for audio. To sync the video and audio I'm using Extractor.getSampleTime() to determine if I should release the audio or video buffers, please see below:
//This is called after configuring MediaCodec(both audio and video)
private void startPlaybackThreads(){
//Audio playback thread
mAudioWorkerThread = new Thread("AudioThread") {
#Override
public void run() {
if (!Thread.interrupted()) {
try {
//Check info below
if (shouldPushAudio()) {
workLoopAudio();
}
} catch (Exception e) {
e.printStackTrace();
}
}
}
};
mAudioWorkerThread.start();
//Video playback thread
mVideoWorkerThread = new Thread("VideoThread") {
#Override
public void run() {
if (!Thread.interrupted()) {
try {
//Check info below
if (shouldPushVideo()) {
workLoopVideo();
}
} catch (Exception e) {
e.printStackTrace();
}
}
}
};
mVideoWorkerThread.start();
}
//Check if more buffers should be sent to the audio decoder
private boolean shouldPushAudio(){
int audioTime =(int) mAudioExtractor.getSampleTime();
int videoTime = (int) mExtractor.getSampleTime();
return audioTime <= videoTime;
}
//Check if more buffers should be sent to the video decoder
private boolean shouldPushVideo(){
int audioTime =(int) mAudioExtractor.getSampleTime();
int videoTime = (int) mExtractor.getSampleTime();
return audioTime > videoTime;
}
Inside workLoopAudio() and workLoopVideo() is all my MediaCodec logic (I decided not to post it because it's not relevant).
So what I do is, I get the sample time of the video and the audio tracks, I then check which one is bigger(further ahead). If the video is "ahead" then I pass more buffers to my audio decoder and visa versa.
This seems to be working fine - The video and audio are playing in sync.
My question:
I would like to know if my approach is correct(is this how we should be doing it, or is there another/better way)? I could not find any working examples of this(written in java/kotlin), thus the question.
EDIT 1:
I've found that the audio trails behind the video (very slightly) when I decode/play a video that was encoded using FFmpeg. If I use a video that was not encoded using FFmpeg then the video and audio syncs perfectly.
The FFmpeg command is nothing out of the ordinary:
-i inputPath -crf 18 -c:v libx264 -preset ultrafast OutputPath
I will be providing additional information below:
I initialize/create AudioTrack like this:
//Audio
mAudioExtractor = new MediaExtractor();
mAudioExtractor.setDataSource(mSource);
int audioTrackIndex = selectAudioTrack(mAudioExtractor);
if (audioTrackIndex < 0){
throw new IOException("Can't find Audio info!");
}
mAudioExtractor.selectTrack(audioTrackIndex);
mAudioFormat = mAudioExtractor.getTrackFormat(audioTrackIndex);
mAudioMime = mAudioFormat.getString(MediaFormat.KEY_MIME);
mAudioChannels = mAudioFormat.getInteger(MediaFormat.KEY_CHANNEL_COUNT);
mAudioSampleRate = mAudioFormat.getInteger(MediaFormat.KEY_SAMPLE_RATE);
final int min_buf_size = AudioTrack.getMinBufferSize(mAudioSampleRate, (mAudioChannels == 1 ? AudioFormat.CHANNEL_OUT_MONO : AudioFormat.CHANNEL_OUT_STEREO), AudioFormat.ENCODING_PCM_16BIT);
final int max_input_size = mAudioFormat.getInteger(MediaFormat.KEY_MAX_INPUT_SIZE);
mAudioInputBufSize = min_buf_size > 0 ? min_buf_size * 4 : max_input_size;
if (mAudioInputBufSize > max_input_size) mAudioInputBufSize = max_input_size;
final int frameSizeInBytes = mAudioChannels * 2;
mAudioInputBufSize = (mAudioInputBufSize / frameSizeInBytes) * frameSizeInBytes;
mAudioTrack = new AudioTrack(AudioManager.STREAM_MUSIC,
mAudioSampleRate,
(mAudioChannels == 1 ? AudioFormat.CHANNEL_OUT_MONO : AudioFormat.CHANNEL_OUT_STEREO),
AudioFormat.ENCODING_PCM_16BIT,
AudioTrack.getMinBufferSize(mAudioSampleRate, mAudioChannels == 1 ? AudioFormat.CHANNEL_OUT_MONO : AudioFormat.CHANNEL_OUT_STEREO, AudioFormat.ENCODING_PCM_16BIT),
AudioTrack.MODE_STREAM);
try {
mAudioTrack.play();
} catch (final Exception e) {
Log.e(TAG, "failed to start audio track playing", e);
mAudioTrack.release();
mAudioTrack = null;
}
And I write to the AudioTrack like this:
//Called from within workLoopAudio, when releasing audio buffers
if (bufferAudioIndex >= 0) {
if (mAudioBufferInfo.size > 0) {
internalWriteAudio(mAudioOutputBuffers[bufferAudioIndex], mAudioBufferInfo.size);
}
mAudioDecoder.releaseOutputBuffer(bufferAudioIndex, false);
}
private boolean internalWriteAudio(final ByteBuffer buffer, final int size) {
if (mAudioOutTempBuf.length < size) {
mAudioOutTempBuf = new byte[size];
}
buffer.position(0);
buffer.get(mAudioOutTempBuf, 0, size);
buffer.clear();
if (mAudioTrack != null)
mAudioTrack.write(mAudioOutTempBuf, 0, size);
return true;
}
"NEW" Question:
The audio trails about 200ms behind the video if I use a video that was encoded using FFmpeg, is there a reason why this could be happening?

It seems like it is working now. I use the same logic as above, but now I keep a reference of the presentationTimeUs returned from MediaCodec.BufferInfo() before calling dequeueOutputBuffer to check if I should continue my video or audio work loop:
// Check if audio work loop should continue
private boolean shouldPushAudio(){
long videoTime = mExtractor.getSampleTime();
return tempAudioPresentationTimeUs <= videoTime;
}
// Check if video work loop should continue
private boolean shouldPushVideo(){
long videoTime = mExtractor.getSampleTime();
return tempAudioPresentationTimeUs >= videoTime;
}
// tempAudioPresentationTimeUs is set right before I call dequeueOutputBuffer
// As shown here:
tempAudioPresentationTimeUs = mAudioBufferInfo.presentationTimeUs;
int outIndex = mAudioDecoder.dequeueOutputBuffer(mAudioBufferInfo, timeout);
By doing this, my video and audio is synced perfectly, even with files that was encoded with FFmpeg(as mentioned in my edit above).
I ran into an issue where my video work loop didn't complete, this was caused by the audio reaching EOS before the video and then returning -1. So I changed my original mVideoWorkerThread to the following:
mVideoWorkerThread = new Thread("VideoThread") {
#Override
public void run() {
if (!Thread.interrupted()) {
try {
if (shouldPushVideo() || audioReachedEOS()) {
workLoopVideo();
}
} catch (Exception e) {
e.printStackTrace();
}
}
}
};
mVideoWorkerThread.start();
private boolean audioReachedEOS() {
return mAudioExtractor.getSampleTime() == -1;
}
So I use audioReachedEOS() to check if my audio MediaExtractor returns -1. If it does then it means that my audio is done, but my video is not, so I continue my video work loop until it is done.
This seems to be working as expected (when I only play/pause the video without seeking). I had another issue with seeking, but I will not elaborate on this.
I will release my application as is and update this answer when I run into problems.

IOIO UART readback problems

I'm extending the BaseIOIOLooper to open up a UART device and send messages. I'm testing with a readback, where I send a packet over a line and receive that packet on another line and print it out. Because I don't want the InputStream.read() method to block, I am handling packet formation and input in a different thread. I have narrowed my problem down to the InputStream.read() method, which returns -1 (no bytes read, but no exception).
Here is what it looks like in the Looper thread:
#Override
protected void setup() throws ConnectionLostException, InterruptedException {
log_.write_log_line(log_header_ + "Beginning IOIO setup.");
// Initialize IOIO UART pins
// Input at pin 1, output at pin 2
try {
inQueue_ = MinMaxPriorityQueue.orderedBy(new ComparePackets())
.maximumSize(QUEUESIZE).create();
outQueue_ = MinMaxPriorityQueue.orderedBy(new ComparePackets())
.maximumSize(QUEUESIZE).create();
ioio_.waitForConnect();
uart_ = ioio_.openUart(1, 2, 38400, Uart.Parity.NONE, Uart.StopBits.ONE);
// Start InputHandler. Takes packets from ELKA on inQueue_
in_= new InputHandler(inQueue_, uart_.getInputStream());
in_.start();
// Start OutputHandler. Takes packets from subprocesses on outQueue_
out_= new OutputHandler(outQueue_);
out_.start();
// Get output stream
os_=uart_.getOutputStream();
// Set default target state
setTargetState(State.TRANSFERRING);
currInPacket_[0]=1; //Initial value to start transferring
log_.write_log_line(log_header_ + "IOIO setup complete.\n\t" +
"Input pin set to 1\n\tOutput pin set to 2\n\tBaud rate set to 38400\n\t" +
"Parity set to even\n\tStop bits set to 1");
} catch (IncompatibilityException e) {
log_.write_log_line(log_header_+e.toString());
} catch (ConnectionLostException e) {
log_.write_log_line(log_header_+e.toString());
} catch (Exception e) {
log_.write_log_line(log_header_+"mystery exception: "+e.toString());
}
}
And in the InputHandler thread:
#Override
public void run() {
boolean notRead;
byte i;
log_.write_log_line(log_header_+"Beginning InputHandler thread");
while (!stop) {
i = 0;
notRead = true;
nextInPacket = new byte[BUFFERSIZE];
readBytes = -1;
//StringBuilder s=new StringBuilder();
//TODO re-implement this with signals
while (i < READATTEMPTS && notRead) {
try {
// Make sure to adjust packet size. Done manually here for speed.
readBytes = is_.read(nextInPacket, 0, BUFFERSIZE);
/* Debugging
for (int j=0;j<nextInPacket.length;j++)
s.append(Byte.toString(nextInPacket[j]));
log_.write_log_line(log_header_+s.toString());
*/
if (readBytes != -1) {
notRead = false;
nextInPacket= new byte[]{1,0,1,0,1,0,1,0,1,0,1,0,1,0,1,0};
synchronized (q_) {
q_.add(nextInPacket);
}
//log_.write_log_line(log_header_ + "Incoming packet contains valid data.");
} else i++;
} catch (IOException e) {
log_.write_log_line(log_header_ + "mystery exception:\n\t" + e.toString());
}
}
if (i>=READATTEMPTS)
log_.write_log_line(log_header_+"Too many read attempts from input stream.");
/*
try {
sleep(100);
} catch (InterruptedException e) {
log_.write_log_line(log_header_+"fuck");
}
*/
}
}
On an oscilloscope, pins 1 and 2 both read an oscillating voltage, albeit at a very high amplitude, which is of some concern. Point is nothing is available to be read from the InputStream in the InputHandler class. Any ideas?

-1 returned from read() should only happen whenever the UART is closed. The closure can happen as result of explicitly calling close() on the Uart object or calling softReset() on the IOIO object.
The Android log might give you some clues about what's going on.
The reading you're seeing on the oscilloscope is suspicious: how high is "very high amplitude"? You should only ever see 0V or 3.3V on those pins, or floating in case the pins where not opened (or closed) for some reason.

java playing sounds in order

Hi I have following java programme that play some sounds.I want to play sounds in order for example after ending of sound1 i want to play sound2 and then sound3 the following is my java code and function of playing sound .
private void playsound(String file)
{
try {
crit = AudioSystem.getClip();
AudioInputStream inputStream1 = AudioSystem.getAudioInputStream(this.getClass().getResource(file));
crit.open(inputStream1);
//if(!crit.isOpen())
{
crit.start();
}
} catch (Exception ex) {
System.out.println(ex.getMessage());
}
}
and calling it as following
playsound("/sounds/filesound1.au");
playsound("/sounds/filesound2.au");
playsound("/sounds/filesound3.au");
the programme is plying sound in parallel which I don't want.I want to play in order
Regards

I got the following code from somewhere that I can't remember right now but it plays the music consequently:
public static void play(ArrayList<String> files){
byte[] buffer = new byte[4096];
for (String filePath : files) {
File file = new File(filePath);
try {
AudioInputStream is = AudioSystem.getAudioInputStream(file);
AudioFormat format = is.getFormat();
SourceDataLine line = AudioSystem.getSourceDataLine(format);
line.open(format);
line.start();
while (is.available() > 0) {
int len = is.read(buffer);
line.write(buffer, 0, len);
}
line.drain();
line.close();
} catch (Exception ex) {
ex.printStackTrace();
}
}
}
The reason this plays the files consequently and not all at the same time is because write blocks until the requested amount of data has been written. This applies even if the requested amount of data to write is greater than the data line's buffer size.
Make sure to include drain() from the code above. drain() waits for the buffer to empty before it close()s.

Playing wavs in Java

So, I'm working on a project for class wherein we have to have a game with background music. I'm trying to play a .wav file as background music, but since I can't use clips (too short for a music file) I have to play with the AudioStream.
In my first implementation, the game would hang until the song finished, so I threw it into its own thread to try and alleviate that. Currently, the game plays very slowly while the song plays. I'm not sure what I need to do to make this thread play nice with my animator thread, because we we're never formally taught threads. Below is my background music player class, please someone tell me what I've done wrong that makes it hog all the system resources.
public class BGMusicPlayer implements Runnable {
File file;
AudioInputStream in;
SourceDataLine line;
int frameSize;
byte[] buffer = new byte [32 * 1024];
Thread player;
boolean playing = false;
boolean fileNotOver = true;
public BGMusicPlayer (File inputFile){
try{
file = inputFile;
in = AudioSystem.getAudioInputStream (inputFile);
AudioFormat format = in.getFormat();
frameSize = format.getFrameSize();
DataLine.Info info =new DataLine.Info (SourceDataLine.class, format);
line = (SourceDataLine) AudioSystem.getLine (info);
line.open();
player = new Thread (this);
player.start();
}
catch(Exception e){
System.out.println("That is not a valid file. No music for you.");
}
}
public void run() {
int readPoint = 0;
int bytesRead = 0;
player.setPriority(Thread.MIN_PRIORITY);
while (fileNotOver) {
if (playing) {
try {
bytesRead = in.read (buffer,
readPoint,
buffer.length - readPoint);
} catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
if (bytesRead == -1) {
fileNotOver = false;
break;
}
int leftover = bytesRead % frameSize;
// send to line
line.write (buffer, readPoint, bytesRead-leftover);
// save the leftover bytes
System.arraycopy (buffer, bytesRead,
buffer, 0,
leftover);
readPoint = leftover;
try {
Thread.sleep(20);
} catch (InterruptedException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}
}
}
public void start() {
playing = true;
if(!player.isAlive())
player.start();
line.start();
}
public void stop() {
playing = false;
line.stop();
}
}

You are pretty close, but there are a couple of unusual things that maybe are contributing to the performance problem.
First off, if you are just playing back a .wav, there shouldn't really be a need to deal with any "readpoint" but a value of 0, and there shouldn't really be a need for a "leftover" computation. When you do the write, it should simply be the same number of bytes that were read in (return value of the read() method).
I'm also unclear why you are doing the ArrayCopy. Can you lose that?
Setting the Thread to low priority, and putting a Sleep--I guess you were hoping those would slow down the audio processing to allow more of your game to process? I've never seen this done before and it is really unusual if it is truly needed. I really recommend getting rid of these as well.
I'm curious where your audio file is coming from. Your not streaming it over the web, are you?
By the way, the way you get your input from a File and place it into an InputStream very likely won't work with Java7. A lot of folks are reporting a bug with that. It turns out it is more correct and efficient to generate a URL from the File, and then get the AudioInputStream using the URL as the argument rather than the file. The error that can come up is a "Mark/Reset" error. (A search on that will show its come up a number of times here.)

How do I handle multiple streams in Java?

I'm trying to run a process and do stuff with its input, output and error streams. The obvious way to do this is to use something like select(), but the only thing I can find in Java that does that is Selector.select(), which takes a Channel. It doesn't appear to be possible to get a Channel from an InputStream or OutputStream (FileStream has a getChannel() method but that doesn't help here)
So, instead I wrote some code to poll all the streams:
while( !out_eof || !err_eof )
{
while( out_str.available() )
{
if( (bytes = out_str.read(buf)) != -1 )
{
// Do something with output stream
}
else
out_eof = true;
}
while( err_str.available() )
{
if( (bytes = err_str.read(buf)) != -1 )
{
// Do something with error stream
}
else
err_eof = true;
}
sleep(100);
}
which works, except that it never terminates. When one of the streams reaches end of file, available() returns zero so read() isn't called and we never get the -1 return that would indicate EOF.
One solution would be a non-blocking way to detect EOF. I can't see one in the docs anywhere. Alternatively is there a better way of doing what I want to do?
I see this question here:
link text
and although it doesn't exactly do what I want, I can probably use that idea, of spawning separate threads for each stream, for the particular problem I have now. But surely that isn't the only way to do it? Surely there must be a way to read from multiple streams without using a thread for each?

As you said, the solution outlined in this Answer is the traditional way of reading both stdout and stderr from a Process. A thread-per-stream is the way to go, even though it is slightly annoying.

You will indeed have to go the route of spawning a Thread for each stream you want to monitor. If your use case allows for combining both stdout and stderr of the process in question you need only one thread, otherwise two are needed.
It took me quite some time to get it right in one of our projects where I have to launch an external process, take its output and do something with it while at the same time looking for errors and process termination and also being able to terminate it when the java app's user cancels the operation.
I created a rather simple class to encapsulate the watching part whose run() method looks something like this:
public void run() {
BufferedReader tStreamReader = null;
try {
while (externalCommand == null && !shouldHalt) {
logger.warning("ExtProcMonitor("
+ (watchStdErr ? "err" : "out")
+ ") Sleeping until external command is found");
Thread.sleep(500);
}
if (externalCommand == null) {
return;
}
tStreamReader =
new BufferedReader(new InputStreamReader(watchStdErr ? externalCommand.getErrorStream()
: externalCommand.getInputStream()));
String tLine;
while ((tLine = tStreamReader.readLine()) != null) {
logger.severe(tLine);
if (filter != null) {
if (filter.matches(tLine)) {
informFilterListeners(tLine);
return;
}
}
}
} catch (IOException e) {
logger.logExceptionMessage(e, "IOException stderr");
} catch (InterruptedException e) {
logger.logExceptionMessage(e, "InterruptedException waiting for external process");
} finally {
if (tStreamReader != null) {
try {
tStreamReader.close();
} catch (IOException e) {
// ignore
}
}
}
}
On the calling side it looks like this:
Thread tExtMonitorThread = new Thread(new Runnable() {
public void run() {
try {
while (externalCommand == null) {
getLogger().warning("Monitor: Sleeping until external command is found");
Thread.sleep(500);
if (isStopRequested()) {
getLogger()
.warning("Terminating external process on user request");
if (externalCommand != null) {
externalCommand.destroy();
}
return;
}
}
int tReturnCode = externalCommand.waitFor();
getLogger().warning("External command exited with code " + tReturnCode);
} catch (InterruptedException e) {
getLogger().logExceptionMessage(e, "Interrupted while waiting for external command to exit");
}
}
}, "ExtCommandWaiter");
ExternalProcessOutputHandlerThread tExtErrThread =
new ExternalProcessOutputHandlerThread("ExtCommandStdErr", getLogger(), true);
ExternalProcessOutputHandlerThread tExtOutThread =
new ExternalProcessOutputHandlerThread("ExtCommandStdOut", getLogger(), true);
tExtMonitorThread.start();
tExtOutThread.start();
tExtErrThread.start();
tExtErrThread.setFilter(new FilterFunctor() {
public boolean matches(Object o) {
String tLine = (String)o;
return tLine.indexOf("Error") > -1;
}
});
FilterListener tListener = new FilterListener() {
private boolean abortFlag = false;
public boolean shouldAbort() {
return abortFlag;
}
public void matched(String aLine) {
abortFlag = abortFlag || (aLine.indexOf("Error") > -1);
}
};
tExtErrThread.addFilterListener(tListener);
externalCommand = new ProcessBuilder(aCommand).start();
tExtErrThread.setProcess(externalCommand);
try {
tExtMonitorThread.join();
tExtErrThread.join();
tExtOutThread.join();
} catch (InterruptedException e) {
// when this happens try to bring the external process down
getLogger().severe("Aborted because auf InterruptedException.");
getLogger().severe("Killing external command...");
externalCommand.destroy();
getLogger().severe("External command killed.");
externalCommand = null;
return -42;
}
int tRetVal = tListener.shouldAbort() ? -44 : externalCommand.exitValue();
externalCommand = null;
try {
getLogger().warning("command exit code: " + tRetVal);
} catch (IllegalThreadStateException ex) {
getLogger().warning("command exit code: unknown");
}
return tRetVal;
Unfortunately I don't have to for a self-contained runnable example, but maybe this helps.
If I had to do it again I would have another look at using the Thread.interrupt() method instead of a self-made stop flag (mind to declare it volatile!), but I leave that for another time. :)