We Keep Coding

MediaRecorder record audio in a loop

I'm developing a sound recognition system. I'm using a tensorflow model developed on python to convert MFCC values to labels. I'm using the MediaRecorder class to record the audio, and I'm doing it in a loop so I can be constantly getting microphone audio and then getting the label from the model. Here is the recording loop:
temp = 0;
while (true) {
audioPath = getApplicationContext().getFilesDir().getAbsolutePath();
audioPath += "/Recording" + temp + ".3gp";
audioFile = new File(audioPath);
mediaRecorder = new MediaRecorder();
mediaRecorder.setAudioSource(MediaRecorder.AudioSource.MIC);
mediaRecorder.setOutputFormat(MediaRecorder.OutputFormat.THREE_GPP);
mediaRecorder.setAudioEncoder(MediaRecorder.AudioEncoder.AMR_NB);
mediaRecorder.setOutputFile(audioPath);
try {
mediaRecorder.prepare();
} catch (IOException e) {
e.printStackTrace();
}
mediaRecorder.start();
sleep(2000);
if (!isRunning) {
mediaRecorder.stop();
return;
}
try {
int amplitude = mediaRecorder.getMaxAmplitude();
Log.d("volume", Integer.toString(amplitude));
//finished = false;
avgVolumeTask task = new avgVolumeTask();
task.execute(amplitude);
} catch (Exception e) {
Log.d("Exception in startMediaRecorder()", e.toString());
}
mediaRecorder.stop();
mediaRecorder.release();
soundRecognition task2 = new soundRecognition();
task2.execute();
audioFile.delete();
temp++;
}
This is the soundRecognition method:
private class soundRecognition extends AsyncTask<Integer, Integer, Long> {
#Override
protected Long doInBackground(Integer... level) {
float[] mfccValues = null;
Interpreter tflite = null;
float[][] labelProbArray = null;
try {
mfccValues = computeMFCC();
labelList = loadLabelList();
labelProbArray = new float[1][labelList.size()];
tflite = new Interpreter(loadModel());
} catch (IOException e) {
e.printStackTrace();
} catch (UnsupportedAudioFileException e) {
e.printStackTrace();
}
tflite.run(mfccValues, labelProbArray);
for (int i = 0; i < labelProbArray[0].length; i++) {
float value = labelProbArray[0][i];
//if (i == 1f){
//Log.d("Output at " + Integer.toString(i) + ": ", Float.toString(value));
//doAlert(i);
//}
}
return null;
}
}
The computeMFCC method is this:
public float[] computeMFCC() throws IOException, UnsupportedAudioFileException {
FileInputStream in2 = new FileInputStream(audioPath);
int i;
// InputStream to byte array
byte[] buf = IOUtils.toByteArray(in2);
in2.close();
i = Integer.MAX_VALUE;
// byte array to short array
short[] shortArr = new short[buf.length / 2];
ByteBuffer.wrap(buf).order(ByteOrder.LITTLE_ENDIAN).asShortBuffer().get(shortArr);
int count = 0;
while (count <= shortArr.length) { // Still have data to process.
for (int n = 0; n < nSubframePerBuf; n++) { // Process audio signal in ArrayList and shift by one subframe each time
int k = 0;
for (i = (n * frameShift); i < (n + 1) * frameShift; i++) {
subx[k] = shortArr[i];
k++;
}
subframeList.add(subx); // Add the current subframe to the subframe list. Later, a number of
}
count++;
}
// Need at least nSubframePerMfccFrame to get one analysis frame
x = extractOneFrameFromList(nSubframePerMfccFrame);
MFCC mfcc = new MFCC(samplePerFrm, 16000, numMfcc);
double[] mfccVals = mfcc.doMFCC(x);
float[] floatArray = new float[mfccVals.length];
for (i = 0 ; i < mfccVals.length; i++)
{
floatArray[i] = (float) mfccVals[i];
}
return floatArray;
}
And the doMFCC method is from a downloaded java file here:
https://github.com/enmwmak/ScreamDetector/blob/master/src/edu/polyu/mfcc/MFCC.java
The issue I'm having is that after a few iterations, I run into the problem that the file doesnt get created, and then get a null error passing the results from the input stream to the tensorflow model.
Possible Issues
One reason could be where the file is stored. I've been trying to send the file to local storage because I was worried that all the devices wouldnt have external storage.
Another reason could be that i'm not calling the sound recognition in the right spot. I waited will after the mediaRecorder is stopped to make sure that the file is written with the mic audio, but when I review the contents of the fileInputStream, it appears to not be working, and in each loop the file is always the same.
Any help would be much appreciated.

It may be tricky to have a sleep(2000) inside while loop.
It may be better to check millis and break until 2000 ms has lapsed.

Sound class sounds layered and screechy on Windows

So, when I'm on Mac, this error did not occur. However, when I am on Windows, any sounds I play multiple times over each other start sounding like they are becoming screechy and layering over each other in an unpleasant way.
Here is relevant code from my Sound class:
public class NewerSound {
private boolean stop = true;
private boolean loopable;
private boolean isUrl;
private URL fileUrl;
private Thread sound;
private double volume = 1.0;
public NewerSound(URL url, boolean loopable) throws UnsupportedAudioFileException, IOException {
isUrl = true;
fileUrl = url;
this.loopable = loopable;
}
public void play() {
stop = false;
Runnable r = new Runnable() {
#Override
public void run() {
do {
try {
AudioInputStream in;
if(!isUrl)
in = getAudioInputStream(new File(fileName));
else
in = getAudioInputStream(fileUrl);
final AudioFormat outFormat = getOutFormat(in.getFormat());
final Info info = new Info(SourceDataLine.class, outFormat);
try(final SourceDataLine line = (SourceDataLine) AudioSystem.getLine(info)) {
if(line != null) {
line.open(outFormat);
line.start();
AudioInputStream inputMystream = AudioSystem.getAudioInputStream(outFormat, in);
stream(inputMystream, line);
line.drain();
line.stop();
}
}
}
catch(UnsupportedAudioFileException | LineUnavailableException | IOException e) {
throw new IllegalStateException(e);
}
} while(loopable && !stop);
}
};
sound = new Thread(r);
sound.start();
}
private AudioFormat getOutFormat(AudioFormat inFormat) {
final int ch = inFormat.getChannels();
final float rate = inFormat.getSampleRate();
return new AudioFormat(PCM_SIGNED, rate, 16, ch, ch * 2, rate, false);
}
private void stream(AudioInputStream in, SourceDataLine line) throws IOException {
byte[] buffer = new byte[4];
for(int n = 0; n != -1 && !stop; n = in.read(buffer, 0, buffer.length)) {
byte[] bufferTemp = new byte[buffer.length];
for(int i = 0; i < bufferTemp.length; i += 2) {
short audioSample = (short) ((short) ((buffer[i + 1] & 0xff) << 8) | (buffer[i] & 0xff));
audioSample = (short) (audioSample * volume);
bufferTemp[i] = (byte) audioSample;
bufferTemp[i + 1] = (byte) (audioSample >> 8);
}
buffer = bufferTemp;
line.write(buffer, 0, n);
}
}
}
It is possible that it could be an issue of accessing the same resources when playing the same sound multiple times over itself when I use the NewerSound.play() method.
Please let me know if any other details are needed. Much appreciated :)

The method you are using to change the volume in the method "stream" is flawed. you have 16-bit encoding, thus it takes two bytes to derive a single audio value. You need to assemble the value from the two byte pairs before the multiplication, then take apart the 16-bit result back into two bytes. There are a number of StackOverflow threads with code to do this.
I don't know if this is the whole reason for the problem you describe but it definitely could be, and definitely needs to be fixed.

Graphing wav file in java

I have been searching for this but none seems to answer my question.
I have been trying to graph/plot a wav file by this:
int result = 0;
try {
result = audioInputStream.read(bytes);
} catch (Exception e) {
e.printStackTrace();
}
and then using the result to be a variable for a graph. I've been thinking if it is correct to change first the result to decibels. Also, am I right to use the result as a variable to be use in the graph? Or is there any way that has to be use in graphing the wav file?

The first thing you need to do is read the samples of the file, this will give you the min/max ranges of the waveform (sound wave)...
File file = new File("...");
AudioInputStream ais = null;
try {
ais = AudioSystem.getAudioInputStream(file);
int frameLength = (int) ais.getFrameLength();
int frameSize = (int) ais.getFormat().getFrameSize();
byte[] eightBitByteArray = new byte[frameLength * frameSize];
int result = ais.read(eightBitByteArray);
int channels = ais.getFormat().getChannels();
int[][] samples = new int[channels][frameLength];
int sampleIndex = 0;
for (int t = 0; t < eightBitByteArray.length;) {
for (int channel = 0; channel < channels; channel++) {
int low = (int) eightBitByteArray[t];
t++;
int high = (int) eightBitByteArray[t];
t++;
int sample = getSixteenBitSample(high, low);
samples[channel][sampleIndex] = sample;
}
sampleIndex++;
}
} catch (Exception exp) {
exp.printStackTrace();
} finally {
try {
ais.close();
} catch (Exception e) {
}
}
//...
protected int getSixteenBitSample(int high, int low) {
return (high << 8) + (low & 0x00ff);
}
Then you would need to determine the min/max values, the next example simply checks for channel 0, but you could use the same concept to check all the available channels...
int min = 0;
int max = 0;
for (int sample : samples[0]) {
max = Math.max(max, sample);
min = Math.min(min, sample);
}
FYI: It would be more efficient to populate this information when you read the file
Once you have this, you can model the samples...but that would depend on framework you intend to use...

Split Wave audio file at silence [duplicate]

How can I detect silence when recording operation is started in Java? What is PCM data? How can I calculate PCM data in Java?
I found the solution :
package bemukan.voiceRecognition.speechToText;
import javax.sound.sampled.*;
import java.io.*;
public class RecordAudio {
private File audioFile;
protected boolean running;
private ByteArrayOutputStream out;
private AudioInputStream inputStream;
final static float MAX_8_BITS_SIGNED = Byte.MAX_VALUE;
final static float MAX_8_BITS_UNSIGNED = 0xff;
final static float MAX_16_BITS_SIGNED = Short.MAX_VALUE;
final static float MAX_16_BITS_UNSIGNED = 0xffff;
private AudioFormat format;
private float level;
private int frameSize;
public RecordAudio(){
getFormat();
}
private AudioFormat getFormat() {
File file = new File("src/Facebook/1.wav");
AudioInputStream stream;
try {
stream = AudioSystem.getAudioInputStream(file);
format=stream.getFormat();
frameSize=stream.getFormat().getFrameSize();
return stream.getFormat();
} catch (UnsupportedAudioFileException e) {
} catch (IOException e) {
}
return null;
}
public void stopAudio() {
running = false;
}
public void recordAudio() {
try {
final AudioFormat format = getFormat();
DataLine.Info info = new DataLine.Info(
TargetDataLine.class, format);
final TargetDataLine line = (TargetDataLine)
AudioSystem.getLine(info);
line.open(format);
line.start();
Runnable runner = new Runnable() {
int bufferSize = (int) format.getSampleRate()
* format.getFrameSize();
byte buffer[] = new byte[bufferSize];
public void run() {
int readPoint = 0;
out = new ByteArrayOutputStream();
running = true;
int sum=0;
while (running) {
int count =
line.read(buffer, 0, buffer.length);
calculateLevel(buffer,0,0);
System.out.println(level);
if (count > 0) {
out.write(buffer, 0, count);
}
}
line.stop();
}
};
Thread captureThread = new Thread(runner);
captureThread.start();
} catch (LineUnavailableException e) {
System.err.println("Line unavailable: " + e);
System.exit(-2);
}
}
public File getAudioFile() {
byte[] audio = out.toByteArray();
InputStream input = new ByteArrayInputStream(audio);
try {
final AudioFormat format = getFormat();
final AudioInputStream ais =
new AudioInputStream(input, format,
audio.length / format.getFrameSize());
AudioSystem.write(ais, AudioFileFormat.Type.WAVE, new File("temp.wav"));
input.close();
System.out.println("New file created!");
} catch (IOException e) {
System.out.println(e.getMessage());
}
return new File("temp.wav");
}
private void calculateLevel (byte[] buffer,
int readPoint,
int leftOver) {
int max = 0;
boolean use16Bit = (format.getSampleSizeInBits() == 16);
boolean signed = (format.getEncoding() ==
AudioFormat.Encoding.PCM_SIGNED);
boolean bigEndian = (format.isBigEndian());
if (use16Bit) {
for (int i=readPoint; i<buffer.length-leftOver; i+=2) {
int value = 0;
// deal with endianness
int hiByte = (bigEndian ? buffer[i] : buffer[i+1]);
int loByte = (bigEndian ? buffer[i+1] : buffer [i]);
if (signed) {
short shortVal = (short) hiByte;
shortVal = (short) ((shortVal << 8) | (byte) loByte);
value = shortVal;
} else {
value = (hiByte << 8) | loByte;
}
max = Math.max(max, value);
} // for
} else {
// 8 bit - no endianness issues, just sign
for (int i=readPoint; i<buffer.length-leftOver; i++) {
int value = 0;
if (signed) {
value = buffer [i];
} else {
short shortVal = 0;
shortVal = (short) (shortVal | buffer [i]);
value = shortVal;
}
max = Math.max (max, value);
} // for
} // 8 bit
// express max as float of 0.0 to 1.0 of max value
// of 8 or 16 bits (signed or unsigned)
if (signed) {
if (use16Bit) { level = (float) max / MAX_16_BITS_SIGNED; }
else { level = (float) max / MAX_8_BITS_SIGNED; }
} else {
if (use16Bit) { level = (float) max / MAX_16_BITS_UNSIGNED; }
else { level = (float) max / MAX_8_BITS_UNSIGNED; }
}
} // calculateLevel
}

How can I detect silence when recording operation is started in Java?
Calculate the dB or RMS value for a group of sound frames and decide at what level it is considered to be 'silence'.
What is PCM data?
Data that is in Pulse-code modulation format.
How can I calculate PCM data in Java?
I do not understand that question. But guessing it has something to do with the speech-recognition tag, I have some bad news. This might theoretically be done using the Java Speech API. But there are apparently no 'speech to text' implementations available for the API (only 'text to speech').
I have to calculate rms for speech-recognition project. But I do not know how can I calculate in Java.
For a single channel that is represented by signal sizes in a double ranging from -1 to 1, you might use this method.
/** Computes the RMS volume of a group of signal sizes ranging from -1 to 1. */
public double volumeRMS(double[] raw) {
double sum = 0d;
if (raw.length==0) {
return sum;
} else {
for (int ii=0; ii<raw.length; ii++) {
sum += raw[ii];
}
}
double average = sum/raw.length;
double sumMeanSquare = 0d;
for (int ii=0; ii<raw.length; ii++) {
sumMeanSquare += Math.pow(raw[ii]-average,2d);
}
double averageMeanSquare = sumMeanSquare/raw.length;
double rootMeanSquare = Math.sqrt(averageMeanSquare);
return rootMeanSquare;
}
There is a byte buffer to save input values from the line, and what I should have to do with this buffer?
If using the volumeRMS(double[]) method, convert the byte values to an array of double values ranging from -1 to 1. ;)

You need to catch the value like a number silence is zero or near
Please adapt your code to your requirement!!!
In this case a variable named UMBRAL (Threshold in spanish)...
Suppose that you have access to WAV file like bytes ByteHeader...
private Integer Byte2PosIntBig(byte Byte24, byte Byte16, byte Byte08, byte Byte00) {
return new Integer (
((Byte24) << 24)|
((Byte16 & 0xFF) << 16)|
((Byte08 & 0xFF) << 8)|
((Byte00 & 0xFF) << 0));
}
Before ....
RandomAccessFile RAFSource = new RandomAccessFile("your old file wav", "r");
Begins here...
int PSData = 44;
byte[] Bytes = new byte[4];
byte[] ByteHeader = new byte[44];
RAFSource.seek(0);
RAFSource.read(ByteHeader);
int WavSize = Byte2PosIntBig(ByteHeader[43],ByteHeader[42],ByteHeader[41],ByteHeader[40]);
int NumBits = Byte2PosIntBig(ByteHeader[35],ByteHeader[34]);
int NumByte = NumBits/8;
for (int i = PSData;i < PSData+WavSize;i+=NumByte) {
int WavSample = 0;
int WavResultI =0;
int WavResultO = 0;
if (NumByte == 2) {
RAFSource.seek(i);
Bytes[0] = RAFSource.readByte();
Bytes[1] = RAFSource.readByte();
WavSample = (int)(((Bytes[1]) << 8)|((Bytes[0] & 0xFF) << 0));
if (Math.abs(WavSample) < UMBRAL) {
//SILENCE DETECTED!!!
}
} else {
RAFSource.seek(i);
WavSample = (short)(RAFSource.readByte() & 0xFF);
short sSamT = (short)WavSample;
sSamT += 128;
double dSamD = (double)sSamT*Multiplier;
if ((double)sSamT < UMBRAL) {
//SILENCE DETECTED!!!
}
}

Audio Mixing with Java (without Mixer API)

I am attempting to mix several different audio streams and trying to get them to play at the same time instead of one-at-a-time.
The code below plays them one-at-a-time and I cannot figure out a solution that does not use the Java Mixer API. Unfortunately, my audio card does not support synchronization using the Mixer API and I am forced to figure out a way to do it through code.
Please advise.
/////CODE IS BELOW////
class MixerProgram {
public static AudioFormat monoFormat;
private JFileChooser fileChooser = new JFileChooser();
private static File[] files;
private int trackCount;
private FileInputStream[] fileStreams = new FileInputStream[trackCount];
public static AudioInputStream[] audioInputStream;
private Thread trackThread[] = new Thread[trackCount];
private static DataLine.Info sourceDataLineInfo = null;
private static SourceDataLine[] sourceLine;
public MixerProgram(String[] s)
{
trackCount = s.length;
sourceLine = new SourceDataLine[trackCount];
audioInputStream = new AudioInputStream[trackCount];
files = new File[s.length];
}
public static void getFiles(String[] s)
{
files = new File[s.length];
for(int i=0; i<s.length;i++)
{
File f = new File(s[i]);
if (!f.exists())
System.err.println("Wave file not found: " + filename);
files[i] = f;
}
}
public static void loadAudioFiles(String[] s)
{
AudioInputStream in = null;
audioInputStream = new AudioInputStream[s.length];
sourceLine = new SourceDataLine[s.length];
for(int i=0;i<s.length;i++){
try
{
in = AudioSystem.getAudioInputStream(files[i]);
}
catch(Exception e)
{
System.err.println("Failed to assign audioInputStream");
}
monoFormat = in.getFormat();
AudioFormat decodedFormat = new AudioFormat(
AudioFormat.Encoding.PCM_SIGNED,
monoFormat.getSampleRate(), 16, monoFormat.getChannels(),
monoFormat.getChannels() * 2, monoFormat.getSampleRate(),
false);
monoFormat = decodedFormat; //give back name
audioInputStream[i] = AudioSystem.getAudioInputStream(decodedFormat, in);
sourceDataLineInfo = new DataLine.Info(SourceDataLine.class, monoFormat);
try
{
sourceLine[i] = (SourceDataLine) AudioSystem.getLine(sourceDataLineInfo);
sourceLine[i].open(monoFormat);
}
catch(LineUnavailableException e)
{
System.err.println("Failed to get SourceDataLine" + e);
}
}
}
public static void playAudioMix(String[] s)
{
final int tracks = s.length;
System.out.println(tracks);
Runnable playAudioMixRunner = new Runnable()
{
int bufferSize = (int) monoFormat.getSampleRate() * monoFormat.getFrameSize();
byte[] buffer = new byte[bufferSize];
public void run()
{
if(tracks==0)
return;
for(int i = 0; i < tracks; i++)
{
sourceLine[i].start();
}
int bytesRead = 0;
while(bytesRead != -1)
{
for(int i = 0; i < tracks; i++)
{
try
{
bytesRead = audioInputStream[i].read(buffer, 0, buffer.length);
}
catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
if(bytesRead >= 0)
{
int bytesWritten = sourceLine[i].write(buffer, 0, bytesRead);
System.out.println(bytesWritten);
}
}
}
}
};
Thread playThread = new Thread(playAudioMixRunner);
playThread.start();
}
}

The problem is that you are not adding the samples together. If we are looking at 4 tracks, 16-bit PCM data, you need to add all the different values together to "mix" them into one final output. So, from a purely-numbers point-of-view, it would look like this:
[Track1] 320 -16 2000 200 400
[Track2] 16 8 123 -87 91
[Track3] -16 -34 -356 1200 805
[Track4] 1011 1230 -1230 -100 19
[Final!] 1331 1188 537 1213 1315
In your above code, you should only be writing a single byte array. That byte array is the final mix of all tracks added together. The problem is that you are writing a byte array for each different track (so there is no mixdown happening, as you observed).
If you want to guarantee you don't have any "clipping", you should take the average of all tracks (so add all four tracks above and divide by 4). However, there are artifacts from choosing that approach (like if you have silence on three tracks and one loud track, the final output will be much quiter than the volume of the one track that is not silent). There are more complicated algorithms you can use to do the mixing, but by then you are writing your own mixer :P.