We Keep Coding

How to interpret output from FFT on Noise library

I'm trying to get the most representative frequency (or first harmonic) from an audio file using the Noise FFT library (https://github.com/paramsen/noise). I have an array with the values of size x and the output array's size is x+2. I'm not familiar with Fourier Transform, so maybe I'm missing something, but from my understanding I should have something that represents the frequencies and stores the magnitude (or in this case a complex number from with to calculate it) of each one.
The thing is: since each position in the array should be a frequency, how can I know the range of the output frequencies, what frequency is each position or something like that?
Edit: This is part of the code I'm using
float[] mono = new float[size];
// I fill the array with the appropiate values
Noise noise = Noise.real(size);
float[] dst = new float[size + 2];
float[] fft = noise.fft(mono, dst);
// The result array has the pairs of real+imaginary floats in a one dimensional array; even indices
// are real, odd indices are imaginary. DC bin is located at index 0, 1, nyquist at index n-2, n-1
double greatest = 0;
int greatestIdx = 0;
for(int i = 0; i < fft.length / 2; i++) {
float real = fft[i * 2];
float imaginary = fft[i * 2 + 1];
double magnitude = Math.sqrt(real*real+imaginary*imaginary);
if (magnitude > greatest) {
greatest = magnitude;
greatestIdx = i;
}
System.out.printf("index: %d, real: %.5f, imaginary: %.5f\n", i, real, imaginary);
}
I just noticed something I had overlooked. When reading the comment just before the for loop (which is from the sample code provided in GitHub) it says that nyquist is located at the last pair of values of the array. From what I searched, nyquist is 22050Hz, so... To know the frequency corresponding to greatestIdx I should map the range [0,size+2] to the range [0,22050] and calculate the new value? It seems like a pretty unprecise measure.
Taking the prior things into account, maybe I should use another library for more precision? If that is the case, what would be one that let me specify the output frequency range or that gives me approximately the human hearing range by default?

I believe that the answer to your question is here if I understand it correctly https://stackoverflow.com/a/4371627/9834835
To determine the frequency for each FFT bin you may use the formula
F = i * sample / nFFt
where:
i = the FFT index
sample = the sample rate
nFft = your FFT size

AudioTrack - short array to byte array distortion using jlayer(java mp3 decoder)

I'm using jLayer to decode MP3 data, with this call:
SampleBuffer output = (SampleBuffer) decoder.decodeFrame(frameHeader, bitstream);
This call which returns the decoded data, returns an array of short[].
output.getBuffer();
When I call AudioTrack write() with that method, it plays fine as I loop through the file:
at.write(output.getBuffer(), 0, output.getBuffer().length);
However, when I convert the short[] array to byte[] array using any of the methods in this answer: https://stackoverflow.com/a/12347176/1176436 the sound gets distorted and jittery:
at.write(output.getBuffer(), 0, output.getBuffer().length);
becomes:
byte[] array = ShortToByte_Twiddle_Method(output.getBuffer());
at.write(array, 0, array.length);
Am I doing anything wrong and what can I do to fix it? Unfortunately I need the pcm data to be in a byte array for another 3rd party library I'm using. The file is 22kHz if that matters and this is how at is being instantiated:
at = new AudioTrack(AudioManager.STREAM_MUSIC, 22050, AudioFormat.CHANNEL_OUT_STEREO,
AudioFormat.ENCODING_PCM_16BIT, 10000 /* 10 second buffer */,
AudioTrack.MODE_STREAM);
Thank you so much in advance.
Edit: This is how I'm instantiating the AudioTrack variable now. So for 44kHz files, the value that is getting sent is 44100, while for 22kHz files, the value is 22050.
at = new AudioTrack(AudioManager.STREAM_MUSIC, decoder.getOutputFrequency(),
decoder.getOutputChannels() > 1 ? AudioFormat.CHANNEL_OUT_STEREO : AudioFormat.CHANNEL_OUT_MONO,
AudioFormat.ENCODING_PCM_16BIT, 10000 /* 10 second buffer */,
AudioTrack.MODE_STREAM);
This is decode method:
public byte[] decode(InputStream inputStream, int startMs, int maxMs) throws IOException {
ByteArrayOutputStream outStream = new ByteArrayOutputStream(1024);
float totalMs = 0;
boolean seeking = true;
try {
Bitstream bitstream = new Bitstream(inputStream);
Decoder decoder = new Decoder();
boolean done = false;
while (!done) {
Header frameHeader = bitstream.readFrame();
if (frameHeader == null) {
done = true;
} else {
totalMs += frameHeader.ms_per_frame();
if (totalMs >= startMs) {
seeking = false;
}
if (!seeking) {
// logger.debug("Handling header: " + frameHeader.layer_string());
SampleBuffer output = (SampleBuffer) decoder.decodeFrame(frameHeader, bitstream);
short[] pcm = output.getBuffer();
for (short s : pcm) {
outStream.write(s & 0xff);
outStream.write((s >> 8) & 0xff);
}
}
if (totalMs >= (startMs + maxMs)) {
done = true;
}
}
bitstream.closeFrame();
}
return outStream.toByteArray();
} catch (BitstreamException e) {
throw new IOException("Bitstream error: " + e);
} catch (DecoderException e) {
throw new IOException("Decoder error: " + e);
}
}
This is how it sounds (wait a few seconds): https://vimeo.com/60951237 (and this is the actual file: http://www.tonycuffe.com/mp3/tail%20toddle.mp3)
Edit: I would have loved to have split the bounty, but instead I have given the bounty to Bill and the accepted answer to Neil. Both were a tremendous help. For those wondering, I ended up rewriting the Sonic native code which helped me move along the process.

As #Bill Pringlemeir says, the problem is that your conversion method doesn't actually convert. A short is a 16 bit number; a byte is an 8 bit number. The method you have chosen doesn't convert the contents of the shorts (ie go from 16 bits to 8 bits for the contents), it changes the way in which the same collection of bits is stored. As you say, you need something like this:
SampleBuffer output = (SampleBuffer) decoder.decodeFrame(frameHeader, bitstream);
byte[] array = MyShortToByte(output.getBuffer());
at.write(array, 0, array.length);
#Bill Pringlemeir's approach is equivalent to dividing all the shorts by 256 to ensure they fit in the byte range:
byte[] MyShortToByte(short[] buffer) {
int N = buffer.length;
ByteBuffer byteBuf = ByteBuffer.allocate(N);
while (N >= i) {
byte b = (byte)(buffer[i]/256); /*convert to byte. */
byteBuf.put(b);
i++;
}
return byteBuf.array();
}
This will work, but will probably give you very quiet, edgy tones. If you can afford the processing time, a two pass approach will probably give better results:
byte[] MyShortToByte(short[] buffer) {
int N = buffer.length;
short min = 0;
short max = 0;
for (int i=0; i<N; i++) {
if (buffer[i] > max) max = buffer[i];
if (buffer[i] < min) min = buffer[i];
}
short scaling = 1+(max-min)/256; // 1+ ensures we stay within range and guarantee no divide by zero if sequence is pure silence ...
ByteBuffer byteBuf = ByteBuffer.allocate(N);
for (int i=0; i<N; i++) {
byte b = (byte)(buffer[i]/scaling); /*convert to byte. */
byteBuf.put(b);
}
return byteBuf.array();
}
Again, beware signed / unsigned issue. The above works signed-> signed and unsigned->unsigned; but not between the two. It may be that you are reading signed shorts (-32768-32767), but need to output unsigned bytes (0-255), ...
If you can afford the processing time, a more precise (smoother) approach would be to go via floats (this also gets round the signed/unsigned issue):
byte[] MyShortToByte(short[] buffer) {
int N = buffer.length;
float f[] = new float[N];
float min = 0.0f;
float max = 0.0f;
for (int i=0; i<N; i++) {
f[i] = (float)(buffer[i]);
if (f[i] > max) max = f[i];
if (f[i] < min) min = f[i];
}
float scaling = 1.0f+(max-min)/256.0f; // +1 ensures we stay within range and guarantee no divide by zero if sequence is pure silence ...
ByteBuffer byteBuf = ByteBuffer.allocate(N);
for (int i=0; i<N; i++) {
byte b = (byte)(f[i]/scaling); /*convert to byte. */
byteBuf.put(b);
}
return byteBuf.array();
}

The issue is with your short to byte conversion. The byte conversion link preserves all information including the high and low byte portions. When you are converting from 16bit to 8bit PCM samples, you must discard the lower byte. My Java skills are weak, so the following may not work verbatim. See also: short to byte conversion.
ByteBuffer byteBuf = ByteBuffer.allocate(N);
while (N >= i) {
/* byte b = (byte)((buffer[i]>>8)&0xff); convert to byte. native endian */
byte b = (byte)(buffer[i]&0xff); /*convert to byte; swapped endian. */
byteBuf.put(b);
i++;
}
That is the following conversion,
AAAA AAAA SBBB BBBB -> AAAA AAAA, +1 if S==1 and positive else -1 if S==1
A is a bit that is kept. B is a discarded bit and S is a bit that you may wish to use for rounding. The rounding is not needed, but it may sound a little better. Basically, 16 bit PCM is higher resolution than 8 bit PCM. You lose those bits when the conversion is done. The short to byte routine tries to preserve all information.
Of course, you must tell the sound library that you are using 8-bit PCM. My guess,
at = new AudioTrack(AudioManager.STREAM_MUSIC, 22050, AudioFormat.CHANNEL_OUT_STEREO,
AudioFormat.ENCODING_PCM_8BIT, 10000 /* 10 second buffer */,
AudioTrack.MODE_STREAM);
If you can only use 16bit PCM to play audio, then you have to do the inverse and convert the 8bit PCM from the library to 16bit PCM for playback. Also note, that typically, 8bit samples are often NOT straight PCM but u-law or a-law encoded. If the 3rd party library uses these formats, the conversion is different but you should be able to code it from the wikipedia links.
NOTE: I have not included the rounding code as overflow and sign handling will complicate the answer. You must check for overflow (Ie, 0x8f + 1 gives 0xff or 255 + 1 giving -1). However, I suspect the library is not straight 8bit PCM.
See Also: Alsa PCM overview, Multi-media wiki entry on PCM - Ultimately Android uses ALSA for sound.
Other factors that must be correct for a PCM raw buffer are sample rate, number of channels (stereo/mono), PCM format including bits, companding, little/big endian and sample interleaving.
EDIT: After some investigation, the JLayer decoder typically returns big endian 16bit values. The Sonic filter, takes a byte but threats them as 16bit little endian underneath. Finally, the AudioTrack class expects 16 bit little endian underneath. I believe that for some reason the JLayer mp3 decoder will return 16bit little endian values. The decode() method in the question does a byte swap of the 16 bit values. Also, the posted audio sounds as if the bytes are swapped.
public byte[] decode(InputStream inputStream, int startMs, int maxMs, bool swap) throws IOException {
...
short[] pcm = output.getBuffer();
for (short s : pcm) {
if(swap) {
outStream.write(s & 0xff);
outStream.write((s >> 8) & 0xff);
} else {
outStream.write((s >> 8) & 0xff);
outStream.write(s & 0xff);
}
}
...
For 44k mp3s, you call the routine with swap = true;. For the 22k mp3 swap = false. This explains all the reported phenomena. I don't know why the JLayer mp3 decoder would sometimes output big endian and other times little endian. I imagine it depends on the source mp3 and not the sample rate.

sound level rms

I have found some code to calculate microphone sound level (RMS):
public int calculateRMSLevel(byte[] audioData) {
// audioData might be buffered data read from a data line
long lSum = 0;
for (int i = 0; i < audioData.length; i++) {
lSum = lSum + audioData[i];
}
double dAvg = lSum / audioData.length;
double sumMeanSquare = 0d;
for (int j = 0; j < audioData.length; j++) {
sumMeanSquare = sumMeanSquare + Math.pow(audioData[j] - dAvg, 2d);
}
double averageMeanSquare = sumMeanSquare / audioData.length;
return (int) (Math.pow(averageMeanSquare, 0.5d) + 0.5);
}
But it only works for the following audio format:
private AudioFormat getAudioFormat() {
float sampleRate = 8000.0F;
int sampleSizeInBits = 8;
int channels = 1;
boolean signed = true;
boolean bigEndian = true;
return new AudioFormat(sampleRate, sampleSizeInBits, channels, signed,
bigEndian);
}
How to extend the code so it can work with different bitness? If I change the bitness to 16 it returns values of around 50 when silence where for 8 bits it returns 1 or 2.Also I would like to graph the sound levels on a graph, how are the sound level values related to time?

The sample rate doesn't matter, but the bit depth, endianness, and, in a different way, number of channels, do matter.
To see why, you must simply notice that the function in question takes a byte array as an argument and processes each value from that array individually. The byte datatype is an 8-bit value. If you want something that works with 16-bit values, you need to use a different datatype (short) or convert to that from bytes.
Once you do that, you will still get different values for 16 bits vs 8 bit because the range is different: 8 bit goes from -128 to +127 and 16 bit goes from -32768 to +32767, but they are both measuring the same thing, meaning they scaling the same real-word values to different represented values.
As for sound-levels and their relationship to time.... well it depends on your sample rate and the size of the arrays going into this function. For example, if your samplerate is 8kHz and you have 2048 samples per buffer, then your function is going to be called 8000/2048 or about 3.9 times per second, meaning your results are coming in at that rate (every 256 milliseconds).

You can always scale your inputs to the same min-max range to get similar results from different formats.
As for sound level w.r.t. time, there isn't any relation other than samples being apart from each other by 1/SampleRate(in Hz) seconds.

Current best way to populate mixed type byte array

I'm trying to send and receive a byte stream in which certain ranges of bytes represent different pieces of data. I've found ways to convert single primitive datatypes into bytes, but I'm wondering if there's a straightforward way to place certain pieces of data into specified byte regions.
For example, I might need to produce or read something like the following:
byte 1 - int
byte 2-5 - int
byte 6-13 - double
byte 14-21 - double
byte 25 - int
byte 26-45 - string
Any suggestions would be appreciated.

Try DataOutputStream/DataInputStream or, for arrays, the ByteBuffer class.
For storing the integer in X bytes, you may use the following method. If you think it is badly named, you may use the much less descriptive i2os name which is used in several (crypto) algorithm descriptions. Note that the returned octet string uses Big Endian encoding of unsigned ints, which you should specify for your protocol.
public static byte[] possitiveIntegerToOctetString(
final long value, final int octets) {
if (value < 0) {
throw new IllegalArgumentException("Cannot encode negative values");
}
if (octets < 1) {
throw new IllegalArgumentException("Cannot encode a number in negative or zero octets");
}
final int longSizeBytes = Long.SIZE / Byte.SIZE;
final int byteBufferSize = Math.max(octets, longSizeBytes);
final ByteBuffer buf = ByteBuffer.allocate(byteBufferSize);
for (int i = 0; i < byteBufferSize - longSizeBytes; i++) {
buf.put((byte) 0x00);
}
buf.mark();
buf.putLong(value);
// more bytes than long encoding
if (octets >= longSizeBytes) {
return buf.array();
}
// less bytes than long encoding (reset to mark first)
buf.reset();
for (int i = 0; i < longSizeBytes - octets; i++) {
if (buf.get() != 0x00) {
throw new IllegalArgumentException("Value does not fit in " + octets + " octet(s)");
}
}
final byte[] result = new byte[octets];
buf.get(result);
return result;
}
EDIT before storing the string, think of a padding mechanism (spaces would be most used), and character-encoding e.g. String.getBytes(Charset.forName("ASCII")) or "Latin-1". Those are the most common encodings with a single byte per character. Calculating the size of "UTF-8" is slightly more difficult (encode first, add 0x20 valued bytes at the end using ByteBuffer).

You may want to consider having a constant size for each data type. For example, the 32-bit Java int will take up 4 bytes a long will take 8, etc. In fact, if you use Java's DataInputStream and DataOutputStreams, you'll basically be doing that anyway. They have really nice methods like read/writeInt, etc.

Simple data serialization in C

I am currently re-designing an application and stumbled upon a problem serializing some data.
Say I have an array of size mxn
double **data;
that I want to serialize into a
char *dataSerialized
using simple delimiters (one for rows, one for elements).
De-serialization is fairly straightforward, counting delimiters and allocating size for the data to be stored. However, what about the serialize function, say
serialize_matrix(double **data, int m, int n, char **dataSerialized);
What would be the best strategy to determine the size needed by the char array and allocate the appropriate memory for it?
Perhaps using some fixed width exponential representation of double's in a string? Is it possible to just convert all bytes of double into char's and have a sizeof(double) aligned char array? How would I keep the accuracy of the numbers intact?
NOTE:
I need the data in a char array, not in binary, not in a file.
The serialized data will be sent over the network using ZeroMQ between a C server and a Java client. Would it be possible, given the array dimensions and sizeof(double) that it can always be accurately reconstructed between those two?

Java has pretty good support for reading raw bytes and converting into whatever you want.
You can decide on a simple wire-format, and then serialize to this in C, and unserialize in Java.
Here's an example of an extremely simple format, with code to unserialize and serialize.
I've written a slightly larger test program that I can dump somewhere if you want; it creates a random data array in C, serializes, writes the serialized string base64-encoded to stdout. The much smaller java-program then reads, decodes and deserializes this.
C code to serialize:
/*
I'm using this format:
32 bit signed int 32 bit signed int See below
[number of elements in outer array] [number of elements in inner array] [elements]
[elements] is buildt like
[element(0,0)][element(0,1)]...[element(0,y)][element(1,0)]...
each element is sendt like a 64 bit iee754 "double". If your C compiler/architecture is doing something different with its "double"'s, look forward to hours of fun :)
I'm using a couple non-standard functions for byte-swapping here, originally from a BSD, but present in glibc>=2.9.
*/
/* Calculate the bytes required to store a message of x*y doubles */
size_t calculate_size(size_t x, size_t y)
{
/* The two dimensions in the array - each in 32 bits - (2 * 4)*/
size_t sz = 8;
/* a 64 bit IEE754 is by definition 8 bytes long :) */
sz += ((x * y) * 8);
/* and a NUL */
sz++;
return sz;
}
/* Helpers */
static char* write_int32(int32_t, char*);
static char* write_double(double, char*);
/* Actual conversion. That wasn't so hard, was it? */
void convert_data(double** src, size_t x, size_t y, char* dst)
{
dst = write_int32((int32_t) x, dst);
dst = write_int32((int32_t) y, dst);
for(int i = 0; i < x; i++) {
for(int j = 0; j < y; j++) {
dst = write_double(src[i][j], dst);
}
}
*dst = '\0';
}
static char* write_int32(int32_t num, char* c)
{
char* byte;
int i = sizeof(int32_t);
/* Convert to network byte order */
num = htobe32(num);
byte = (char*) (&num);
while(i--) {
*c++ = *byte++;
}
return c;
}
static char* write_double(double d, char* c)
{
/* Here I'm assuming your C programs use IEE754 'double' precision natively.
If you don't, you should be able to convert into this format. A helper library most likely already exists for your platform.
Note that IEE754 endianess isn't defined, but in practice, normal platforms use the same byte order as they do for integers.
*/
char* byte;
int i = sizeof(uint64_t);
uint64_t num = *((uint64_t*)&d);
/* convert to network byte order */
num = htobe64(num);
byte = (char*) (&num);
while(i--) {
*c++ = *byte++;
}
return c;
}
Java code to unserialize:
/* The raw char array from c is now read into the byte[] `bytes` in java */
DataInputStream stream = new DataInputStream(new ByteArrayInputStream(bytes));
int dim_x; int dim_y;
double[][] data;
try {
dim_x = stream.readInt();
dim_y = stream.readInt();
data = new double[dim_x][dim_y];
for(int i = 0; i < dim_x; ++i) {
for(int j = 0; j < dim_y; ++j) {
data[i][j] = stream.readDouble();
}
}
System.out.println("Client:");
System.out.println("Dimensions: "+dim_x+" x "+dim_y);
System.out.println("Data:");
for(int i = 0; i < dim_x; ++i) {
for(int j = 0; j < dim_y; ++j) {
System.out.print(" "+data[i][j]);
}
System.out.println();
}
} catch(IOException e) {
System.err.println("Error reading input");
System.err.println(e.getMessage());
System.exit(1);
}

If you are writing a binary file, you should think of a good way to serialize the actual binary data (64bit) of your double. This could go from directly writing the content of the double to the file (minding endianness) to some more elaborate normalizing serialization schemes (e.g. with a well-defined representation of NaN). That's up to you really. If you expect to be basically among homogeneous architectures, a direct memory dump would probably suffice.
If you want to write to a text file and a are looking for an ASCII representation, I would strongly discourage a decimal numerical representation. Instead, you could convert the 64-bit raw data to ASCII using base64 or something like that.
You really want to keep all the precision that you have in your double!