I am working on a side project at work where I would like to read/write SAS Transport files. The challenge is that numbers are encoded in 64-bit IBM floating point numbers. While I have been able to find plenty of great resources for reading a byte array (containing an IBM float) into a IEEE 32-bit floats and 64-bit floats, I'm struggling to find the code to convert floats/doubles back to IBM floats.
I recently found some code for writing a 32-bit IEEE float back out to a byte array (containing an IBM float). It seems to be working, so I've been trying to translate it to a 64-bit version. I've reversed engineered where most of the magic numbers are coming from, but I've been stumped for over a week now.
I have also tried to translate the functions listed at the end of the SAS Transport documentation to Java, but I've run into a lot of issues related to endiness, Java's lack of unsigned types, and so on. Can anyone provide the code to convert doubles to IBM floating point format?
Just to show the progress I've made, here are some shortened versions of the code I've written so far:
This grabs a 32-bit IBM float from a byte array and generates an IEEE float:
public static double fromIBMFloat(byte[] data, int offset) {
int temp = readIntFromBuffer(data, offset);
int mantissa = temp & 0x00FFFFFF;
int exponent = ((temp >> 24) & 0x7F) - 64;
boolean isNegative = (temp & 0x80000000) != 0;
double result = mantissa * Math.pow(2, 4 * exponent - 24);
if (isNegative) {
result = -result;
}
return result;
}
This is the same thing for 64-bit:
public static double fromIBMDouble(byte[] data, int offset) {
long temp = readLongFromBuffer(data, offset);
long mantissa = temp & 0x00FFFFFFFFFFFFFFL;
long exponent = ((temp >> 56) & 0x7F) - 64;
boolean isNegative = (temp & 0x8000000000000000L) != 0;
double result = mantissa * Math.pow(2, 4 * exponent - 24);
if (isNegative) {
result = -result;
}
return result;
}
Great! These work for going to IEEE floats, but now I need to go the other way. This simple implementation seems to be working for 32-bit floats:
public static void toIBMFloat(double value, byte[] xport, int offset) {
if (value == 0.0 || Double.isNaN(value) || Double.isInfinite(value)) {
writeIntToBuffer(xport, offset, 0);
return;
}
int fconv = Float.floatToIntBits((float)value);
int fmant = (fconv & 0x007FFFFF) | 0x00800000;
int temp = (fconv & 0x7F800000) >> 23;
int t = (temp & 0xFF) - 126;
while ((t & 0x3) != 0) {
++t;
fmant >>= 1;
}
fconv = (fconv & 0x80000000) | (((t >> 2) + 64) << 24) | fmant;
writeIntToBuffer(xport, offset, fconv);
}
Now, the only thing left is to translate that to work with 64-bit IBM floats. A lot of the magic numbers listed relate to the number of bits in the IEEE 32-bit floating point exponent (8-bits) and mantissa (23-bit). So for 64-bit, I just need to switch those to use the 11-bit exponent and 52-bit mantissa. But where does that 126 come from? What is the point of the 0x3 in the while loop?
Any help breaking down the 32-bit version so I can implement a 64-bit version would be greatly appreciated.
I circled back and took another swing at the C implementations provided at the end of the SAS transport documentation. It turns out the issue wasn't with my implementation; it was an issue with my tests.
TL;DR
These are my 64-bit implementations:
public static void writeIBMDouble(double value, byte[] data, int offset) {
long ieee8 = Double.doubleToLongBits(value);
long ieee1 = (ieee8 >>> 32) & 0xFFFFFFFFL;
long ieee2 = ieee8 & 0xFFFFFFFFL;
writeLong(0L, data, offset);
long xport1 = ieee1 & 0x000FFFFFL;
long xport2 = ieee2;
int ieee_exp = 0;
if (xport2 != 0 || ieee1 != 0) {
ieee_exp = (int)(((ieee1 >>> 16) & 0x7FF0) >>> 4) - 1023;
int shift = ieee_exp & 0x3;
xport1 |= 0x00100000L;
if (shift != 0) {
xport1 <<= shift;
xport1 |= ((byte)(((ieee2 >>> 24) & 0xE0) >>> (5 + (3 - shift))));
xport2 <<= shift;
}
xport1 |= (((ieee_exp >>> 2) + 65) | ((ieee1 >>> 24) & 0x80)) << 24;
}
if (-260 <= ieee_exp && ieee_exp <= 248) {
long temp = ((xport1 & 0xFFFFFFFFL) << 32) | (xport2 & 0xFFFFFFFFL);
writeLong(temp, data, offset);
return;
}
writeLong(0xFFFFFFFFFFFFFFFFL, data, offset);
if (ieee_exp > 248) {
data[offset] = 0x7F;
}
}
public static void writeLong(long value, byte[] buffer, int offset) {
buffer[offset] = (byte)(value >>> 56);
buffer[offset + 1] = (byte)(value >>> 48);
buffer[offset + 2] = (byte)(value >>> 40);
buffer[offset + 3] = (byte)(value >>> 32);
buffer[offset + 4] = (byte)(value >>> 24);
buffer[offset + 5] = (byte)(value >>> 16);
buffer[offset + 6] = (byte)(value >>> 8);
buffer[offset + 7] = (byte)value;
}
And:
public static double readIBMDouble(byte[] data, int offset) {
long temp = readLong(data, offset);
long ieee = 0L;
long xport1 = temp >>> 32;
long xport2 = temp & 0x00000000FFFFFFFFL;
long ieee1 = xport1 & 0x00ffffff;
long ieee2 = xport2;
if (ieee2 == 0L && xport1 == 0L) {
return Double.longBitsToDouble(ieee);
}
int shift = 0;
int nib = (int)xport1;
if ((nib & 0x00800000) != 0) {
shift = 3;
} else if ((nib & 0x00400000) != 0) {
shift = 2;
} else if ((nib & 0x00200000) != 0) {
shift = 1;
}
if (shift != 0) {
ieee1 >>>= shift;
ieee2 = (xport2 >>> shift) | ((xport1 & 0x00000007) << (29 + (3 - shift)));
}
ieee1 &= 0xffefffff;
ieee1 |= (((((long)(data[offset] & 0x7f) - 65) << 2) + shift + 1023) << 20) | (xport1 & 0x80000000);
ieee = ieee1 << 32 | ieee2;
return Double.longBitsToDouble(ieee);
}
public static long readLong(byte[] buffer, int offset) {
long result = unsignedByteToLong(buffer[offset]) << 56;
result |= unsignedByteToLong(buffer[offset + 1]) << 48;
result |= unsignedByteToLong(buffer[offset + 2]) << 40;
result |= unsignedByteToLong(buffer[offset + 3]) << 32;
result |= unsignedByteToLong(buffer[offset + 4]) << 24;
result |= unsignedByteToLong(buffer[offset + 5]) << 16;
result |= unsignedByteToLong(buffer[offset + 6]) << 8;
result |= unsignedByteToLong(buffer[offset + 7]);
return result;
}
private static long unsignedByteToLong(byte value) {
return (long)value & 0xFF;
}
These are basically a one-to-one translation from what's in the document, except I convert the byte[] into a long up-front and just do bit-twiddling instead of working directly with bytes.
I also realized the code in the documentation had some special cases included for "missing" values that are specific to the SAS transport standard and have nothing to do with IBM hexidecimal floating point numbers. In fact, the Double.longBitsToDouble method detects the invalid bit-sequence and just sets the value to NaN. I moved this code out since it wasn't going to work anyway.
The good thing is that as part of this exercise I did learn a lot of tricks to bit manipulation in Java. For instance, a lot of the issues I ran into involving sign were resolved by using the >>> operator instead of the >> operator. Other than that, you just need to be careful upcasting to mask with 0xFF, 0xFFFF, etc. to make sure the sign is ignored.
I also learned about ByteBuffer which can facilitate loading back and forth among byte[] and primitives/strings; however, that comes with some minor overhead. But it would handle any endianness issues. It turns out endianness wasn't even a concern since most architectures in use today (x86) are little endian to begin with.
It seems reading/writing SAS transport files is a pretty common need, especially in the clinical trials arena so hopefully anyone working in Java/C# won't have to go through the trouble I did.
I'm trying to send Java's signed integers over TCP to a C client.
At the Java side, I write the integers to the outputstream like so:
static ByteBuffer wrapped = ByteBuffer.allocateDirect(4); // big-endian by default
public static void putInt(OutputStream out, int nr) throws IOException {
wrapped.rewind();
wrapped.putInt(nr);
wrapped.rewind();
for (int i = 0; i < 4; i++)
out.write(wrapped.get());
}
At the C side, I read the integers like so:
int cnt = 0;
char buf[1];
char sizebuf[4];
while(cnt < 4) {
iResult = recv(ConnectSocket, buf, 1, 0);
if (iResult <= 0) continue;
sizebuf[cnt] = buf[0];
cnt++;
}
However, how do I convert the char array to an integer in C?
Edit
I have tried the following (and the reverse):
int charsToInt(char* array) {
return (array[3] << 24) | (array[2] << 16) | (array[1] << 8) | array[0];
}
Edited again, because I forgot the tags.
Data
For example of what happens currently:
I receive:
char 0
char 0
char 12
char -64
the int becomes 2448
and use the function for creating the int from the char array:
int charsToInt(char* array) {
return ntohl(*((int*) array));
}
I expect the signed integer: 3264
Update
I will investigate more after some sleep..
Update
I have a Java client which interprets the integers correctly and receives the exact same bytes:
0
0
12
-64
That depends on endianness, but you want either:
int x = sizebuf[0] +
(sizebuf[1] << 8) +
(sizebuf[2] << 16) +
(sizebuf[3] << 24);
or:
int x = sizebuf[3] +
(sizebuf[2] << 8) +
(sizebuf[1] << 16) +
(sizebuf[0] << 24);
Note that sizebuf needs to have an unsigned type for this to work correctly. Otherwise you need to mask off any sign-extended values you don't want:
int x = (sizebuf[3] & 0x000000ff) +
((sizebuf[2] << 8) & 0x0000ff00) +
((sizebuf[1] << 16) & 0x00ff0000) +
((sizebuf[0] << 24) & 0xff000000);
The classical C library has the method you want already, and it is independent from the machine endianness: ntohl!
// buf is a char */uint8_t *
uint32_t from_network = *((uint32_t *) buf);
uint32_t ret = ntohl(from_network);
This, and htonl for the reverse etc expect that the "network order" is big endian.
(the code above presupposes that buf has at least 4 bytes; the return type, and argument type, of ntohl and htonl are uint32_t; the JLS defines an int as 4 bytes so you are guaranteed the result)
To convert you char array, one possibility is to cast it to int* and to store the result :
int result = *((int*) sizebuf)
This is valid and one line. Other possibility is to compute integer from chars.
for (i = 0 ; i < 4; i++)
result = result << sizeof(char) + buf[0]
Choose the one that you prefer.
Alexis.
Edit :
sizeof(char) is 1 because sizeof return a Byte result. So the right line is :
result = result << (sizeof(char) * 8) + buf[0]
This question is usually asked as a part of another question but it turns out that the answer is long. I've decided to answer it here so I can link to it elsewhere.
Although I'm not aware of a way that Java can produce audio samples for us at this time, if that changes in the future, this can be a place for it. I know that JavaFX has some stuff like this, for example AudioSpectrumListener, but still not a way to access samples directly.
I'm using javax.sound.sampled for playback and/or recording but I'd like to do something with the audio.
Perhaps I'd like to display it visually or process it in some way.
How do I access audio sample data to do that with Java Sound?
See also:
Java Sound Tutorials (Official)
Java Sound Resources (Unofficial)
Well, the simplest answer is that at the moment Java can't produce sample data for the programmer.
This quote is from the official tutorial:
There are two ways to apply signal processing:
You can use any processing supported by the mixer or its component lines, by querying for Control objects and then setting the controls as the user desires. Typical controls supported by mixers and lines include gain, pan, and reverberation controls.
If the kind of processing you need isn't provided by the mixer or its lines, your program can operate directly on the audio bytes, manipulating them as desired.
This page discusses the first technique in greater detail, because there is no special API for the second technique.
Playback with javax.sound.sampled largely acts as a bridge between the file and the audio device. The bytes are read in from the file and sent off.
Don't assume the bytes are meaningful audio samples! Unless you happen to have an 8-bit AIFF file, they aren't. (On the other hand, if the samples are definitely 8-bit signed, you can do arithmetic with them. Using 8-bit is one way to avoid the complexity described here, if you're just playing around.)
So instead, I'll enumerate the types of AudioFormat.Encoding and describe how to decode them yourself. This answer will not cover how to encode them, but it's included in the complete code example at the bottom. Encoding is mostly just the decoding process in reverse.
This is a long answer but I wanted to give a thorough overview.
A Little About Digital Audio
Generally when digital audio is explained, we're referring to Linear Pulse-Code Modulation (LPCM).
A continuous sound wave is sampled at regular intervals and the amplitudes are quantized to integers of some scale.
Shown here is a sine wave sampled and quantized to 4-bit:
(Notice that the most positive value in two's complement representation is 1 less than the most negative value. This is a minor detail to be aware of. For example if you're clipping audio and forget this, the positive clips will overflow.)
When we have audio on the computer, we have an array of these samples. A sample array is what we want to turn the byte array in to.
To decode PCM samples, we don't care much about the sample rate or number of channels, so I won't be saying much about them here. Channels are usually interleaved, so that if we had an array of them, they'd be stored like this:
Index 0: Sample 0 (Left Channel)
Index 1: Sample 0 (Right Channel)
Index 2: Sample 1 (Left Channel)
Index 3: Sample 1 (Right Channel)
Index 4: Sample 2 (Left Channel)
Index 5: Sample 2 (Right Channel)
...
In other words, for stereo, the samples in the array just alternate between left and right.
Some Assumptions
All of the code examples will assume the following declarations:
byte[] bytes; The byte array, read from the AudioInputStream.
float[] samples; The output sample array that we're going to fill.
float sample; The sample we're currently working on.
long temp; An interim value used for general manipulation.
int i; The position in the byte array where the current sample's data starts.
We'll normalize all of the samples in our float[] array to the range of -1f <= sample <= 1f. All of the floating-point audio I've seen comes this way and it's pretty convenient.
If our source audio doesn't already come like that (as is for e.g. integer samples), we can normalize them ourselves using the following:
sample = sample / fullScale(bitsPerSample);
Where fullScale is 2bitsPerSample - 1, i.e. Math.pow(2, bitsPerSample-1).
How do I coerce the byte array in to meaningful data?
The byte array contains the sample frames split up and all in a line. This is actually very straight-forward except for something called endianness, which is the ordering of the bytes in each sample packet.
Here's a diagram. This sample (packed in to a byte array) holds the decimal value 9999:
24-bit sample as big-endian:
bytes[i] bytes[i + 1] bytes[i + 2]
┌──────┐ ┌──────┐ ┌──────┐
00000000 00100111 00001111
24-bit sample as little-endian:
bytes[i] bytes[i + 1] bytes[i + 2]
┌──────┐ ┌──────┐ ┌──────┐
00001111 00100111 00000000
They hold the same binary values; however, the byte orders are reversed.
In big-endian, the more significant bytes come before the less significant bytes.
In little-endian, the less significant bytes come before the more significant bytes.
WAV files are stored in little-endian order and AIFF files are stored in big-endian order. Endianness can be obtained from AudioFormat.isBigEndian.
To concatenate the bytes and put them in to our long temp variable, we:
Bitwise AND each byte with the mask 0xFF (which is 0b1111_1111) to avoid sign-extension when the byte is automatically promoted. (char, byte and short are promoted to int when arithmetic is performed on them.) See also What does value & 0xff do in Java?
Bit shift each byte in to position.
Bitwise OR the bytes together.
Here's a 24-bit example:
long temp;
if (isBigEndian) {
temp = (
((bytes[i ] & 0xffL) << 16)
| ((bytes[i + 1] & 0xffL) << 8)
| (bytes[i + 2] & 0xffL)
);
} else {
temp = (
(bytes[i ] & 0xffL)
| ((bytes[i + 1] & 0xffL) << 8)
| ((bytes[i + 2] & 0xffL) << 16)
);
}
Notice that the shift order is reversed based on endianness.
This can also be generalized to a loop, which can be seen in the full code at the bottom of this answer. (See the unpackAnyBit and packAnyBit methods.)
Now that we have the bytes concatenated together, we can take a few more steps to turn them in to a sample. The next steps depend on the actual encoding.
How do I decode Encoding.PCM_SIGNED?
The two's complement sign must be extended. This means that if the most significant bit (MSB) is set to 1, we fill all the bits above it with 1s. The arithmetic right-shift (>>) will do the filling for us automatically if the sign bit is set, so I usually do it this way:
int bitsToExtend = Long.SIZE - bitsPerSample;
float sample = (temp << bitsToExtend) >> bitsToExtend.
(Where Long.SIZE is 64. If our temp variable wasn't a long, we'd use something else. If we used e.g. int temp instead, we'd use 32.)
To understand how this works, here's a diagram of sign-extending 8-bit to 16-bit:
11111111 is the byte value -1, but the upper bits of the short are 0.
Shift the byte's MSB in to the MSB position of the short.
0000 0000 1111 1111
<< 8
───────────────────
1111 1111 0000 0000
Shift it back and the right-shift fills all the upper bits with 1s.
We now have the short value of -1.
1111 1111 0000 0000
>> 8
───────────────────
1111 1111 1111 1111
Positive values (that had a 0 in the MSB) are left unchanged. This is a nice property of the arithmetic right-shift.
Then normalize the sample, as described in Some Assumptions.
You might not need to write explicit sign-extension if your code is simple
Java does sign-extension automatically when converting from one integral type to a larger type, for example byte to int. If you know that your input and output format are always signed, you can use the automatic sign-extension while concatenating bytes in the earlier step.
Recall from the section above (How do I coerce the byte array in to meaningful data?) that we used b & 0xFF to prevent sign-extension from occurring. If you just remove the & 0xFF from the highest byte, sign-extension will happen automatically.
For example, the following decodes signed, big-endian, 16-bit samples:
for (int i = 0; i < bytes.length; i++) {
int sample = (bytes[i] << 8) // high byte is sign-extended
| (bytes[i + 1] & 0xFF); // low byte is not
// ...
}
How do I decode Encoding.PCM_UNSIGNED?
We turn it in to a signed number. Unsigned samples are simply offset so that, for example:
An unsigned value of 0 corresponds to the most negative signed value.
An unsigned value of 2bitsPerSample - 1 corresponds to the signed value of 0.
An unsigned value of 2bitsPerSample corresponds to the most positive signed value.
So this turns out to be pretty simple. Just subtract the offset:
float sample = temp - fullScale(bitsPerSample);
Then normalize the sample, as described in Some Assumptions.
How do I decode Encoding.PCM_FLOAT?
This is new since Java 7.
In practice, floating-point PCM is typically either IEEE 32-bit or IEEE 64-bit and already normalized to the range of ±1.0. The samples can be obtained with the utility methods Float#intBitsToFloat and Double#longBitsToDouble.
// IEEE 32-bit
float sample = Float.intBitsToFloat((int) temp);
// IEEE 64-bit
double sampleAsDouble = Double.longBitsToDouble(temp);
float sample = (float) sampleAsDouble; // or just use double for arithmetic
How do I decode Encoding.ULAW and Encoding.ALAW?
These are companding compression codecs that are more common in telephones and such. They're supported by javax.sound.sampled I assume because they're used by Sun's Au format. (However, it's not limited to just this type of container. For example, WAV can contain these encodings.)
You can conceptualize A-law and μ-law like they're a floating-point format. These are PCM formats but the range of values is non-linear.
There are two ways to decode them. I'll show the way which uses the mathematical formula. You can also decode them by manipulating the binary directly which is described in this blog post but it's more esoteric-looking.
For both, the compressed data is 8-bit. Standardly A-law is 13-bit when decoded and μ-law is 14-bit when decoded; however, applying the formula yields a range of ±1.0.
Before you can apply the formula, there are three things to do:
Some of the bits are standardly inverted for storage due to reasons involving data integrity.
They're stored as sign and magnitude (rather than two's complement).
The formula also expects a range of ±1.0, so the 8-bit value has to be scaled.
For μ-law all the bits are inverted, so:
temp ^= 0xffL; // 0xff == 0b1111_1111
(Note that we can't use ~, because we don't want to invert the high bits of the long.)
For A-law, every other bit is inverted, so:
temp ^= 0x55L; // 0x55 == 0b0101_0101
(XOR can be used to do inversion. See How do you set, clear and toggle a bit?)
To convert from sign and magnitude to two's complement, we:
Check to see if the sign bit was set.
If so, clear the sign bit and negate the number.
// 0x80 == 0b1000_0000
if ((temp & 0x80L) != 0) {
temp ^= 0x80L;
temp = -temp;
}
Then scale the encoded numbers, the same way as described in Some Assumptions:
sample = temp / fullScale(8);
Now we can apply the expansion.
The μ-law formula translated to Java is then:
sample = (float) (
signum(sample)
*
(1.0 / 255.0)
*
(pow(256.0, abs(sample)) - 1.0)
);
The A-law formula translated to Java is then:
float signum = signum(sample);
sample = abs(sample);
if (sample < (1.0 / (1.0 + log(87.7)))) {
sample = (float) (
sample * ((1.0 + log(87.7)) / 87.7)
);
} else {
sample = (float) (
exp((sample * (1.0 + log(87.7))) - 1.0) / 87.7
);
}
sample = signum * sample;
Here's the full example code for the SimpleAudioConversion class.
package mcve.audio;
import javax.sound.sampled.AudioFormat;
import javax.sound.sampled.AudioFormat.Encoding;
import static java.lang.Math.*;
/**
* <p>Performs simple audio format conversion.</p>
*
* <p>Example usage:</p>
*
* <pre>{#code AudioInputStream ais = ... ;
* SourceDataLine line = ... ;
* AudioFormat fmt = ... ;
*
* // do setup
*
* for (int blen = 0; (blen = ais.read(bytes)) > -1;) {
* int slen;
* slen = SimpleAudioConversion.decode(bytes, samples, blen, fmt);
*
* // do something with samples
*
* blen = SimpleAudioConversion.encode(samples, bytes, slen, fmt);
* line.write(bytes, 0, blen);
* }}</pre>
*
* #author Radiodef
* #see Overview on Stack Overflow
*/
public final class SimpleAudioConversion {
private SimpleAudioConversion() {}
/**
* Converts from a byte array to an audio sample float array.
*
* #param bytes the byte array, filled by the AudioInputStream
* #param samples an array to fill up with audio samples
* #param blen the return value of AudioInputStream.read
* #param fmt the source AudioFormat
*
* #return the number of valid audio samples converted
*
* #throws NullPointerException if bytes, samples or fmt is null
* #throws ArrayIndexOutOfBoundsException
* if bytes.length is less than blen or
* if samples.length is less than blen / bytesPerSample(fmt.getSampleSizeInBits())
*/
public static int decode(byte[] bytes,
float[] samples,
int blen,
AudioFormat fmt) {
int bitsPerSample = fmt.getSampleSizeInBits();
int bytesPerSample = bytesPerSample(bitsPerSample);
boolean isBigEndian = fmt.isBigEndian();
Encoding encoding = fmt.getEncoding();
double fullScale = fullScale(bitsPerSample);
int i = 0;
int s = 0;
while (i < blen) {
long temp = unpackBits(bytes, i, isBigEndian, bytesPerSample);
float sample = 0f;
if (encoding == Encoding.PCM_SIGNED) {
temp = extendSign(temp, bitsPerSample);
sample = (float) (temp / fullScale);
} else if (encoding == Encoding.PCM_UNSIGNED) {
temp = unsignedToSigned(temp, bitsPerSample);
sample = (float) (temp / fullScale);
} else if (encoding == Encoding.PCM_FLOAT) {
if (bitsPerSample == 32) {
sample = Float.intBitsToFloat((int) temp);
} else if (bitsPerSample == 64) {
sample = (float) Double.longBitsToDouble(temp);
}
} else if (encoding == Encoding.ULAW) {
sample = bitsToMuLaw(temp);
} else if (encoding == Encoding.ALAW) {
sample = bitsToALaw(temp);
}
samples[s] = sample;
i += bytesPerSample;
s++;
}
return s;
}
/**
* Converts from an audio sample float array to a byte array.
*
* #param samples an array of audio samples to encode
* #param bytes an array to fill up with bytes
* #param slen the return value of the decode method
* #param fmt the destination AudioFormat
*
* #return the number of valid bytes converted
*
* #throws NullPointerException if samples, bytes or fmt is null
* #throws ArrayIndexOutOfBoundsException
* if samples.length is less than slen or
* if bytes.length is less than slen * bytesPerSample(fmt.getSampleSizeInBits())
*/
public static int encode(float[] samples,
byte[] bytes,
int slen,
AudioFormat fmt) {
int bitsPerSample = fmt.getSampleSizeInBits();
int bytesPerSample = bytesPerSample(bitsPerSample);
boolean isBigEndian = fmt.isBigEndian();
Encoding encoding = fmt.getEncoding();
double fullScale = fullScale(bitsPerSample);
int i = 0;
int s = 0;
while (s < slen) {
float sample = samples[s];
long temp = 0L;
if (encoding == Encoding.PCM_SIGNED) {
temp = (long) (sample * fullScale);
} else if (encoding == Encoding.PCM_UNSIGNED) {
temp = (long) (sample * fullScale);
temp = signedToUnsigned(temp, bitsPerSample);
} else if (encoding == Encoding.PCM_FLOAT) {
if (bitsPerSample == 32) {
temp = Float.floatToRawIntBits(sample);
} else if (bitsPerSample == 64) {
temp = Double.doubleToRawLongBits(sample);
}
} else if (encoding == Encoding.ULAW) {
temp = muLawToBits(sample);
} else if (encoding == Encoding.ALAW) {
temp = aLawToBits(sample);
}
packBits(bytes, i, temp, isBigEndian, bytesPerSample);
i += bytesPerSample;
s++;
}
return i;
}
/**
* Computes the block-aligned bytes per sample of the audio format,
* using Math.ceil(bitsPerSample / 8.0).
* <p>
* Round towards the ceiling because formats that allow bit depths
* in non-integral multiples of 8 typically pad up to the nearest
* integral multiple of 8. So for example, a 31-bit AIFF file will
* actually store 32-bit blocks.
*
* #param bitsPerSample the return value of AudioFormat.getSampleSizeInBits
* #return The block-aligned bytes per sample of the audio format.
*/
public static int bytesPerSample(int bitsPerSample) {
return (int) ceil(bitsPerSample / 8.0); // optimization: ((bitsPerSample + 7) >>> 3)
}
/**
* Computes the largest magnitude representable by the audio format,
* using Math.pow(2.0, bitsPerSample - 1). Note that for two's complement
* audio, the largest positive value is one less than the return value of
* this method.
* <p>
* The result is returned as a double because in the case that
* bitsPerSample is 64, a long would overflow.
*
* #param bitsPerSample the return value of AudioFormat.getBitsPerSample
* #return the largest magnitude representable by the audio format
*/
public static double fullScale(int bitsPerSample) {
return pow(2.0, bitsPerSample - 1); // optimization: (1L << (bitsPerSample - 1))
}
private static long unpackBits(byte[] bytes,
int i,
boolean isBigEndian,
int bytesPerSample) {
switch (bytesPerSample) {
case 1: return unpack8Bit(bytes, i);
case 2: return unpack16Bit(bytes, i, isBigEndian);
case 3: return unpack24Bit(bytes, i, isBigEndian);
default: return unpackAnyBit(bytes, i, isBigEndian, bytesPerSample);
}
}
private static long unpack8Bit(byte[] bytes, int i) {
return bytes[i] & 0xffL;
}
private static long unpack16Bit(byte[] bytes,
int i,
boolean isBigEndian) {
if (isBigEndian) {
return (
((bytes[i ] & 0xffL) << 8)
| (bytes[i + 1] & 0xffL)
);
} else {
return (
(bytes[i ] & 0xffL)
| ((bytes[i + 1] & 0xffL) << 8)
);
}
}
private static long unpack24Bit(byte[] bytes,
int i,
boolean isBigEndian) {
if (isBigEndian) {
return (
((bytes[i ] & 0xffL) << 16)
| ((bytes[i + 1] & 0xffL) << 8)
| (bytes[i + 2] & 0xffL)
);
} else {
return (
(bytes[i ] & 0xffL)
| ((bytes[i + 1] & 0xffL) << 8)
| ((bytes[i + 2] & 0xffL) << 16)
);
}
}
private static long unpackAnyBit(byte[] bytes,
int i,
boolean isBigEndian,
int bytesPerSample) {
long temp = 0;
if (isBigEndian) {
for (int b = 0; b < bytesPerSample; b++) {
temp |= (bytes[i + b] & 0xffL) << (
8 * (bytesPerSample - b - 1)
);
}
} else {
for (int b = 0; b < bytesPerSample; b++) {
temp |= (bytes[i + b] & 0xffL) << (8 * b);
}
}
return temp;
}
private static void packBits(byte[] bytes,
int i,
long temp,
boolean isBigEndian,
int bytesPerSample) {
switch (bytesPerSample) {
case 1: pack8Bit(bytes, i, temp);
break;
case 2: pack16Bit(bytes, i, temp, isBigEndian);
break;
case 3: pack24Bit(bytes, i, temp, isBigEndian);
break;
default: packAnyBit(bytes, i, temp, isBigEndian, bytesPerSample);
break;
}
}
private static void pack8Bit(byte[] bytes, int i, long temp) {
bytes[i] = (byte) (temp & 0xffL);
}
private static void pack16Bit(byte[] bytes,
int i,
long temp,
boolean isBigEndian) {
if (isBigEndian) {
bytes[i ] = (byte) ((temp >>> 8) & 0xffL);
bytes[i + 1] = (byte) ( temp & 0xffL);
} else {
bytes[i ] = (byte) ( temp & 0xffL);
bytes[i + 1] = (byte) ((temp >>> 8) & 0xffL);
}
}
private static void pack24Bit(byte[] bytes,
int i,
long temp,
boolean isBigEndian) {
if (isBigEndian) {
bytes[i ] = (byte) ((temp >>> 16) & 0xffL);
bytes[i + 1] = (byte) ((temp >>> 8) & 0xffL);
bytes[i + 2] = (byte) ( temp & 0xffL);
} else {
bytes[i ] = (byte) ( temp & 0xffL);
bytes[i + 1] = (byte) ((temp >>> 8) & 0xffL);
bytes[i + 2] = (byte) ((temp >>> 16) & 0xffL);
}
}
private static void packAnyBit(byte[] bytes,
int i,
long temp,
boolean isBigEndian,
int bytesPerSample) {
if (isBigEndian) {
for (int b = 0; b < bytesPerSample; b++) {
bytes[i + b] = (byte) (
(temp >>> (8 * (bytesPerSample - b - 1))) & 0xffL
);
}
} else {
for (int b = 0; b < bytesPerSample; b++) {
bytes[i + b] = (byte) ((temp >>> (8 * b)) & 0xffL);
}
}
}
private static long extendSign(long temp, int bitsPerSample) {
int bitsToExtend = Long.SIZE - bitsPerSample;
return (temp << bitsToExtend) >> bitsToExtend;
}
private static long unsignedToSigned(long temp, int bitsPerSample) {
return temp - (long) fullScale(bitsPerSample);
}
private static long signedToUnsigned(long temp, int bitsPerSample) {
return temp + (long) fullScale(bitsPerSample);
}
// mu-law constant
private static final double MU = 255.0;
// A-law constant
private static final double A = 87.7;
// natural logarithm of A
private static final double LN_A = log(A);
private static float bitsToMuLaw(long temp) {
temp ^= 0xffL;
if ((temp & 0x80L) != 0) {
temp = -(temp ^ 0x80L);
}
float sample = (float) (temp / fullScale(8));
return (float) (
signum(sample)
*
(1.0 / MU)
*
(pow(1.0 + MU, abs(sample)) - 1.0)
);
}
private static long muLawToBits(float sample) {
double sign = signum(sample);
sample = abs(sample);
sample = (float) (
sign * (log(1.0 + (MU * sample)) / log(1.0 + MU))
);
long temp = (long) (sample * fullScale(8));
if (temp < 0) {
temp = -temp ^ 0x80L;
}
return temp ^ 0xffL;
}
private static float bitsToALaw(long temp) {
temp ^= 0x55L;
if ((temp & 0x80L) != 0) {
temp = -(temp ^ 0x80L);
}
float sample = (float) (temp / fullScale(8));
float sign = signum(sample);
sample = abs(sample);
if (sample < (1.0 / (1.0 + LN_A))) {
sample = (float) (sample * ((1.0 + LN_A) / A));
} else {
sample = (float) (exp((sample * (1.0 + LN_A)) - 1.0) / A);
}
return sign * sample;
}
private static long aLawToBits(float sample) {
double sign = signum(sample);
sample = abs(sample);
if (sample < (1.0 / A)) {
sample = (float) ((A * sample) / (1.0 + LN_A));
} else {
sample = (float) ((1.0 + log(A * sample)) / (1.0 + LN_A));
}
sample *= sign;
long temp = (long) (sample * fullScale(8));
if (temp < 0) {
temp = -temp ^ 0x80L;
}
return temp ^ 0x55L;
}
}
This is how you get the actual sample data from the currently playing sound. The other excellent answer will tell you what the data means. Haven't tried it on another OS than my Windows 10 machine YMMV. For me it pulls the current system default recording device. On Windows set it to "Stereo Mix" instead of "Microphone" to get playing sound. You may have to toggle "Show Disabled Devices" to see "Stereo Mix".
import javax.sound.sampled.*;
public class SampleAudio {
private static long extendSign(long temp, int bitsPerSample) {
int extensionBits = 64 - bitsPerSample;
return (temp << extensionBits) >> extensionBits;
}
public static void main(String[] args) throws LineUnavailableException {
float sampleRate = 8000;
int sampleSizeBits = 16;
int numChannels = 1; // Mono
AudioFormat format = new AudioFormat(sampleRate, sampleSizeBits, numChannels, true, true);
TargetDataLine tdl = AudioSystem.getTargetDataLine(format);
tdl.open(format);
tdl.start();
if (!tdl.isOpen()) {
System.exit(1);
}
byte[] data = new byte[(int)sampleRate*10];
int read = tdl.read(data, 0, (int)sampleRate*10);
if (read > 0) {
for (int i = 0; i < read-1; i = i + 2) {
long val = ((data[i] & 0xffL) << 8L) | (data[i + 1] & 0xffL);
long valf = extendSign(val, 16);
System.out.println(i + "\t" + valf);
}
}
tdl.close();
}
}
I am writing a FIX/FAST decoder for negative numbers as described below:
My question is:
How to fill the high-end bits of a Java byte with 1s as it is described above? I am probably unaware of some bit manipulation magic I need to in this conversion.
So I need to go from 01000110 00111010 01011101 to 11110001 10011101 01011101.
I know how to shift by 7 to drop the 8th bit. What I don't know is how to fill the high-end bits with 1s.
It seems like the question you're asking doesn't really match up with the problem you're trying to solve. You're not trying to fill in the high bits with 1; you're trying to decode a stop-bit-encoded integer from a buffer, which involves discarding the sign bits while combining the payload bits. And, of course, you want to stop after you find a byte with a 1 in the stop bit position. The method below should decode the value correctly:
private static final byte SIGN_BIT = (byte)0x40;
private static final byte STOP_BIT = (byte)0x80;
private static final byte PAYLOAD_MASK = 0x7F;
public static int decodeInt(final ByteBuffer buffer) {
int value = 0;
int currentByte = buffer.get();
if ((currentByte & SIGN_BIT) > 0)
value = -1;
value = (value << 7) | (currentByte & PAYLOAD_MASK);
if ((currentByte & STOP_BIT) != 0)
return value;
currentByte = buffer.get();
value = (value << 7) | (currentByte & PAYLOAD_MASK);
if ((currentByte & STOP_BIT) != 0)
return value;
currentByte = buffer.get();
value = (value << 7) | (currentByte & PAYLOAD_MASK);
if ((currentByte & STOP_BIT) != 0)
return value;
currentByte = buffer.get();
value = (value << 7) | (currentByte & PAYLOAD_MASK);
if ((currentByte & STOP_BIT) != 0)
return value;
currentByte = buffer.get();
value = (value << 7) | (currentByte & PAYLOAD_MASK);
return value;
}
A loop would be cleaner, but I unrolled it manually since messaging protocols tend to be hot code paths, and there's a fixed maximum byte length (5 bytes). For simplicity's sake, I read the bytes from a ByteBuffer, so you may need to adjust the logic based on how you're reading the encoded data.
Fillig the high bits might go as:
int fillHighBits(int b) { // 0001abcd
int n = Integer.highestOneBit(b); // 00010000
n = ~n; // 11101111
++n; // 11110000
return (n | b) 0xFF; // 1111abcd
}
As expression
(~Integer.highestOneBit(b) + 1) | b
Though the examples you gave lets me doubt this is what you want.
This can be done very simply using a simple accumulator where you shift in 7 bits at a time. You need to keep track of how many bits you have in the accumulator.
Sign extension can be performed by simple logical shift left follwed by arithmetic shift right (by the same distance) to copy the topmost bit to all unused positions.
byte[] input = new byte[] { 0x46, 0x3A, (byte) 0xDD };
int accumulator = 0;
int bitCount = 0;
for (byte b : input) {
accumulator = (accumulator << 7) | (b & 0x7F);
bitCount += 7;
}
// now sign extend the bits in accumulator
accumulator <<= (32 - bitCount);
accumulator >>= (32 - bitCount);
System.out.println(Integer.toHexString(accumulator));
The whole trick is that >>N operator replicates the top bit N times.
do logical OR (|) with a number which has highend bits set to 1 and rest are 0
for example:
1010101010101010
OR 1111111100000000
--------------------
11111111101010101
something like this:
int x = ...;
x = x | 0xF000;
I have a spec which reads the next two bytes are signed int.
To read that in java i have the following
When i read a signed int in java using the following code i get a value of 65449
Logic for calculation of unsigned
int a =(byte[1] & 0xff) <<8
int b =(byte[0] & 0xff) <<0
int c = a+b
I believe this is wrong because if i and with 0xff i get an unsigned equivalent
so i removed the & 0xff and the logic as given below
int a = byte[1] <<8
int b = byte[0] << 0
int c = a+b
which gives me the value -343
byte[1] =-1
byte[0]=-87
I tried to offset these values with the way the spec reads but this looks wrong.Since the size of the heap doesnt fall under this.
Which is the right way to do for signed int calculation in java?
Here is how the spec goes
somespec() { xtype 8 uint8 xStyle 16 int16 }
xStyle :A signed integer that represents an offset (in bytes) from the start of this Widget() structure to the start of an xStyle() structure that expresses inherited styles for defined by page widget as well as styles that apply specifically to this widget.
If you value is a signed 16-bit you want a short and int is 32-bit which can also hold the same values but not so naturally.
It appears you wants a signed little endian 16-bit value.
byte[] bytes =
short s = ByteBuffer.wrap(bytes).order(ByteOrder.LITTLE_ENDIAN).getShort();
or
short s = (short) ((bytes[0] & 0xff) | (bytes[1] << 8));
BTW: You can use an int but its not so simple.
// to get a sign extension.
int i = ((bytes[0] & 0xff) | (bytes[1] << 8)) << 16 >> 16;
or
int i = (bytes[0] & 0xff) | (short) (bytes[1] << 8));
Assuming that bytes[1] is the MSB, and bytes[0] is the LSB, and that you want the answer to be a 16 bit signed integer:
short res16 = ((bytes[1] << 8) | bytes[0]);
Then to get a 32 bit signed integer:
int res32 = res16; // sign extends.
By the way, the specification should say which of the two bytes is the MSB, and which is the LSB. If it doesn't and if there aren't any examples, you can't implement it!
Somewhere in the spec it will say how an "int16" is represented. Paste THAT part. Or paste a link to the spec so that we can read it ourselves.
Take a look on DataInputStream.readInt(). You can either steel code from there or just use DataInputStream: wrap your input stream with it and then read typed data easily.
For your convenience this is the code:
public final int readInt() throws IOException {
int ch1 = in.read();
int ch2 = in.read();
int ch3 = in.read();
int ch4 = in.read();
if ((ch1 | ch2 | ch3 | ch4) < 0)
throw new EOFException();
return ((ch1 << 24) + (ch2 << 16) + (ch3 << 8) + (ch4 << 0));
}
I can't compile it right now, but I would do (assuming byte1 and byte0 are realling of byte type).
int result = byte1;
result = result << 8;
result = result | byte0; //(binary OR)
if (result & 0x8000 == 0x8000) { //sign extension
result = result | 0xFFFF0000;
}
if byte1 and byte0 are ints, you will need to make the `&0xFF
UPDATE because Java forces the expression of an if to be a boolean
do you have a way of finding a correct output for a given input?
technically, an int size is 4 bytes, so with just 2 bytes you can't reach the sign bit.
I ran across this same problem reading a MIDI file. A MIDI file has signed 16 bit as well as signed 32 bit integers. In a MIDI file, the most significant bytes come first (big-endian).
Here's what I did. It might be crude, but it maintains the sign. If the least significant bytes come first (little-endian), reverse the order of the indexes.
pos is the position in the byte array where the number starts.
length is the length of the integer, either 2 or 4. Yes, a 2 byte integer is a short, but we all work with ints.
private int convertBytes(byte[] number, int pos, int length) {
int output = 0;
if (length == 2) {
output = ((int) number[pos]) << 24;
output |= convertByte(number[pos + 1]) << 16;
output >>= 16;
} else if (length == 4) {
output = ((int) number[pos]) << 24;
output |= convertByte(number[pos + 1]) << 16;
output |= convertByte(number[pos + 2]) << 8;
output |= convertByte(number[pos + 3]);
}
return output;
}
private int convertByte(byte number) {
return (int) number & 0xff;
}