We Keep Coding

Java: IEEE Doubles to IBM Float

I am working on a side project at work where I would like to read/write SAS Transport files. The challenge is that numbers are encoded in 64-bit IBM floating point numbers. While I have been able to find plenty of great resources for reading a byte array (containing an IBM float) into a IEEE 32-bit floats and 64-bit floats, I'm struggling to find the code to convert floats/doubles back to IBM floats.
I recently found some code for writing a 32-bit IEEE float back out to a byte array (containing an IBM float). It seems to be working, so I've been trying to translate it to a 64-bit version. I've reversed engineered where most of the magic numbers are coming from, but I've been stumped for over a week now.
I have also tried to translate the functions listed at the end of the SAS Transport documentation to Java, but I've run into a lot of issues related to endiness, Java's lack of unsigned types, and so on. Can anyone provide the code to convert doubles to IBM floating point format?
Just to show the progress I've made, here are some shortened versions of the code I've written so far:
This grabs a 32-bit IBM float from a byte array and generates an IEEE float:
public static double fromIBMFloat(byte[] data, int offset) {
int temp = readIntFromBuffer(data, offset);
int mantissa = temp & 0x00FFFFFF;
int exponent = ((temp >> 24) & 0x7F) - 64;
boolean isNegative = (temp & 0x80000000) != 0;
double result = mantissa * Math.pow(2, 4 * exponent - 24);
if (isNegative) {
result = -result;
}
return result;
}
This is the same thing for 64-bit:
public static double fromIBMDouble(byte[] data, int offset) {
long temp = readLongFromBuffer(data, offset);
long mantissa = temp & 0x00FFFFFFFFFFFFFFL;
long exponent = ((temp >> 56) & 0x7F) - 64;
boolean isNegative = (temp & 0x8000000000000000L) != 0;
double result = mantissa * Math.pow(2, 4 * exponent - 24);
if (isNegative) {
result = -result;
}
return result;
}
Great! These work for going to IEEE floats, but now I need to go the other way. This simple implementation seems to be working for 32-bit floats:
public static void toIBMFloat(double value, byte[] xport, int offset) {
if (value == 0.0 || Double.isNaN(value) || Double.isInfinite(value)) {
writeIntToBuffer(xport, offset, 0);
return;
}
int fconv = Float.floatToIntBits((float)value);
int fmant = (fconv & 0x007FFFFF) | 0x00800000;
int temp = (fconv & 0x7F800000) >> 23;
int t = (temp & 0xFF) - 126;
while ((t & 0x3) != 0) {
++t;
fmant >>= 1;
}
fconv = (fconv & 0x80000000) | (((t >> 2) + 64) << 24) | fmant;
writeIntToBuffer(xport, offset, fconv);
}
Now, the only thing left is to translate that to work with 64-bit IBM floats. A lot of the magic numbers listed relate to the number of bits in the IEEE 32-bit floating point exponent (8-bits) and mantissa (23-bit). So for 64-bit, I just need to switch those to use the 11-bit exponent and 52-bit mantissa. But where does that 126 come from? What is the point of the 0x3 in the while loop?
Any help breaking down the 32-bit version so I can implement a 64-bit version would be greatly appreciated.

I circled back and took another swing at the C implementations provided at the end of the SAS transport documentation. It turns out the issue wasn't with my implementation; it was an issue with my tests.
TL;DR
These are my 64-bit implementations:
public static void writeIBMDouble(double value, byte[] data, int offset) {
long ieee8 = Double.doubleToLongBits(value);
long ieee1 = (ieee8 >>> 32) & 0xFFFFFFFFL;
long ieee2 = ieee8 & 0xFFFFFFFFL;
writeLong(0L, data, offset);
long xport1 = ieee1 & 0x000FFFFFL;
long xport2 = ieee2;
int ieee_exp = 0;
if (xport2 != 0 || ieee1 != 0) {
ieee_exp = (int)(((ieee1 >>> 16) & 0x7FF0) >>> 4) - 1023;
int shift = ieee_exp & 0x3;
xport1 |= 0x00100000L;
if (shift != 0) {
xport1 <<= shift;
xport1 |= ((byte)(((ieee2 >>> 24) & 0xE0) >>> (5 + (3 - shift))));
xport2 <<= shift;
}
xport1 |= (((ieee_exp >>> 2) + 65) | ((ieee1 >>> 24) & 0x80)) << 24;
}
if (-260 <= ieee_exp && ieee_exp <= 248) {
long temp = ((xport1 & 0xFFFFFFFFL) << 32) | (xport2 & 0xFFFFFFFFL);
writeLong(temp, data, offset);
return;
}
writeLong(0xFFFFFFFFFFFFFFFFL, data, offset);
if (ieee_exp > 248) {
data[offset] = 0x7F;
}
}
public static void writeLong(long value, byte[] buffer, int offset) {
buffer[offset] = (byte)(value >>> 56);
buffer[offset + 1] = (byte)(value >>> 48);
buffer[offset + 2] = (byte)(value >>> 40);
buffer[offset + 3] = (byte)(value >>> 32);
buffer[offset + 4] = (byte)(value >>> 24);
buffer[offset + 5] = (byte)(value >>> 16);
buffer[offset + 6] = (byte)(value >>> 8);
buffer[offset + 7] = (byte)value;
}
And:
public static double readIBMDouble(byte[] data, int offset) {
long temp = readLong(data, offset);
long ieee = 0L;
long xport1 = temp >>> 32;
long xport2 = temp & 0x00000000FFFFFFFFL;
long ieee1 = xport1 & 0x00ffffff;
long ieee2 = xport2;
if (ieee2 == 0L && xport1 == 0L) {
return Double.longBitsToDouble(ieee);
}
int shift = 0;
int nib = (int)xport1;
if ((nib & 0x00800000) != 0) {
shift = 3;
} else if ((nib & 0x00400000) != 0) {
shift = 2;
} else if ((nib & 0x00200000) != 0) {
shift = 1;
}
if (shift != 0) {
ieee1 >>>= shift;
ieee2 = (xport2 >>> shift) | ((xport1 & 0x00000007) << (29 + (3 - shift)));
}
ieee1 &= 0xffefffff;
ieee1 |= (((((long)(data[offset] & 0x7f) - 65) << 2) + shift + 1023) << 20) | (xport1 & 0x80000000);
ieee = ieee1 << 32 | ieee2;
return Double.longBitsToDouble(ieee);
}
public static long readLong(byte[] buffer, int offset) {
long result = unsignedByteToLong(buffer[offset]) << 56;
result |= unsignedByteToLong(buffer[offset + 1]) << 48;
result |= unsignedByteToLong(buffer[offset + 2]) << 40;
result |= unsignedByteToLong(buffer[offset + 3]) << 32;
result |= unsignedByteToLong(buffer[offset + 4]) << 24;
result |= unsignedByteToLong(buffer[offset + 5]) << 16;
result |= unsignedByteToLong(buffer[offset + 6]) << 8;
result |= unsignedByteToLong(buffer[offset + 7]);
return result;
}
private static long unsignedByteToLong(byte value) {
return (long)value & 0xFF;
}
These are basically a one-to-one translation from what's in the document, except I convert the byte[] into a long up-front and just do bit-twiddling instead of working directly with bytes.
I also realized the code in the documentation had some special cases included for "missing" values that are specific to the SAS transport standard and have nothing to do with IBM hexidecimal floating point numbers. In fact, the Double.longBitsToDouble method detects the invalid bit-sequence and just sets the value to NaN. I moved this code out since it wasn't going to work anyway.
The good thing is that as part of this exercise I did learn a lot of tricks to bit manipulation in Java. For instance, a lot of the issues I ran into involving sign were resolved by using the >>> operator instead of the >> operator. Other than that, you just need to be careful upcasting to mask with 0xFF, 0xFFFF, etc. to make sure the sign is ignored.
I also learned about ByteBuffer which can facilitate loading back and forth among byte[] and primitives/strings; however, that comes with some minor overhead. But it would handle any endianness issues. It turns out endianness wasn't even a concern since most architectures in use today (x86) are little endian to begin with.
It seems reading/writing SAS transport files is a pretty common need, especially in the clinical trials arena so hopefully anyone working in Java/C# won't have to go through the trouble I did.

Python bitshifting to java

I found a python code on github, and I need to do the same thing in java, I have almost converted it to java but I'm getting an warning saying Shift operation '>>' by overly large constant value
this is the python code that I'm trying to convert
if i > 32:
return (int(((j >> 32) & ((1 << i))))) >> i
return (int((((1 << i)) & j))) >> i
and this is the java code I made trying to convert from the python code
if (i > 32) {
return (j >> 32) & (1 << i) >> i;
}
return ((1 << i) & j) >> i;
the warning is in this line (j >> 32)

Since Java's int is 32 bits (See here), shifting it 32 bits to the right leaves nothing from the original int, and therefore doesn't make much sense

This doesn't really make sense to me because shifting 32 bits in an int leaves nothing, How ever if you want to implant the same method using long, here is the code I wrote to do so.
public int bitShift(long j, int i) {
return i > 32 ? ((int) ((j >> 32) & ((long) (1 << i)))) >> i : ((int) (j & ((long) (1 << i)))) >> i;
}

Hex to Bytes, ruby and java

I have the following code to convert hex string to bytes in Java:
String s = "longhex";
int len = s.length();
byte[] data = new byte[(len / 2)];
for (int i = 0; i < len; i += 2)
{
data[i / 2] = (byte) ((Character.digit(s.charAt(i), 16) << 4) + Character.digit(s.charAt(i + 1), 16));
}
is this a correct way to reproduce it in ruby?
s = "longhex"
bytes = []
(0..s.length / 2 - 1).step(2).each do |i|
bytes[i / 2] = s[i].ord << 4 + s[i + 1].ord
end

No it is not correct. << has a lower operator precedence than +. Note even in java there are parentheses around shift operator. Also, it’s not ruby, it’s c written with an almost ruby syntax.
str.codepoints.
each_slice(2).
map { |f, l| (f << 4) + l }
would probably do what you want, but without seeing an expected outcome it’s hard to say.
Correct version as by Ilya is:
str.scan(/.{1}/).
each_slice(2).
map { |f, l| (Integer(f,16) << 4) + Integer(l,16) }

Reverse all bits in an int and return the int

What integer should be returned when we reverse all bits of integer 1? How do we do that with Java code?
No java built in functions should be used. Shouldn't use String reverse, converting to string etc. Only bitwise operations allowed.
import java.util.*;
import java.lang.*;
import java.io.*;
class BitReverseInt
{
public static void main (String[] args) throws java.lang.Exception{
System.out.println(reverser(1));
}
public static int reverser(int given){
int input = given;
int temp = 0;
int output = 0;
while(input > 0){
output = output << 1;
temp = input & 1;
input = input >> 1;
output = output | temp;
}
return output;
}
}

Bit reversal can be done by interchanging adjacent single bits, then interchanging adjacent 2-bit fields, then 4-bits, and so on as shown below. These five assignment statements can be executed in any order.
/********************************************************
* These are the bit masks used in the bit reversal process
0x55555555 = 01010101010101010101010101010101
0xAAAAAAAA = 10101010101010101010101010101010
0x33333333 = 00110011001100110011001100110011
0xCCCCCCCC = 11001100110011001100110011001100
0x0F0F0F0F = 00001111000011110000111100001111
0xF0F0F0F0 = 11110000111100001111000011110000
0x00FF00FF = 00000000111111110000000011111111
0xFF00FF00 = 11111111000000001111111100000000
0x0000FFFF = 00000000000000001111111111111111
0xFFFF0000 = 11111111111111110000000000000000
*/
uint x = 23885963; // 00000001011011000111100010001011
x = (x & 0x55555555) << 1 | (x & 0xAAAAAAAA) >> 1;
x = (x & 0x33333333) << 2 | (x & 0xCCCCCCCC) >> 2;
x = (x & 0x0F0F0F0F) << 4 | (x & 0xF0F0F0F0) >> 4;
x = (x & 0x00FF00FF) << 8 | (x & 0xFF00FF00) >> 8;
x = (x & 0x0000FFFF) << 16 | (x & 0xFFFF0000) >> 16;
// result x == 3508418176 11010001000111100011011010000000
By looking at each intermediary result you can see what is happening.
Hopefully this will give you what you need to sort it out in your head. John Doe's answer consolidates steps 4 and 5 in to a single expression. This will work on most machines.

Here is the actual implementation of Integer.reverse(int).
public static int reverse(int i) {
// HD, Figure 7-1
i = (i & 0x55555555) << 1 | (i >>> 1) & 0x55555555;
i = (i & 0x33333333) << 2 | (i >>> 2) & 0x33333333;
i = (i & 0x0f0f0f0f) << 4 | (i >>> 4) & 0x0f0f0f0f;
i = (i << 24) | ((i & 0xff00) << 8) |
((i >>> 8) & 0xff00) | (i >>> 24);
return i;
}

You can use a do while loop like this:
public static int reverse(int number){
int reverse = 0;
int remainder = 0;
do{
remainder = number%10;
reverse = reverse*10 + remainder;
number = number/10;
}while(number > 0);
return reverse;
}
And for bitwise operation: here it goes:
// value=your integer, numBitsInt=how much bit you will use to reverse
public static int reverseIntBitwise(int value, int numBitsInt) {
int i = 0, rev = 0, bit;
while (i++ < numBitsInt) {
bit = value & 1;
value = value >> 1;
rev = rev ^ bit;
if (i < numBitsInt)
rev = rev << 1;
}
return rev;
}

Well there are multiple ways to reverse the bits of the given number in Java.
First, Java language has inbuild bitwise complement operator(~). So (~number) reverses the bits of number.
Second, one can use the Integer.reverse(number)
Third, if this is a part of test or you just want to play with bits, you can refer the code below.
/*
The logic uses moving bitmask from right to left:
1. Get the bit of given number, by binary and(&) with bitmask
2. XOR(^) with the bitmask, so here we reverse the bit.
3. OR(|) this reversed bit with the result(result has all Zero initially)
This logic is repeated for each 32 bits by moving the mask from right to left,
one bit at a time using (<<) left shift operator on bitmask.
*/
public class ReverseBits {
public static int reverseBits(int input) {
print("Input", input);
int bitmask = 1;
int result = 0;
do {
//print("Bitmask", bitmask);
result = result | (bitmask ^ (input & bitmask)) ;
//print("Result", result);
bitmask = bitmask << 1;
} while (bitmask != 0);
print("Reverse", result);
return result;
}
public static void print(String label, int input) {
System.out.println(label +"\t:"+Integer.toBinaryString(input));
}
public static void main(String[] args) {
reverseBits(Integer.MIN_VALUE);
reverseBits(Integer.MAX_VALUE);
reverseBits(reverseBits(170));
}
}
Output:
Input :10000000000000000000000000000000
Reverse :1111111111111111111111111111111
Input :1111111111111111111111111111111
Reverse :10000000000000000000000000000000
Input :10101010
Reverse :11111111111111111111111101010101
Input :11111111111111111111111101010101
Reverse :10101010

return Integer.reverse(given);
Integer.reverse Reference

I need to convert C code block, which calculate checksum to Java code. I have no idea how to do this (

I have an electronic board on which I am printing the data. To calculate the checksum byte of output are using the following algorithm:
word CountCS(byte *Buff, byte Cnt) //Cnt - count of bytes
{
dword cs1, cs2, m;
byte k;
k=0;
cs1=0;
cs2=0;
while (Cnt) {
m=*Buff;
cs1 +=m;
m ^=0x5A;
if (k) m=(m >> k) | (m << (8-k));
k=(k+1) & 0x07;
cs2 +=m;
Buff++;
Cnt--;
};
return (cs2<<8) | (cs1 & 0xFF);
}
The manufacturer of scoreboard provide code in C, I need to convert it to java. I've try to convert, but result is wrong, here java code
public int checksum(int [] buffer, int count) {
int cs1 =0, cs2 = 0, m;
int k = 0, i = 0;
while (count != 0) {
m = buffer[i];
cs1 += m;
m ^= 0x5a;
if (k != 0) m = (m >> k) | (m << (8-k));
k = (k +1) & 0x07;
cs2 += m;
i++;
count--;
}
return (cs2 << 8) | (cs1 & 0xff);
}
Can anyone see what I'm doing wrong?
Just sorry, but I'm new to java.
Thanks a lot, problem is solved

if (k) m=(m >> k) | (m << (8-k));
should be
if (k != 0) m = (m >> k) | (m << (8-k));
not
if (k == 0) m = (m >> k) | (m << (8-k));

Your variable sizes are different: k (byte vs int). In java int is 32 bit signed. byte in C++ may be dependent on platform, more likely 8 or 16 bits unsigned. So it would have different values as you add and multiply/shift, etc.