I need to write a method which reads a variable number of bits from a ByteBuffer and converts those bytes into a primitive int or long (if more than 32 bits). I am not very proficient at working with bits, but I did look at some code examples and answers to similar(ish) questions and here is the best that I could come up with (bb refers to an internal ByteBuffer):
public int readBits(int number) {
int initialPosition = bb.position();
int value = 0;
int remaining = number;
int position = bb.position();
while (remaining > 0) {
int index = (position >> 3);
int bit = (position & 7);
int bitsLeft = Math.min(8 - bit, remaining);
int nibble = (bb.get(index) >> (bit & BIT_MASKS[bitsLeft]));
value |= nibble << (number - remaining);
position += bitsLeft;
remaining -= bitsLeft;
}
bb.position(initialPosition + ((position - initialPosition) >> 3));
return value;
}
The BIT_MASKS array is defined as following:
private static final int[] BIT_MASKS = new int[] { 0, 0x1, 0x3, 0x7, 0xf, 0x1f, 0x3f, 0x7f, 0xff };
I tested this method using this test code:
public static void main(String[] args) {
int nm = 4;
BitSet bs = new BitSet(nm);
bs.set(2);
bs.set(3);
BitBuffer bb = new BitBuffer(ByteBuffer.wrap(bs.toByteArray()));
System.out.println(bb.readBits(nm));
}
I expected a value of 6, but it prints 12 and I'm not sure why.
I would appreciate some help with this. Thanks.
Related
I am not getting the expected results when I tried to re-assemble the two bytes in Java (from C++ via JNI). Example below:
C++:
uint8_t originalValue = 0x38;
uint8_t high = originalValue & 0x0f0; //equals to 0x30
uint8_t low = originalValue << 4; //equals to 0x80
sendByte(high);
delayNs(1);
sendByte(low);
Java:
private int[] instructionRegister = new int[2];
private void processData(byte[] data) {
if (data.length <= 0)
return;
if (data.length == 1) {
int tmp = data[0];
String hexTmp = ByteUtils.toHexString(false, (byte) tmp).toUpperCase();
switch (tmp) {
case CMD_INSTRUCTION: {
log.info("\t>> INSTRUCTION: {}", hexTmp);
processingInstruction = true;
break;
}
case CMD_DATA: {
log.info("\t>> DATA: {}", hexTmp);
break;
}
default: {
log.info("\t\t- {}", hexTmp);
if (processingInstruction) {
if (instructionCount == 0) {
instructionRegister[0] = tmp;
instructionCount++;
} else {
instructionRegister[1] = tmp;
int combined = instructionRegister[0] | ((instructionRegister[1] >> 4);
log.info("\t>> Combined: ({}, {}) = {}",
ByteUtils.toHexString(false, (byte) instructionRegister[0]),
ByteUtils.toHexString(false, (byte) instructionRegister[1]),
ByteUtils.toHexString(false, (byte) combined));
instructionRegister = new int[2];
instructionCount = 0;
processingInstruction = false;
}
}
}
}
Result:
>> Combined: (30 , 80 ) = f8
I expected this to be 0x38 but got 0xF8. When I manually did the calculation in speedcrunch, I get the expected results.
Like I said in my comment, the problem is converting a signed byte holding 0x80 to an int (using a widening conversion), because the byte is negative, and the sign then gets extended, so you get ffffff80 in the int. If you then shift that to the right by 4 and convert back to a byte, you get: 0xf8, and if you binary-or that with 0x30 it's still 0xf8.
It looks like you're only doing such a conversion here:
int tmp = data[0];
You can change this line to:
int tmp = Byte.toUnsignedInt(data[0]);
To do a conversion as if the byte was unsigned.
To demonstrate the difference:
byte b = (byte) 0x80; // force overflow, making -128
int i1 = b;
int i2 = Byte.toUnsignedInt(b);
System.out.println(String.format("%x", b)); // 80
System.out.println(String.format("%x", i1)); // ffffff80
System.out.println(String.format("%x", i2)); // 80
I have been working on different projects for my Computer Organization class and we have been working on BitWise operations. Our current task is writing a homemade 'rotateLeft' method for java.
While java already has on, by using Integer.rotateLeft, my current task is writing one to work with this program.
Note: The int variables equal these bit strings
x1 = 3 = 00000000000000000000000000000011
x2 = -11= 1111111111111111111111111111110101
My current program is:
public class Demo
{
public static void main(String[]args)
{
int x1=3, x2=-11;
System.out.print("x1: ");
BitWise.printbit(x1);
System.out.print("rotateLeft(x1,2): ");
BitWise.printbit(rotateLeft(x1,2));
System.out.print("x2: ");
BitWise.printbit(x2);
System.out.print("rotateLeft(x2,2): ");
BitWise.printbit(rotateLeft(x2,2));
}
public static int rotateLeft(int i, int distance)
{
int mask= i>>distance;
return mask;
}
}
This operation works for the x1 bit pattern, however, it merely shifts the bits, not actually rotates them.
Any suggestions?
this works for me:
public static void main(String[] args) {
int x1 = 3;
int x2 = -11;
int x1IntegerRotated = Integer.rotateLeft(x1, 2);
int x1SelfRotated = rotateLeft(x1, 2);
System.out.printf("x1 = %d(%s)%n", x1, printIntBitwise(x1));
System.out.printf("x1IntegerRotated = %d(%s)%n", x1IntegerRotated, printIntBitwise(x1IntegerRotated));
System.out.printf("x1SelfRotated = %d(%s)%n", x1SelfRotated, printIntBitwise(x1SelfRotated));
System.out.println();
int x2IntegerRotated = Integer.rotateLeft(x2, 2);
int x2SelfRotated = rotateLeft(x2, 2);
System.out.printf("x2 = %d(%s)%n", x2, printIntBitwise(x2));
System.out.printf("x2IntegerRotated = %d(%s)%n", x2IntegerRotated, printIntBitwise(x2IntegerRotated));
System.out.printf("x2SelfRotated = %d(%s)%n", x2SelfRotated, printIntBitwise(x2SelfRotated));
}
private static int rotateLeft(int value, int distance) {
int mask = (1 << distance) - 1;
int leftPart = (value << distance) & (~mask);
int rightPart = (value >> (32 - distance)) & (mask);
int result = leftPart | rightPart;
return result;
}
private static String printIntBitwise(int a) {
StringBuilder sb = new StringBuilder();
for(int i = 1; i <= 32; i++) {
sb.append(Math.abs((a & (1 << (32 - i))) >> (32 - i)));
}
return sb.toString();
}
Ok, I have figured out a way of doing this:
//This is a helper function; it returns an int where the leftmost num bits are 1 and the rest are 0
static int get1s(int num) {
int buf = 0;
for (int i = 31; i>31-num;i--) {
buf += 1 << i;
}
return buf;
}
static int rotateLeft(int i, int distance) {
int end = i & get1s(distance);
int mov = end >>> 32 - distance;
int shift = i << distance;
return shift + mov;
}
Basically, the way that this works, line by line, is as follows:
Set end equal to just the leftmost distance bits, without actually shifting it.
Set mov equal to end shifted enough to make it the rightmost distance bits.
Set shift equal to the shifted value.
Return shift plus mov, thereby placing what were once the leftmost bits at the right.
If you want to see how this works in more detail, you can print the results after each step:
static int rotateLeftWithPrint(int i, int distance) {
int end = i & get1s(distance);
System.out.println(Integer.toBinaryString(end));
int mov = end >>> 32 - distance;
System.out.println(Integer.toBinaryString(mov));
int shift = i << distance;
System.out.println(Integer.toBinaryString(shift));
System.out.println(Integer.toBinaryString(shift+mov));
return shift + mov;
}
(NOTE) Integer.toBinaryString does not show leading zeroes, so that's why it won't always print Strings of the same length.
EDIT - Thought this might be useful, you can add binary literals by placing 0b before the digits; 3 is equivalent to 0b00000000000000000000000000000011.
How do I shift a byte array n positions to the right? For instance shifting a 16 byte array right 29 positions? I read somewhere it can be done using a long? Would using a long work like this:
Long k1 = byte array from 0 to 7
Long k2 = byte array from 8 to 15
Then right rotating these two longs using Long.rotateRight(Long x, number of rotations).How would the two longs be joined back into a byte array?
I believe you can do this using java.math.BigInteger which supports shifts on arbitrarily large numbers. This has advantage of simplicity, but disadvantage of not padding into original byte array size, i.e. input could be 16 bytes but output might only be 10 etc, requiring additional logic.
BigInteger approach
byte [] array = new byte[]{0x7F,0x11,0x22,0x33,0x44,0x55,0x66,0x77};
// create from array
BigInteger bigInt = new BigInteger(array);
// shift
BigInteger shiftInt = bigInt.shiftRight(4);
// back to array
byte [] shifted = shiftInt.toByteArray();
// print it as hex
for (byte b : shifted) {
System.out.print(String.format("%x", b));
}
Output
7f1122334455667 <== shifted 4 to the right. Looks OK
Long manipulation
I don't know why you'd want to do this as rotateRight() as this makes life more difficult, you have to blank at the bits that appear at the left hand side in K1 etc. You'd be better with using shift IMO as describe below. I've used a shift of 20 as divisible by 4 so easier to see the nibbles move in the output.
1) Use ByteBuffer to form two longs from 16 byte array
byte[] array = { 0x00, 0x00, 0x11, 0x11, 0x22, 0x22, 0x33, 0x33, 0x44, 0x44, 0x55, 0x55, 0x66, 0x66, 0x77, 0x77 };
ByteBuffer buffer = ByteBuffer.wrap(array);
long k1 = buffer.getLong();
long k2 = buffer.getLong();
2) Shift each long n bits to the right
int n = 20;
long k1Shift = k1 >> n;
long k2Shift = k2 >> n;
System.out.println(String.format("%016x => %016x", k1, k1Shift));
System.out.println(String.format("%016x => %016x", k2, k2Shift));
0000111122223333 => 0000000001111222
4444555566667777 => 0000044445555666
Determine bits from k1 that "got pushed off the edge"
long k1CarryBits = (k1 << (64 - n));
System.out.println(String.format("%016x => %016x", k1, k1CarryBits));
0000111122223333 => 2333300000000000
Join the K1 carry bits onto K2 on right hand side
long k2WithCarray = k2Shift | k1CarryBits;
System.out.println(String.format("%016x => %016x", k2Shift, k2WithCarray));
0000044445555666 => 2333344445555666
Write the two longs back into a ByteBuffer and extract as a byte array
buffer.position(0);
buffer.putLong(k1Shift);
buffer.putLong(k2WithCarray);
for (byte each : buffer.array()) {
System.out.print(Long.toHexString(each));
}
000011112222333344445555666
Here is what I came up with to shift a byte array by some arbitrary number of bits left:
/**
* Shifts input byte array len bits left.This method will alter the input byte array.
*/
public static byte[] shiftLeft(byte[] data, int len) {
int word_size = (len / 8) + 1;
int shift = len % 8;
byte carry_mask = (byte) ((1 << shift) - 1);
int offset = word_size - 1;
for (int i = 0; i < data.length; i++) {
int src_index = i+offset;
if (src_index >= data.length) {
data[i] = 0;
} else {
byte src = data[src_index];
byte dst = (byte) (src << shift);
if (src_index+1 < data.length) {
dst |= data[src_index+1] >>> (8-shift) & carry_mask;
}
data[i] = dst;
}
}
return data;
}
1. Manually implemented
Here are left and right shift implementation without using BigInteger (ie. without creating a copy of the input array) and with unsigned right shift (BigInteger only supports arithmetic shifts of course)
Left Shift <<
/**
* Left shift of whole byte array by shiftBitCount bits.
* This method will alter the input byte array.
*/
static byte[] shiftLeft(byte[] byteArray, int shiftBitCount) {
final int shiftMod = shiftBitCount % 8;
final byte carryMask = (byte) ((1 << shiftMod) - 1);
final int offsetBytes = (shiftBitCount / 8);
int sourceIndex;
for (int i = 0; i < byteArray.length; i++) {
sourceIndex = i + offsetBytes;
if (sourceIndex >= byteArray.length) {
byteArray[i] = 0;
} else {
byte src = byteArray[sourceIndex];
byte dst = (byte) (src << shiftMod);
if (sourceIndex + 1 < byteArray.length) {
dst |= byteArray[sourceIndex + 1] >>> (8 - shiftMod) & carryMask;
}
byteArray[i] = dst;
}
}
return byteArray;
}
Unsigned Right Shift >>>
/**
* Unsigned/logical right shift of whole byte array by shiftBitCount bits.
* This method will alter the input byte array.
*/
static byte[] shiftRight(byte[] byteArray, int shiftBitCount) {
final int shiftMod = shiftBitCount % 8;
final byte carryMask = (byte) (0xFF << (8 - shiftMod));
final int offsetBytes = (shiftBitCount / 8);
int sourceIndex;
for (int i = byteArray.length - 1; i >= 0; i--) {
sourceIndex = i - offsetBytes;
if (sourceIndex < 0) {
byteArray[i] = 0;
} else {
byte src = byteArray[sourceIndex];
byte dst = (byte) ((0xff & src) >>> shiftMod);
if (sourceIndex - 1 >= 0) {
dst |= byteArray[sourceIndex - 1] << (8 - shiftMod) & carryMask;
}
byteArray[i] = dst;
}
}
return byteArray;
}
Used in this class by this Project.
2. Using BigInteger
Be aware that BigInteger internally converts the byte array into an int[] array so this may not be the most optimized solution:
Arithmetic Left Shift <<:
byte[] result = new BigInteger(byteArray).shiftLeft(3).toByteArray();
Arithmetic Right Shift >>:
byte[] result = new BigInteger(byteArray).shiftRight(2).toByteArray();
3. External Library
Using the Bytes java library*:
Add to pom.xml:
<dependency>
<groupId>at.favre.lib</groupId>
<artifactId>bytes</artifactId>
<version>{latest-version}</version>
</dependency>
Code example:
Bytes b = Bytes.wrap(someByteArray);
b.leftShift(3);
b.rightShift(3);
byte[] result = b.array();
*Full Disclaimer: I am the developer.
The is an old post, but I want to update Adam's answer.
The long solution works with a few tweak.
In order to rotate, use >>> instead of >>, because >> will pad with significant bit, changing the original value.
second, the printbyte function seems to miss leading 00 when it prints.
use this instead.
private String getHexString(byte[] b) {
StringBuilder result = new StringBuilder();
for (int i = 0; i < b.length; i++)
result.append(Integer.toString((b[i] & 0xff) + 0x100, 16)
.substring(1));
return result.toString();
}
I found the code for adler32 here http://developer.classpath.org/doc/java/util/zip/Adler32-source.html
however my update code looks like following
private int a = 1, b = 0;
public void update(byte[] buf, int offset, int len)
{
for (int i = offset; i < len; i++) {
a = (a + buf[i]) % MOD_ADLER;
b = (b + a) % MOD_ADLER;
}
}
as oppose code on the link
public void update(byte[] buf, int offset, int len)
{
for (int i = offset; i < len; i++) {
a = (a + (buf[i] & 0xff)) % MOD_ADLER; // <<=== Why & 0xff ?
b = (b + a) % MOD_ADLER;
}
}
I do not understand need to AND with 0xff as buf[i] is already 8 bytes, I understand it will be promoted to int as a is int, still the type promotion shouldn't change value in byte. May be I'm missing some more details as it doesnt work without & 0xff, I tested it with the values computed by java.util.zip.Adler32
Thanks for the answer, however it will only matter for values that result in negative numbers, for example in my test
byte a = -4;
int n1 = a & 0xff;
int n2 = a;
out.printf(" a %4d %s\n", a,Integer.toBinaryString(a));
out.printf("n1 %4d %s\n",n1,Integer.toBinaryString(n1));
out.printf("n2 %4d %s\n",n2,Integer.toBinaryString(n2));
prints
a -4 11111111111111111111111111111100
n1 252 11111100
n2 -4 11111111111111111111111111111100
The problem is that byte is signed in Java. Hence, the automatic type promotion byte -> int would always give a integer in the range (-128,127) instead of (0,255), as we want. The & 0xff operation fixes that.
I am calculating the int equivalent of a given set of bits and storing that in memory. From there, I would like to determine all 1 value bits from the original bitmask. Example:
33 --> [1,6]
97 --> [1,6,7]
Ideas for an implementation in Java?
On BitSet
Use java.util.BitSet to store, well, a set of bits.
Here's how you can convert from an int to a BitSet, based on which bits in the int is set:
static BitSet fromInt(int num) {
BitSet bs = new BitSet();
for (int k = 0; k < Integer.SIZE; k++) {
if (((num >> k) & 1) == 1) {
bs.set(k);
}
}
return bs;
}
So now you can do the following:
System.out.println(fromInt(33)); // prints "{0, 5}"
System.out.println(fromInt(97)); // prints "{0, 5, 6}"
And just for completeness, here's the reverse transformation:
static int toInt(BitSet bs) {
int num = 0;
for (int k = -1; (k = bs.nextSetBit(k + 1)) != -1; ) {
num |= (1 << k);
}
return num;
}
So composing both together, we always get back the original number:
System.out.println(toInt(fromInt(33))); // prints "33"
System.out.println(toInt(fromInt(97))); // prints "97"
On 0-based indexing
Note that this uses 0-based indexing, which is the more commonly used indexing for bits (and most everything else in Java). This is also more correct. In the following, ^ denotes exponentiation:
33 = 2^0 + 2^5 = 1 + 32 97 = 2^0 + 2^5 + 2^6 = 1 + 32 + 64
33 -> {0, 5} 97 -> {0, 5, 6}
If you insist on using 1-based indexing, however, you can use bs.set(k+1); and (1 << (k-1)) in the above snippets. I would advise strongly against this recommendation, however.
Related questions
What does the ^ operator do in Java? -- it's actually not exponentiation
For bit fiddling, java.lang.Integer has some very helpful static methods. Try this code as a starting base for your problem:
public int[] extractBitNumbers(int value) {
// determine how many ones are in value
int bitCount = Integer.bitCount(value);
// allocate storage
int[] oneBits = new int[bitCount];
int putIndex = 0;
// loop until no more bits are set
while (value != 0) {
// find the number of the lowest set bit
int bitNo = Integer.numberOfTrailingZeros(value);
// store the bit number in array
oneBits[putIndex++] = bitNo+1;
// clear the bit we just processed from the value
value &= ~(1 << bitNo);
}
return oneBits;
}
I can show you C# implementation, Java should be very similar.
int value = 33;
int index = 1;
while (value > 0)
{
if ((value % 2) == 1)
Console.WriteLine(index);
index++;
value /= 2;
}
If you want to get an array like that you'll likely need to loop the number of bits you want to check & the integer with a bit shifted 1 for each step.
Something like (pseudo):
Init array
mask = 1
for (0 to BitCount):
if Integer & mask
array[] = pos
mask << 1
A bit-crunching variation would be something like:
int[] getBits(int value) {
int bitValue = 1;
int index = 1;
int[] bits = new int[33];
while (value >= bitValue)
{
bits[index++] = (value & bitValue);
bitValue << 1; // or: bitValue *= 2;
}
return bits;
}
Note that since the bits are indexed from 1 as you requested, bits[0] is left unused.