We Keep Coding

BitCount Method

Can anyone Explain this bitcount method.
public static int bitCount(int i) {
// Hacker's Delight, Figure 5-2
i -= (i >> 1) & 0x55555555;
i = (i & 0x33333333) + ((i >> 2) & 0x33333333);
i = ((i >> 4) + i) & 0x0F0F0F0F;
i += i >> 8;
i += i >> 16;
return i & 0x0000003F;
}

It's based on three observations,
The bitcount of a single bit is that bit itself.
The bitcount of the concatenation of two bitstrings is the sum of their bitcounts.
The bitcount of any string takes no more bits than that string itself.
The first two points together give you a simple recursive algorithm, split the string in two halves, recurse on both, return the sum. The base case is a single bit which you return. Simple so far.
The third observation is very important, it means that if you replace each substring with its bitcount, it will always fit in the space available to it. It means that if you give every count twice as much space (by separating the odd and the even groups), you can sum them and there will be no carry from one group into the next. Then you can rewrite it to this form:
i = (i & 0x55555555) + ((i >> 1) & 0x55555555); // sum groups of 1
i = (i & 0x33333333) + ((i >> 2) & 0x33333333); // sum groups of 2
i = (i & 0x0f0f0f0f) + ((i >> 4) & 0x0f0f0f0f); // sum groups of 4
...
And so on. What happens here is that on the left side of the + we take the even groups (0th, 2nd, 4th etc) and on right side we take the odd groups and align them with their corresponding even groups. For example, summing groups of 2:
[10 01 00 01 00 01 10 10] // note that 11 cannot occur
split
even: [0001 0001 0001 0010]
odd: [1000 0000 0000 1000]
align: [0010 0000 0000 0010]
sum: [0011 0001 0001 0100]
Then Hacker's Delight uses various tricks to optimize some operations out, for example groups of 4 can be summed using only the masking at the end, because the counts go up to 4 at most so summing them directly gives 8 at most, which still fits in the 4 bits available to it.

Why don't you add some logging code to display i in binary at each step and see if you can work out what's happening?
Or reduce it to a smaller numebr of bits (8, say) and work through it on paper.
It'll give you a much better feel for the code than someone simply explaining it to you.
This page may help.

This algorithm dates back at least to HAKMEM item 169
LDB B,[014300,,A] ;or MOVE B,A then LSH B,-1
AND B,[333333,,333333]
SUB A,B
LSH B,-1
AND B,[333333,,333333]
SUBB A,B ;each octal digit is replaced by number of 1's in it
LSH B,-3
ADD A,B
AND A,[070707,,070707]
IDIVI A,77 ;casting out 63.'s
These ten instructions, with constants extended, would work on word lengths up to 62.; eleven suffice up to 254..

Get n Least Significant Bits from an Int

This seems fairly straightforward, but I cant find an answer. If I have an int X, what is the best way to get N least significant bits from this int, in Java?

This should work for all non-negative N < 33 32:
x & ((1 << N) - 1)
It's worth elaborating on how this works for N == 31 and N == 32. For N == 31, we get 1 << N == Integer.MIN_VALUE. When you subtract 1 from that, Java silently wraps around to Integer.MAX_VALUE, which is exactly what you need. For N == 32, the 1 bit is shifted completely out, so 1 << N == 0; then (1 << N) - 1 == -1, which is all 32 bits set.
For N == 32, this unfortunately doesn't work because (thanks, #zstring!) the << operator only shifts by the right side mod 32. Instead, if you want to avoid testing for that case specially, you could use:
x & ((int)(1L << N) - 1)
By shifting a long, you get the full 32-bit shift, which, after casting back to an int, gets you 0. Subtracting 1 gives you -1 and x & -1 is just x for any int value x (and x is the value of the lower 32 bits of x).

Ted's approach is likely to be faster but here is another approach
x << -N >>> -N
This shift all the bit up and then down to chop off the top bits.
int i = -1;
System.out.println(Integer.toBinaryString(i));
i = i << -5 >>> -5;
System.out.println(Integer.toBinaryString(i));
prints
11111111111111111111111111111111
11111

You can also use a mask. If you use the & bitwise operator you can then remove whatever bit you would want to remove (say the highest x bits);
int mask = 0x7FFFFFFF //Example mask where you will remove the
// most significant bit
// (0x7 = 0111b and 0xF = 1111b).
int result = numberToProcess & mask; //And apply the mask with the &bitwise op.
The disadvantage to this is that you will need to make a mask for each bit, so perhaps this is better seen as another method of approach in general.

Conversion formula from decimal value to byte array not working

I have a formula in a specification for a binary file. The spec gives details of the meaning of the various bytes in the heading.
In particular, one formula states this about 2 of the bytes:
Byte 1 --> 7 6 5 4 3 2 1 0
Byte 2 --> 7 6 5 4 3 2 1 0
R Roll
If 'R' = 0, Roll Angle not available
If 'R' = 1, Roll Angle = [((Byte 84 & 0x7F)<<8) | (Byte 85) – 900] / 10
I need to take a value such as 4.3 and convert it to two bytes such that it will be able to be converted back to 4.3 using the above formula. The part that puzzles me the most is subtracting the 900.
This is what I have so far:
private byte[] getRollBytes(BigDecimal[] positionData) {
BigDecimal roll = positionData[4];
roll = roll.multiply(BigDecimal.TEN);
roll = roll.add(new BigDecimal(900));
short shortval = roll.shortValue();
byte[] rollBytes = new byte[2];
ByteBuffer headingbuf = ByteBuffer.wrap(rollBytes);
headingbuf.order(ByteOrder.BIG_ENDIAN);
headingbuf.putShort(shortval);
//set the leftmost bit of the two bytes to 'on', meaning data is available
rollBytes[0] = (byte) (rollBytes[0] | (0x80));
//testing the result with my formula doesn't give me the right answer:
float testFloat = (float) (((((rollBytes[0] & 0x7F) <<8) | rollBytes[1]) - 900) /10);
return rollBytes;
}
I think something is getting lost in the conversion between short and byte...

One problem is that you are casting to a short. This can't be correct, because a value like 4.3 is not an integer. You probably want to convert back to a BigDecimal:
BigDecimal testVal = ((rollBytes[0] & 0x7F) <<8) | rollBytes[1]) - 900;
testVal = val.divide(BigDecimal.valueOf(10));
Another problem is that you seem to be adding 900 twice—once when computing shortVal and once again with rollBytes[1] = (byte) (rollBytes[1] + 900);

use of the bitwise operators to pack multiple values in one int

Low level bit manipulation has never been my strong point. I will appreciate some help in understanding the following use case of bitwise operators.Consider...
int age, gender, height, packed_info;
. . . // Assign values
// Pack as AAAAAAA G HHHHHHH using shifts and "or"
packed_info = (age << 8) | (gender << 7) | height;
// Unpack with shifts and masking using "and"
height = packed_info & 0x7F; // This constant is binary ...01111111
gender = (packed_info >> 7) & 1;
age = (packed_info >> 8);
I am not sure what this code is accomplishing and how? Why use the magic number 0x7F ? How is the packing and unpacking accomplished?
Source

As the comment says, we're going to pack the age, gender and height into 15 bits, of the format:
AAAAAAAGHHHHHHH
Let's start with this part:
(age << 8)
To start with, age has this format:
age = 00000000AAAAAAA
where each A can be 0 or 1.
<< 8 moves the bits 8 places to the left, and fills in the gaps with zeroes. So you get:
(age << 8) = AAAAAAA00000000
Similarly:
gender = 00000000000000G
(gender << 7) = 0000000G0000000
height = 00000000HHHHHHH
Now we want to combine these into one variable. The | operator works by looking at each bit, and returning 1 if the bit is 1 in either of the inputs. So:
0011 | 0101 = 0111
If a bit is 0 in one input, then you get the bit from the other input. Looking at (age << 8), (gender << 7) and height, you'll see that, if a bit is 1 for one of these, it's 0 for the others. So:
packed_info = (age << 8) | (gender << 7) | height = AAAAAAAGHHHHHHH
Now we want to unpack the bits. Let's start with the height. We want to get the last 7 bits, and ignore the first 8. To do this, we use the & operator, which returns 1 only if both of the input bits are 1. So:
0011 & 0101 = 0001
So:
packed_info = AAAAAAAGHHHHHHH
0x7F = 000000001111111
(packed_info & 0x7F) = 00000000HHHHHHH = height
To get the age, we can just push everything 8 places to the right, and we're left with 0000000AAAAAAAA. So age = (packed_info >> 8).
Finally, to get the gender, we push everything 7 places to the right to get rid of the height. We then only care about the last bit:
packed_info = AAAAAAAGHHHHHHH
(packed_info >> 7) = 0000000AAAAAAAG
1 = 000000000000001
(packed_info >> 7) & 1 = 00000000000000G

This could be a rather long lesson in bit manipulation but first let me point you too the bit masking article on Wikipedia.
packed_info = (age << 8) | (gender << 7) | height;
Take age and move it's value over 8 bits then take gender and move it over 7 bits and height will occupy the last bits.
age = 0b101
gender = 0b1
height = 0b1100
packed_info = 0b10100000000
| 0b00010000000
| 0b00000001100
/* which is */
packed_info = 0b10110001100
Unpacking does the reverse but uses masks like 0x7F (which is 0b 01111111) to trim out the other values in the field.
gender = (packed_info >> 7) & 1;
Would work like...
gender = 0b1011 /* shifted 7 here but still has age on the other side */
& 0b0001
/* which is */
gender = 0b1
Note that ANDing anything to 1 is the same as "keeping" that bit and ANDing with 0 is the same as "ignoring" that bit.

If you were going to store a date as a number, maybe you would accomplish it by multiplying the year by 10000, the month by 100 and adding the day. A date such as July, 2, 2011 would be encoded as the number 20110702:
year * 10000 + month * 100 + day -> yyyymmdd
2011 * 10000 + 7 * 100 + 2 -> 20110702
We can say that we encoded the date in a yyyymmdd mask. We could describe this operation as
Shift the year 4 positions to the left,
shift the month 2 positions to the left and
leave the day as is.
Then combine the three values together.
This is the same thing that is happenning with the age, gender and height encoding, only that the author is thinking in binary.
See the ranges that those values may have:
age: 0 to 127 years
gender: M or F
height: 0 to 127 inches
If we translate those values to binary, we would have this:
age: 0 to 1111111b (7 binary digits, or bits)
gender: 0 or 1 (1 bit)
height: 0 to 1111111b (7 bits also)
With this in mind, we can encode the age-gender-height data with the mask aaaaaaaghhhhhhh, only that here we are talking about binary digits, not decimal digits.
So,
Shift the age 8 bits to the left,
shift the gender 7 bits to the left and
leave the height as is.
Then combine all three values together.
In binary, the Shift-Left operator (<<) moves a value n positions to the left. The "Or" operator ("|" in many languages) combines values together. Therefore:
(age << 8) | (gender << 7) | height
Now, how to "decode" those values?
It's easier in binary than in decimal:
You "mask away" the height,
shift the gender 7 bits to the right and mask that away also, and finally
shift the age 8 bits to the right.
The Shift-Right operator (>>) moves a value n positions to the right (whatever digits shifted "out" of the rightmost position are lost). The "And" binary operator ("&" in many languages) masks bits. To do that it needs a mask, indicating which bits to preserve and which bits to destroy (1 bits are preserved). Therefore:
height = value & 1111111b (preserve the 7 rightmost bits)
gender = (value >> 1) & 1 (preserve just one bit)
age = (value >> 8)
Since 1111111b in hex is 0x7f in most languages, that's the reason for that magic number. You would have the same effect by using 127 (which is 1111111b in decimal).

Same requirement I have faced many times. It is very easy with the help of Bitwise AND operator. Just qualify your values with increasing powers of two(2). To store multiple values, ADD their relative number ( power of 2 ) and get the SUM. This SUM will consolidate your selected values. HOW ?
Just do Bitwise AND with every value and it will give zero (0) for values which were not selected and non-zero for which are selected.
Here is the explanation:
1) Values ( YES, NO, MAYBE )
2) Assignment to power of two(2)
YES = 2^0 = 1 = 00000001
NO = 2^1 = 2 = 00000010
MAYBE = 2^2 = 4 = 00000100
3) I choose YES and MAYBE hence SUM:
SUM = 1 + 4 = 5
SUM = 00000001 + 00000100 = 00000101
This value will store both YES as well as MAYBE. HOW?
1 & 5 = 1 ( non zero )
2 & 5 = 0 ( zero )
4 & 5 = 4 ( non zero )
Hence SUM consists of
1 = 2^0 = YES
4 = 2^2 = MAYBE.
For more detailed explanation and implementation visit my blog

A more condense answer:
AAAAAAA G HHHHHHH
Packing:
packed = age << 8 | gender << 7 | height
Alternatively you can just sum components if ie when used in MySQL SUM aggregate function
packed = age << 8 + gender << 7 + height
Unpacking:
age = packed >> 8 // no mask required
gender = packed >> 7 & ((1 << 1) - 1) // applying mask (for gender it is just 1)
height = packed & ((1 << 7) - 1) // applying mask
Another (longer) example:
Say you have an IP address you want to pack, however it is a fictional IP address eg
132.513.151.319. Note that some components greater then 256 which requires more then 8 bits unlike real ip addresses.
First we need to figure out what offset we need to use to be able to store the max number.
Lets say with our fictional IPs no component can be bigger then 999 that means we need 10 bits of storage per component (allows numbers up to 1014).
packed = (comp1 << 0 * 10) | (comp1 << 1 * 10) | (comp1 << 2 * 10) | (comp1 << 3 * 10)
Which gives dec 342682502276 or bin 100111111001001011110000000010010000100
Now lets unpack the value
comp1 = (packed >> 0 * 10) & ((1 << 10) - 1) // 132
comp2 = (packed >> 1 * 10) & ((1 << 10) - 1) // 513
comp3 = (packed >> 2 * 10) & ((1 << 10) - 1) // 151
comp4 = (packed >> 3 * 10) & ((1 << 10) - 1) // 319
Where (1 << 10) - 1 is a binary mask we use to hide bits on the left beyond the 10 right most bits we are interested in.
Same example using MySQL query
SELECT
(#offset := 10) AS `No of bits required for each component`,
(#packed := (132 << 0 * #offset) |
(513 << 1 * #offset) |
(151 << 2 * #offset) |
(319 << 3 * #offset)) AS `Packed value (132.513.151.319)`,
BIN(#packed) AS `Packed value (bin)`,
(#packed >> 0 * #offset) & ((1 << #offset) - 1) `Component 1`,
(#packed >> 1 * #offset) & ((1 << #offset) - 1) `Component 2`,
(#packed >> 2 * #offset) & ((1 << #offset) - 1) `Component 3`,
(#packed >> 3 * #offset) & ((1 << #offset) - 1) `Component 4`;

The left shift operator means "multiply by two, this many times". In binary, multiplying a number by two is the same as adding a zero to the right side.
The right shift operator is the reverse of the left shift operator.
The pipe operator is "or", meaning overlay two binary numbers on top of each other, and where there is a 1 in either number the result in that column is a 1.
So, let's extract the operation for packed_info:
// Create age, shifted left 8 times:
// AAAAAAA00000000
age_shifted = age << 8;
// Create gender, shifted left 7 times:
// 0000000G0000000
gender_shifted = gender << 7;
// "Or" them all together:
// AAAAAAA00000000
// 0000000G0000000
// 00000000HHHHHHH
// ---------------
// AAAAAAAGHHHHHHH
packed_info = age_shifted | gender_shifted | height;
And the unpacking is the reverse.
// Grab the lowest 7 bits:
// AAAAAAAGHHHHHHH &
// 000000001111111 =
// 00000000HHHHHHH
height = packed_info & 0x7F;
// right shift the 'height' bits into the bit bucket, and grab the lowest 1 bit:
// AAAAAAAGHHHHHHH
// >> 7
// 0000000AAAAAAAG &
// 000000000000001 =
// 00000000000000G
gender = (packed_info >> 7) & 1;
// right shift the 'height' and 'gender' bits into the bit bucket, and grab the result:
// AAAAAAAGHHHHHHH
// >> 8
// 00000000AAAAAAA
age = (packed_info >> 8);

You can see the expression x & mask as an operation that removes from x the bits that are not present (i.e., have value 0) in mask. That means, packed_info & 0x7F removes from packed_info all the bits that are above the seventh bit.
Example: if packed_info is 1110010100101010 in binary, then packed_info & 0x7f will be
1110010100101010
0000000001111111
----------------
0000000000101010
So, in height we get the lower 7 bits of packed_info.
Next, we are shifting the whole packed_info by 7, this way we remove the information which we have already read out. So we get (for the value from previous example) 111001010 The gender is stored at the next bit, so with the same trick: & 1 we are extracting only that bit from the information. The rest of the information is contained at offset 8.
Packing back is not complicated, too: you take age, shift it 8 bits (so you get 1110010100000000 from 11100101), shift the gender by 7 (so you get 00000000), and take the height (assuming it would fit lower 7 bits). Then, you are composing all of them together:
1110010100000000
0000000000000000
0000000000101010
----------------
1110010100101010

Binary representation in Java

I am finding it difficult to understand and work with this binary representation in java:
With the help of the user Jon Skeet, I understood that binary representation should be built this way.
Here's a code sample:
public class chack {
public static void main(String[] args) {
int num2=2;
int num3=3;
int num4=4;
int num1=1;
int nirbinary = (num1 << 24) | (num2 << 16) | (num3 << 8) | num4;
System.out.println(nirbinary);
String nir= Integer.toBinaryString(nirbinary);
System.out.println(nir);
}
}
Couple of question:
How does one get num1 (for example) back from an int who is already in this binary
why do I get 16909060 when I print nirbinary- what does it stands for?
How does one get num1 (for example) back from an int who is already in this binary
representation?
Thank you

I am not completely sure what you are missing, so I will just explain how you can convert integers to binary strings back and forth in java.
You can get a binary string from an integer like so:
int i = 1234;
String binString = Integer.toBinaryString(i);
and you can convert the string back to an integer this way:
int iNew = Integer.parseInt(binString, 2);
Note the second argument to Integer.parseInt() is the desired base of the number. 2 is binary, 8 is octal, 10 decimal, etc.

16909060 stands for the number 16909060.
It is (1 * 224) + (2 * 216) + (3 * 28) + 4.
To get num1 back out, just right-shift the result the same amount you left-shifted and mask out the other bytes (not always necessary for num1(*), but for the others):
int num1 = nirbinary >> 24 & 0xFF;
int num2 = nirbinary >> 16 & 0xFF;
int num3 = nirbinary >> 8 & 0xFF;
int num4 = nirbinary & 0xFF;
Note that nirbinary is not "a binary representation". Or more precisely: it's no more or less binary than num1, num2, num3 and num4: internally all numbers (and characters, and booleans, ...) are stored in binary.
(*) note that if num1 is > 127, then you either need to use >>> to do the right-shift or use the & 0xFF in order to ensure that the correct value is restored. The difference between >> and >>> are the "new" bits inserted on the "left" side of the value: With >> they will depend on the highest-value bit (known as sign-extension) and with >>> they will always be 0.

Every int is a number, it's not binary, hex or decimal, it's just a number. the statement (num1 << 24) | (num2 << 16) | (num3 << 8) | num4; is a binary manipulation of 4 ints into another int. It doesn't change the representation of nirbinary to binary, since nirbinary has no representation, because (again) it's just a number.
Integer.toBinaryString(nirbinary) returns the binary representation of nirbinary which means "how would nibinary look like in base-2".
If you have a String which is a binary representation of a number, you could get its value, by using Integer.parseint(yourbinaryrepresentation, yourbase); for example - Integer.parseint(nir, 2);
And another thing:
You can't always get back one of the numbers back from nirbinary, since you performed a bit manipulation that is not reversible, for example:
int i1 = 5; //binary 0101
int i2 = 4; //binary 0100
int i3 = i1 | i2; //binary 0101
you cannot recognize each of your variables (i1, i2) since they have a common bit, i3 could have been the result of or on two other numbers:
int i1 = 1; //binary 0101
int i2 = 4; //binary 0100
int i3 = i1 | i2; //binary 0101
in your case, if each number is smaller than 256, you can reverse it with the following operation:
int myoldnumber = (nirbinary >> previousShift) & 0xff;
for example, to retrieve num1 you can do:
int retrievedNum1 = (nirbinary >> 24) & 0xff;

Here no need to depend only on binary or any other format...
one flexible built in function is available
That prints whichever format you want in your program..
Integer.toString(int,representation);
Integer.toString(100,8) // prints 144 --octal representation
Integer.toString(100,2) // prints 1100100 --binary representation
Integer.toString(100,16) //prints 64 --Hex representation
Integer.toString(100,5) // prints 400 --Base 5

When working with bitshifting and integers I would recommend you think in hexadecimal numbers, that will usually make life a lot easier. Just keep in mind that 8 bits represent 1 byte and 1 byte covers the hex-range from 0x00 to 0xFF
Since num1 to num4 are smaller than 10, their decimal representation is equal to their hex representiation, ie 1 = 0x01, 2 = 0x02 etc..
As I told you: 1 Byte is 8 bits. In your bitshifting operation you always shift multiple of 8.
So 0x01 << 8 => 0x0100
0x01 << 16 => 0x010000
etc.
So you basically only add zero bytes, which of course increases the value.
What you do next is to | them, a bitwise or. This means that two bitfields get modified in such a way that the result has a 1 at one place if at least one of the input values as a 1 there. Since your shifted ints contain only zero at the back, a bitwise or is nothing else then to put the value in this spot.
E.g:
(0x01 << 8) | 0x02
0x01 << 8 will produce 0x0100. Now you simply have to replace the last 00 with 02, since you or them: 0x0102
If you want to recreate the original int, you have to mask the part that int represents (this is easy since the parts do not overlap in your example) and then shift it back.
E.g.
Say ou produced 0x010203 and want to have only 0x02. You now have to mask shift it back 0x010203 >> 8 which will put the 02 in the last part. Now simply mask this last part 0x0102 && 0xFF. This will set all but the last 8 bits to zero

it's basically 1 * 2^24 + 2 * 2^16 + 3 * 2^8 + 4 = 16909060
You can get num1 by doing num1 = nirbinary >> 24.

What did you expect instead?
To get the most significant byte from an int i:
(i >> 24) & 0xff