We Keep Coding

The best performance implementation of counting the number of true of an integer? [duplicate]

8 bits representing the number 7 look like this:
00000111
Three bits are set.
What are the algorithms to determine the number of set bits in a 32-bit integer?

This is known as the 'Hamming Weight', 'popcount' or 'sideways addition'.
Some CPUs have a single built-in instruction to do it and others have parallel instructions which act on bit vectors. Instructions like x86's popcnt (on CPUs where it's supported) will almost certainly be fastest for a single integer. Some other architectures may have a slow instruction implemented with a microcoded loop that tests a bit per cycle (citation needed - hardware popcount is normally fast if it exists at all.).
The 'best' algorithm really depends on which CPU you are on and what your usage pattern is.
Your compiler may know how to do something that's good for the specific CPU you're compiling for, e.g. C++20 std::popcount(), or C++ std::bitset<32>::count(), as a portable way to access builtin / intrinsic functions (see another answer on this question). But your compiler's choice of fallback for target CPUs that don't have hardware popcnt might not be optimal for your use-case. Or your language (e.g. C) might not expose any portable function that could use a CPU-specific popcount when there is one.
Portable algorithms that don't need (or benefit from) any HW support
A pre-populated table lookup method can be very fast if your CPU has a large cache and you are doing lots of these operations in a tight loop. However it can suffer because of the expense of a 'cache miss', where the CPU has to fetch some of the table from main memory. (Look up each byte separately to keep the table small.) If you want popcount for a contiguous range of numbers, only the low byte is changing for groups of 256 numbers, making this very good.
If you know that your bytes will be mostly 0's or mostly 1's then there are efficient algorithms for these scenarios, e.g. clearing the lowest set with a bithack in a loop until it becomes zero.
I believe a very good general purpose algorithm is the following, known as 'parallel' or 'variable-precision SWAR algorithm'. I have expressed this in a C-like pseudo language, you may need to adjust it to work for a particular language (e.g. using uint32_t for C++ and >>> in Java):
GCC10 and clang 10.0 can recognize this pattern / idiom and compile it to a hardware popcnt or equivalent instruction when available, giving you the best of both worlds. (https://godbolt.org/z/qGdh1dvKK)
int numberOfSetBits(uint32_t i)
{
// Java: use int, and use >>> instead of >>. Or use Integer.bitCount()
// C or C++: use uint32_t
i = i - ((i >> 1) & 0x55555555); // add pairs of bits
i = (i & 0x33333333) + ((i >> 2) & 0x33333333); // quads
i = (i + (i >> 4)) & 0x0F0F0F0F; // groups of 8
return (i * 0x01010101) >> 24; // horizontal sum of bytes
}
For JavaScript: coerce to integer with |0 for performance: change the first line to i = (i|0) - ((i >> 1) & 0x55555555);
This has the best worst-case behaviour of any of the algorithms discussed, so will efficiently deal with any usage pattern or values you throw at it. (Its performance is not data-dependent on normal CPUs where all integer operations including multiply are constant-time. It doesn't get any faster with "simple" inputs, but it's still pretty decent.)
References:
https://graphics.stanford.edu/~seander/bithacks.html
https://catonmat.net/low-level-bit-hacks for bithack basics, like how subtracting 1 flips contiguous zeros.
https://en.wikipedia.org/wiki/Hamming_weight
http://gurmeet.net/puzzles/fast-bit-counting-routines/
http://aggregate.ee.engr.uky.edu/MAGIC/#Population%20Count%20(Ones%20Count)
How this SWAR bithack works:
i = i - ((i >> 1) & 0x55555555);
The first step is an optimized version of masking to isolate the odd / even bits, shifting to line them up, and adding. This effectively does 16 separate additions in 2-bit accumulators (SWAR = SIMD Within A Register). Like (i & 0x55555555) + ((i>>1) & 0x55555555).
The next step takes the odd/even eight of those 16x 2-bit accumulators and adds again, producing 8x 4-bit sums. The i - ... optimization isn't possible this time so it does just mask before / after shifting. Using the same 0x33... constant both times instead of 0xccc... before shifting is a good thing when compiling for ISAs that need to construct 32-bit constants in registers separately.
The final shift-and-add step of (i + (i >> 4)) & 0x0F0F0F0F widens to 4x 8-bit accumulators. It masks after adding instead of before, because the maximum value in any 4-bit accumulator is 4, if all 4 bits of the corresponding input bits were set. 4+4 = 8 which still fits in 4 bits, so carry between nibble elements is impossible in i + (i >> 4).
So far this is just fairly normal SIMD using SWAR techniques with a few clever optimizations. Continuing on with the same pattern for 2 more steps can widen to 2x 16-bit then 1x 32-bit counts. But there is a more efficient way on machines with fast hardware multiply:
Once we have few enough "elements", a multiply with a magic constant can sum all the elements into the top element. In this case byte elements. Multiply is done by left-shifting and adding, so a multiply of x * 0x01010101 results in x + (x<<8) + (x<<16) + (x<<24). Our 8-bit elements are wide enough (and holding small enough counts) that this doesn't produce carry into that top 8 bits.
A 64-bit version of this can do 8x 8-bit elements in a 64-bit integer with a 0x0101010101010101 multiplier, and extract the high byte with >>56. So it doesn't take any extra steps, just wider constants. This is what GCC uses for __builtin_popcountll on x86 systems when the hardware popcnt instruction isn't enabled. If you can use builtins or intrinsics for this, do so to give the compiler a chance to do target-specific optimizations.
With full SIMD for wider vectors (e.g. counting a whole array)
This bitwise-SWAR algorithm could parallelize to be done in multiple vector elements at once, instead of in a single integer register, for a speedup on CPUs with SIMD but no usable popcount instruction. (e.g. x86-64 code that has to run on any CPU, not just Nehalem or later.)
However, the best way to use vector instructions for popcount is usually by using a variable-shuffle to do a table-lookup for 4 bits at a time of each byte in parallel. (The 4 bits index a 16 entry table held in a vector register).
On Intel CPUs, the hardware 64bit popcnt instruction can outperform an SSSE3 PSHUFB bit-parallel implementation by about a factor of 2, but only if your compiler gets it just right. Otherwise SSE can come out significantly ahead. Newer compiler versions are aware of the popcnt false dependency problem on Intel.
https://github.com/WojciechMula/sse-popcount state-of-the-art x86 SIMD popcount for SSSE3, AVX2, AVX512BW, AVX512VBMI, or AVX512 VPOPCNT. Using Harley-Seal across vectors to defer popcount within an element. (Also ARM NEON)
Counting 1 bits (population count) on large data using AVX-512 or AVX-2
related: https://github.com/mklarqvist/positional-popcount - separate counts for each bit-position of multiple 8, 16, 32, or 64-bit integers. (Again, x86 SIMD including AVX-512 which is really good at this, with vpternlogd making Harley-Seal very good.)

Some languages portably expose the operation in a way that can use efficient hardware support if available, otherwise some library fallback that's hopefully decent.
For example (from a table by language):
C++ has std::bitset<>::count(), or C++20 std::popcount(T x)
Java has java.lang.Integer.bitCount() (also for Long or BigInteger)
C# has System.Numerics.BitOperations.PopCount()
Python has int.bit_count() (since 3.10)
Not all compilers / libraries actually manage to use HW support when it's available, though. (Notably MSVC, even with options that make std::popcount inline as x86 popcnt, its std::bitset::count still always uses a lookup table. This will hopefully change in future versions.)
Also consider the built-in functions of your compiler when the portable language doesn't have this basic bit operation. In GNU C for example:
int __builtin_popcount (unsigned int x);
int __builtin_popcountll (unsigned long long x);
In the worst case (no single-instruction HW support) the compiler will generate a call to a function (which in current GCC uses a shift/and bit-hack like this answer, at least for x86). In the best case the compiler will emit a cpu instruction to do the job. (Just like a * or / operator - GCC will use a hardware multiply or divide instruction if available, otherwise will call a libgcc helper function.) Or even better, if the operand is a compile-time constant after inlining, it can do constant-propagation to get a compile-time-constant popcount result.
The GCC builtins even work across multiple platforms. Popcount has almost become mainstream in the x86 architecture, so it makes sense to start using the builtin now so you can recompile to let it inline a hardware instruction when you compile with -mpopcnt or something that includes that (e.g. https://godbolt.org/z/Ma5e5a). Other architectures have had popcount for years, but in the x86 world there are still some ancient Core 2 and similar vintage AMD CPUs in use.
On x86, you can tell the compiler that it can assume support for popcnt instruction with -mpopcnt (also implied by -msse4.2). See GCC x86 options. -march=nehalem -mtune=skylake (or -march= whatever CPU you want your code to assume and to tune for) could be a good choice. Running the resulting binary on an older CPU will result in an illegal-instruction fault.
To make binaries optimized for the machine you build them on, use -march=native (with gcc, clang, or ICC).
MSVC provides an intrinsic for the x86 popcnt instruction, but unlike gcc it's really an intrinsic for the hardware instruction and requires hardware support.
Using std::bitset<>::count() instead of a built-in
In theory, any compiler that knows how to popcount efficiently for the target CPU should expose that functionality through ISO C++ std::bitset<>. In practice, you might be better off with the bit-hack AND/shift/ADD in some cases for some target CPUs.
For target architectures where hardware popcount is an optional extension (like x86), not all compilers have a std::bitset that takes advantage of it when available. For example, MSVC has no way to enable popcnt support at compile time, and it's std::bitset<>::count always uses a table lookup, even with /Ox /arch:AVX (which implies SSE4.2, which in turn implies the popcnt feature.) (Update: see below; that does get MSVC's C++20 std::popcount to use x86 popcnt, but still not its bitset<>::count. MSVC could fix that by updating their standard library headers to use std::popcount when available.)
But at least you get something portable that works everywhere, and with gcc/clang with the right target options, you get hardware popcount for architectures that support it.
#include <bitset>
#include <limits>
#include <type_traits>
template<typename T>
//static inline // static if you want to compile with -mpopcnt in one compilation unit but not others
typename std::enable_if<std::is_integral<T>::value, unsigned >::type
popcount(T x)
{
static_assert(std::numeric_limits<T>::radix == 2, "non-binary type");
// sizeof(x)*CHAR_BIT
constexpr int bitwidth = std::numeric_limits<T>::digits + std::numeric_limits<T>::is_signed;
// std::bitset constructor was only unsigned long before C++11. Beware if porting to C++03
static_assert(bitwidth <= std::numeric_limits<unsigned long long>::digits, "arg too wide for std::bitset() constructor");
typedef typename std::make_unsigned<T>::type UT; // probably not needed, bitset width chops after sign-extension
std::bitset<bitwidth> bs( static_cast<UT>(x) );
return bs.count();
}
See asm from gcc, clang, icc, and MSVC on the Godbolt compiler explorer.
x86-64 gcc -O3 -std=gnu++11 -mpopcnt emits this:
unsigned test_short(short a) { return popcount(a); }
movzx eax, di # note zero-extension, not sign-extension
popcnt rax, rax
ret
unsigned test_int(int a) { return popcount(a); }
mov eax, edi
popcnt rax, rax # unnecessary 64-bit operand size
ret
unsigned test_u64(unsigned long long a) { return popcount(a); }
xor eax, eax # gcc avoids false dependencies for Intel CPUs
popcnt rax, rdi
ret
PowerPC64 gcc -O3 -std=gnu++11 emits (for the int arg version):
rldicl 3,3,0,32 # zero-extend from 32 to 64-bit
popcntd 3,3 # popcount
blr
This source isn't x86-specific or GNU-specific at all, but only compiles well with gcc/clang/icc, at least when targeting x86 (including x86-64).
Also note that gcc's fallback for architectures without single-instruction popcount is a byte-at-a-time table lookup. This isn't wonderful for ARM, for example.
C++20 has std::popcount(T)
Current libstdc++ headers unfortunately define it with a special case if(x==0) return 0; at the start, which clang doesn't optimize away when compiling for x86:
#include <bit>
int bar(unsigned x) {
return std::popcount(x);
}
clang 11.0.1 -O3 -std=gnu++20 -march=nehalem (https://godbolt.org/z/arMe5a)
# clang 11
bar(unsigned int): # #bar(unsigned int)
popcnt eax, edi
cmove eax, edi # redundant: if popcnt result is 0, return the original 0 instead of the popcnt-generated 0...
ret
But GCC compiles nicely:
# gcc 10
xor eax, eax # break false dependency on Intel SnB-family before Ice Lake.
popcnt eax, edi
ret
Even MSVC does well with it, as long as you use -arch:AVX or later (and enable C++20 with -std:c++latest). https://godbolt.org/z/7K4Gef
int bar(unsigned int) PROC ; bar, COMDAT
popcnt eax, ecx
ret 0
int bar(unsigned int) ENDP ; bar

In my opinion, the "best" solution is the one that can be read by another programmer (or the original programmer two years later) without copious comments. You may well want the fastest or cleverest solution which some have already provided but I prefer readability over cleverness any time.
unsigned int bitCount (unsigned int value) {
unsigned int count = 0;
while (value > 0) { // until all bits are zero
if ((value & 1) == 1) // check lower bit
count++;
value >>= 1; // shift bits, removing lower bit
}
return count;
}
If you want more speed (and assuming you document it well to help out your successors), you could use a table lookup:
// Lookup table for fast calculation of bits set in 8-bit unsigned char.
static unsigned char oneBitsInUChar[] = {
// 0 1 2 3 4 5 6 7 8 9 A B C D E F (<- n)
// =====================================================
0, 1, 1, 2, 1, 2, 2, 3, 1, 2, 2, 3, 2, 3, 3, 4, // 0n
1, 2, 2, 3, 2, 3, 3, 4, 2, 3, 3, 4, 3, 4, 4, 5, // 1n
: : :
4, 5, 5, 6, 5, 6, 6, 7, 5, 6, 6, 7, 6, 7, 7, 8, // Fn
};
// Function for fast calculation of bits set in 16-bit unsigned short.
unsigned char oneBitsInUShort (unsigned short x) {
return oneBitsInUChar [x >> 8]
+ oneBitsInUChar [x & 0xff];
}
// Function for fast calculation of bits set in 32-bit unsigned int.
unsigned char oneBitsInUInt (unsigned int x) {
return oneBitsInUShort (x >> 16)
+ oneBitsInUShort (x & 0xffff);
}
These rely on specific data type sizes so they're not that portable. But, since many performance optimisations aren't portable anyway, that may not be an issue. If you want portability, I'd stick to the readable solution.

From Hacker's Delight, p. 66, Figure 5-2
int pop(unsigned x)
{
x = x - ((x >> 1) & 0x55555555);
x = (x & 0x33333333) + ((x >> 2) & 0x33333333);
x = (x + (x >> 4)) & 0x0F0F0F0F;
x = x + (x >> 8);
x = x + (x >> 16);
return x & 0x0000003F;
}
Executes in ~20-ish instructions (arch dependent), no branching.Hacker's Delight is delightful! Highly recommended.

I think the fastest way—without using lookup tables and popcount—is the following. It counts the set bits with just 12 operations.
int popcount(int v) {
v = v - ((v >> 1) & 0x55555555); // put count of each 2 bits into those 2 bits
v = (v & 0x33333333) + ((v >> 2) & 0x33333333); // put count of each 4 bits into those 4 bits
return ((v + (v >> 4) & 0xF0F0F0F) * 0x1010101) >> 24;
}
It works because you can count the total number of set bits by dividing in two halves, counting the number of set bits in both halves and then adding them up. Also know as Divide and Conquer paradigm. Let's get into detail..
v = v - ((v >> 1) & 0x55555555);
The number of bits in two bits can be 0b00, 0b01 or 0b10. Lets try to work this out on 2 bits..
---------------------------------------------
| v | (v >> 1) & 0b0101 | v - x |
---------------------------------------------
0b00 0b00 0b00
0b01 0b00 0b01
0b10 0b01 0b01
0b11 0b01 0b10
This is what was required: the last column shows the count of set bits in every two bit pair. If the two bit number is >= 2 (0b10) then and produces 0b01, else it produces 0b00.
v = (v & 0x33333333) + ((v >> 2) & 0x33333333);
This statement should be easy to understand. After the first operation we have the count of set bits in every two bits, now we sum up that count in every 4 bits.
v & 0b00110011 //masks out even two bits
(v >> 2) & 0b00110011 // masks out odd two bits
We then sum up the above result, giving us the total count of set bits in 4 bits. The last statement is the most tricky.
c = ((v + (v >> 4) & 0xF0F0F0F) * 0x1010101) >> 24;
Let's break it down further...
v + (v >> 4)
It's similar to the second statement; we are counting the set bits in groups of 4 instead. We know—because of our previous operations—that every nibble has the count of set bits in it. Let's look an example. Suppose we have the byte 0b01000010. It means the first nibble has its 4bits set and the second one has its 2bits set. Now we add those nibbles together.
v = 0b01000010
(v >> 4) = 0b00000100
v + (v >> 4) = 0b01000010 + 0b00000100
It gives us the count of set bits in a byte, in the second nibble 0b01000110 and therefore we mask the first four bytes of all the bytes in the number (discarding them).
0b01000110 & 0x0F = 0b00000110
Now every byte has the count of set bits in it. We need to add them up all together. The trick is to multiply the result by 0b10101010 which has an interesting property. If our number has four bytes, A B C D, it will result in a new number with these bytes A+B+C+D B+C+D C+D D. A 4 byte number can have maximum of 32 bits set, which can be represented as 0b00100000.
All we need now is the first byte which has the sum of all set bits in all the bytes, and we get it by >> 24. This algorithm was designed for 32 bit words but can be easily modified for 64 bit words.

If you happen to be using Java, the built-in method Integer.bitCount will do that.

I got bored, and timed a billion iterations of three approaches. Compiler is gcc -O3. CPU is whatever they put in the 1st gen Macbook Pro.
Fastest is the following, at 3.7 seconds:
static unsigned char wordbits[65536] = { bitcounts of ints between 0 and 65535 };
static int popcount( unsigned int i )
{
return( wordbits[i&0xFFFF] + wordbits[i>>16] );
}
Second place goes to the same code but looking up 4 bytes instead of 2 halfwords. That took around 5.5 seconds.
Third place goes to the bit-twiddling 'sideways addition' approach, which took 8.6 seconds.
Fourth place goes to GCC's __builtin_popcount(), at a shameful 11 seconds.
The counting one-bit-at-a-time approach was waaaay slower, and I got bored of waiting for it to complete.
So if you care about performance above all else then use the first approach. If you care, but not enough to spend 64Kb of RAM on it, use the second approach. Otherwise use the readable (but slow) one-bit-at-a-time approach.
It's hard to think of a situation where you'd want to use the bit-twiddling approach.
Edit: Similar results here.

unsigned int count_bit(unsigned int x)
{
x = (x & 0x55555555) + ((x >> 1) & 0x55555555);
x = (x & 0x33333333) + ((x >> 2) & 0x33333333);
x = (x & 0x0F0F0F0F) + ((x >> 4) & 0x0F0F0F0F);
x = (x & 0x00FF00FF) + ((x >> 8) & 0x00FF00FF);
x = (x & 0x0000FFFF) + ((x >> 16)& 0x0000FFFF);
return x;
}
Let me explain this algorithm.
This algorithm is based on Divide and Conquer Algorithm. Suppose there is a 8bit integer 213(11010101 in binary), the algorithm works like this(each time merge two neighbor blocks):
+-------------------------------+
| 1 | 1 | 0 | 1 | 0 | 1 | 0 | 1 | <- x
| 1 0 | 0 1 | 0 1 | 0 1 | <- first time merge
| 0 0 1 1 | 0 0 1 0 | <- second time merge
| 0 0 0 0 0 1 0 1 | <- third time ( answer = 00000101 = 5)
+-------------------------------+

This is one of those questions where it helps to know your micro-architecture. I just timed two variants under gcc 4.3.3 compiled with -O3 using C++ inlines to eliminate function call overhead, one billion iterations, keeping the running sum of all counts to ensure the compiler doesn't remove anything important, using rdtsc for timing (clock cycle precise).
inline int pop2(unsigned x, unsigned y)
{
x = x - ((x >> 1) & 0x55555555);
y = y - ((y >> 1) & 0x55555555);
x = (x & 0x33333333) + ((x >> 2) & 0x33333333);
y = (y & 0x33333333) + ((y >> 2) & 0x33333333);
x = (x + (x >> 4)) & 0x0F0F0F0F;
y = (y + (y >> 4)) & 0x0F0F0F0F;
x = x + (x >> 8);
y = y + (y >> 8);
x = x + (x >> 16);
y = y + (y >> 16);
return (x+y) & 0x000000FF;
}
The unmodified Hacker's Delight took 12.2 gigacycles. My parallel version (counting twice as many bits) runs in 13.0 gigacycles. 10.5s total elapsed for both together on a 2.4GHz Core Duo. 25 gigacycles = just over 10 seconds at this clock frequency, so I'm confident my timings are right.
This has to do with instruction dependency chains, which are very bad for this algorithm. I could nearly double the speed again by using a pair of 64-bit registers. In fact, if I was clever and added x+y a little sooner I could shave off some shifts. The 64-bit version with some small tweaks would come out about even, but count twice as many bits again.
With 128 bit SIMD registers, yet another factor of two, and the SSE instruction sets often have clever short-cuts, too.
There's no reason for the code to be especially transparent. The interface is simple, the algorithm can be referenced on-line in many places, and it's amenable to comprehensive unit test. The programmer who stumbles upon it might even learn something. These bit operations are extremely natural at the machine level.
OK, I decided to bench the tweaked 64-bit version. For this one sizeof(unsigned long) == 8
inline int pop2(unsigned long x, unsigned long y)
{
x = x - ((x >> 1) & 0x5555555555555555);
y = y - ((y >> 1) & 0x5555555555555555);
x = (x & 0x3333333333333333) + ((x >> 2) & 0x3333333333333333);
y = (y & 0x3333333333333333) + ((y >> 2) & 0x3333333333333333);
x = (x + (x >> 4)) & 0x0F0F0F0F0F0F0F0F;
y = (y + (y >> 4)) & 0x0F0F0F0F0F0F0F0F;
x = x + y;
x = x + (x >> 8);
x = x + (x >> 16);
x = x + (x >> 32);
return x & 0xFF;
}
That looks about right (I'm not testing carefully, though). Now the timings come out at 10.70 gigacycles / 14.1 gigacycles. That later number summed 128 billion bits and corresponds to 5.9s elapsed on this machine. The non-parallel version speeds up a tiny bit because I'm running in 64-bit mode and it likes 64-bit registers slightly better than 32-bit registers.
Let's see if there's a bit more OOO pipelining to be had here. This was a bit more involved, so I actually tested a bit. Each term alone sums to 64, all combined sum to 256.
inline int pop4(unsigned long x, unsigned long y,
unsigned long u, unsigned long v)
{
enum { m1 = 0x5555555555555555,
m2 = 0x3333333333333333,
m3 = 0x0F0F0F0F0F0F0F0F,
m4 = 0x000000FF000000FF };
x = x - ((x >> 1) & m1);
y = y - ((y >> 1) & m1);
u = u - ((u >> 1) & m1);
v = v - ((v >> 1) & m1);
x = (x & m2) + ((x >> 2) & m2);
y = (y & m2) + ((y >> 2) & m2);
u = (u & m2) + ((u >> 2) & m2);
v = (v & m2) + ((v >> 2) & m2);
x = x + y;
u = u + v;
x = (x & m3) + ((x >> 4) & m3);
u = (u & m3) + ((u >> 4) & m3);
x = x + u;
x = x + (x >> 8);
x = x + (x >> 16);
x = x & m4;
x = x + (x >> 32);
return x & 0x000001FF;
}
I was excited for a moment, but it turns out gcc is playing inline tricks with -O3 even though I'm not using the inline keyword in some tests. When I let gcc play tricks, a billion calls to pop4() takes 12.56 gigacycles, but I determined it was folding arguments as constant expressions. A more realistic number appears to be 19.6gc for another 30% speed-up. My test loop now looks like this, making sure each argument is different enough to stop gcc from playing tricks.
hitime b4 = rdtsc();
for (unsigned long i = 10L * 1000*1000*1000; i < 11L * 1000*1000*1000; ++i)
sum += pop4 (i, i^1, ~i, i|1);
hitime e4 = rdtsc();
256 billion bits summed in 8.17s elapsed. Works out to 1.02s for 32 million bits as benchmarked in the 16-bit table lookup. Can't compare directly, because the other bench doesn't give a clock speed, but looks like I've slapped the snot out of the 64KB table edition, which is a tragic use of L1 cache in the first place.
Update: decided to do the obvious and create pop6() by adding four more duplicated lines. Came out to 22.8gc, 384 billion bits summed in 9.5s elapsed. So there's another 20% Now at 800ms for 32 billion bits.

Why not iteratively divide by 2?
count = 0
while n > 0
if (n % 2) == 1
count += 1
n /= 2
I agree that this isn't the fastest, but "best" is somewhat ambiguous. I'd argue though that "best" should have an element of clarity

The Hacker's Delight bit-twiddling becomes so much clearer when you write out the bit patterns.
unsigned int bitCount(unsigned int x)
{
x = ((x >> 1) & 0b01010101010101010101010101010101)
+ (x & 0b01010101010101010101010101010101);
x = ((x >> 2) & 0b00110011001100110011001100110011)
+ (x & 0b00110011001100110011001100110011);
x = ((x >> 4) & 0b00001111000011110000111100001111)
+ (x & 0b00001111000011110000111100001111);
x = ((x >> 8) & 0b00000000111111110000000011111111)
+ (x & 0b00000000111111110000000011111111);
x = ((x >> 16)& 0b00000000000000001111111111111111)
+ (x & 0b00000000000000001111111111111111);
return x;
}
The first step adds the even bits to the odd bits, producing a sum of bits in each two. The other steps add high-order chunks to low-order chunks, doubling the chunk size all the way up, until we have the final count taking up the entire int.

For a happy medium between a 232 lookup table and iterating through each bit individually:
int bitcount(unsigned int num){
int count = 0;
static int nibblebits[] =
{0, 1, 1, 2, 1, 2, 2, 3, 1, 2, 2, 3, 2, 3, 3, 4};
for(; num != 0; num >>= 4)
count += nibblebits[num & 0x0f];
return count;
}
From http://ctips.pbwiki.com/CountBits

This can be done in O(k), where k is the number of bits set.
int NumberOfSetBits(int n)
{
int count = 0;
while (n){
++ count;
n = (n - 1) & n;
}
return count;
}

It's not the fastest or best solution, but I found the same question in my way, and I started to think and think. finally I realized that it can be done like this if you get the problem from mathematical side, and draw a graph, then you find that it's a function which has some periodic part, and then you realize the difference between the periods... so here you go:
unsigned int f(unsigned int x)
{
switch (x) {
case 0:
return 0;
case 1:
return 1;
case 2:
return 1;
case 3:
return 2;
default:
return f(x/4) + f(x%4);
}
}

I think the Brian Kernighan's method will be useful too...
It goes through as many iterations as there are set bits. So if we have a 32-bit word with only the high bit set, then it will only go once through the loop.
int countSetBits(unsigned int n) {
unsigned int n; // count the number of bits set in n
unsigned int c; // c accumulates the total bits set in n
for (c=0;n>0;n=n&(n-1)) c++;
return c;
}
Published in 1988, the C Programming Language 2nd Ed. (by Brian W. Kernighan and Dennis M. Ritchie) mentions this in exercise 2-9. On April 19, 2006 Don Knuth pointed out to me that this method "was first published by Peter Wegner in CACM 3 (1960), 322. (Also discovered independently by Derrick Lehmer and published in 1964 in a book edited by Beckenbach.)"

The function you are looking for is often called the "sideways sum" or "population count" of a binary number. Knuth discusses it in pre-Fascicle 1A, pp11-12 (although there was a brief reference in Volume 2, 4.6.3-(7).)
The locus classicus is Peter Wegner's article "A Technique for Counting Ones in a Binary Computer", from the Communications of the ACM, Volume 3 (1960) Number 5, page 322. He gives two different algorithms there, one optimized for numbers expected to be "sparse" (i.e., have a small number of ones) and one for the opposite case.

private int get_bits_set(int v)
{
int c; // 'c' accumulates the total bits set in 'v'
for (c = 0; v>0; c++)
{
v &= v - 1; // Clear the least significant bit set
}
return c;
}

Few open questions:-
If the number is negative then?
If the number is 1024 , then the "iteratively divide by 2" method will iterate 10 times.
we can modify the algo to support the negative number as follows:-
count = 0
while n != 0
if ((n % 2) == 1 || (n % 2) == -1
count += 1
n /= 2
return count
now to overcome the second problem we can write the algo like:-
int bit_count(int num)
{
int count=0;
while(num)
{
num=(num)&(num-1);
count++;
}
return count;
}
for complete reference see :
http://goursaha.freeoda.com/Miscellaneous/IntegerBitCount.html

I use the below code which is more intuitive.
int countSetBits(int n) {
return !n ? 0 : 1 + countSetBits(n & (n-1));
}
Logic : n & (n-1) resets the last set bit of n.
P.S : I know this is not O(1) solution, albeit an interesting solution.

What do you means with "Best algorithm"? The shorted code or the fasted code? Your code look very elegant and it has a constant execution time. The code is also very short.
But if the speed is the major factor and not the code size then I think the follow can be faster:
static final int[] BIT_COUNT = { 0, 1, 1, ... 256 values with a bitsize of a byte ... };
static int bitCountOfByte( int value ){
return BIT_COUNT[ value & 0xFF ];
}
static int bitCountOfInt( int value ){
return bitCountOfByte( value )
+ bitCountOfByte( value >> 8 )
+ bitCountOfByte( value >> 16 )
+ bitCountOfByte( value >> 24 );
}
I think that this will not more faster for a 64 bit value but a 32 bit value can be faster.

I wrote a fast bitcount macro for RISC machines in about 1990. It does not use advanced arithmetic (multiplication, division, %), memory fetches (way too slow), branches (way too slow), but it does assume the CPU has a 32-bit barrel shifter (in other words, >> 1 and >> 32 take the same amount of cycles.) It assumes that small constants (such as 6, 12, 24) cost nothing to load into the registers, or are stored in temporaries and reused over and over again.
With these assumptions, it counts 32 bits in about 16 cycles/instructions on most RISC machines. Note that 15 instructions/cycles is close to a lower bound on the number of cycles or instructions, because it seems to take at least 3 instructions (mask, shift, operator) to cut the number of addends in half, so log_2(32) = 5, 5 x 3 = 15 instructions is a quasi-lowerbound.
#define BitCount(X,Y) \
Y = X - ((X >> 1) & 033333333333) - ((X >> 2) & 011111111111); \
Y = ((Y + (Y >> 3)) & 030707070707); \
Y = (Y + (Y >> 6)); \
Y = (Y + (Y >> 12) + (Y >> 24)) & 077;
Here is a secret to the first and most complex step:
input output
AB CD Note
00 00 = AB
01 01 = AB
10 01 = AB - (A >> 1) & 0x1
11 10 = AB - (A >> 1) & 0x1
so if I take the 1st column (A) above, shift it right 1 bit, and subtract it from AB, I get the output (CD). The extension to 3 bits is similar; you can check it with an 8-row boolean table like mine above if you wish.
Don Gillies

if you're using C++ another option is to use template metaprogramming:
// recursive template to sum bits in an int
template <int BITS>
int countBits(int val) {
// return the least significant bit plus the result of calling ourselves with
// .. the shifted value
return (val & 0x1) + countBits<BITS-1>(val >> 1);
}
// template specialisation to terminate the recursion when there's only one bit left
template<>
int countBits<1>(int val) {
return val & 0x1;
}
usage would be:
// to count bits in a byte/char (this returns 8)
countBits<8>( 255 )
// another byte (this returns 7)
countBits<8>( 254 )
// counting bits in a word/short (this returns 1)
countBits<16>( 256 )
you could of course further expand this template to use different types (even auto-detecting bit size) but I've kept it simple for clarity.
edit: forgot to mention this is good because it should work in any C++ compiler and it basically just unrolls your loop for you if a constant value is used for the bit count (in other words, I'm pretty sure it's the fastest general method you'll find)

C++20 std::popcount
The following proposal has been merged http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2019/p0553r4.html and should add it to a the <bit> header.
I expect the usage to be like:
#include <bit>
#include <iostream>
int main() {
std::cout << std::popcount(0x55) << std::endl;
}
I'll give it a try when support arrives to GCC, GCC 9.1.0 with g++-9 -std=c++2a still doesn't support it.
The proposal says:
Header: <bit>
namespace std {
// 25.5.6, counting
template<class T>
constexpr int popcount(T x) noexcept;
and:
template<class T>
constexpr int popcount(T x) noexcept;
Constraints: T is an unsigned integer type (3.9.1 [basic.fundamental]).
Returns: The number of 1 bits in the value of x.
std::rotl and std::rotr were also added to do circular bit rotations: Best practices for circular shift (rotate) operations in C++

You can do:
while(n){
n = n & (n-1);
count++;
}
The logic behind this is the bits of n-1 is inverted from rightmost set bit of n.
If n=6, i.e., 110 then 5 is 101 the bits are inverted from rightmost set bit of n.
So if we & these two we will make the rightmost bit 0 in every iteration and always go to the next rightmost set bit. Hence, counting the set bit. The worst time complexity will be O(log n) when every bit is set.

I'm particularly fond of this example from the fortune file:
#define BITCOUNT(x) (((BX_(x)+(BX_(x)>>4)) & 0x0F0F0F0F) % 255)
#define BX_(x) ((x) - (((x)>>1)&0x77777777)
- (((x)>>2)&0x33333333)
- (((x)>>3)&0x11111111))
I like it best because it's so pretty!

Java JDK1.5
Integer.bitCount(n);
where n is the number whose 1's are to be counted.
check also,
Integer.highestOneBit(n);
Integer.lowestOneBit(n);
Integer.numberOfLeadingZeros(n);
Integer.numberOfTrailingZeros(n);
//Beginning with the value 1, rotate left 16 times
n = 1;
for (int i = 0; i < 16; i++) {
n = Integer.rotateLeft(n, 1);
System.out.println(n);
}

I found an implementation of bit counting in an array with using of SIMD instruction (SSSE3 and AVX2). It has in 2-2.5 times better performance than if it will use __popcnt64 intrinsic function.
SSSE3 version:
#include <smmintrin.h>
#include <stdint.h>
const __m128i Z = _mm_set1_epi8(0x0);
const __m128i F = _mm_set1_epi8(0xF);
//Vector with pre-calculated bit count:
const __m128i T = _mm_setr_epi8(0, 1, 1, 2, 1, 2, 2, 3, 1, 2, 2, 3, 2, 3, 3, 4);
uint64_t BitCount(const uint8_t * src, size_t size)
{
__m128i _sum = _mm128_setzero_si128();
for (size_t i = 0; i < size; i += 16)
{
//load 16-byte vector
__m128i _src = _mm_loadu_si128((__m128i*)(src + i));
//get low 4 bit for every byte in vector
__m128i lo = _mm_and_si128(_src, F);
//sum precalculated value from T
_sum = _mm_add_epi64(_sum, _mm_sad_epu8(Z, _mm_shuffle_epi8(T, lo)));
//get high 4 bit for every byte in vector
__m128i hi = _mm_and_si128(_mm_srli_epi16(_src, 4), F);
//sum precalculated value from T
_sum = _mm_add_epi64(_sum, _mm_sad_epu8(Z, _mm_shuffle_epi8(T, hi)));
}
uint64_t sum[2];
_mm_storeu_si128((__m128i*)sum, _sum);
return sum[0] + sum[1];
}
AVX2 version:
#include <immintrin.h>
#include <stdint.h>
const __m256i Z = _mm256_set1_epi8(0x0);
const __m256i F = _mm256_set1_epi8(0xF);
//Vector with pre-calculated bit count:
const __m256i T = _mm256_setr_epi8(0, 1, 1, 2, 1, 2, 2, 3, 1, 2, 2, 3, 2, 3, 3, 4,
0, 1, 1, 2, 1, 2, 2, 3, 1, 2, 2, 3, 2, 3, 3, 4);
uint64_t BitCount(const uint8_t * src, size_t size)
{
__m256i _sum = _mm256_setzero_si256();
for (size_t i = 0; i < size; i += 32)
{
//load 32-byte vector
__m256i _src = _mm256_loadu_si256((__m256i*)(src + i));
//get low 4 bit for every byte in vector
__m256i lo = _mm256_and_si256(_src, F);
//sum precalculated value from T
_sum = _mm256_add_epi64(_sum, _mm256_sad_epu8(Z, _mm256_shuffle_epi8(T, lo)));
//get high 4 bit for every byte in vector
__m256i hi = _mm256_and_si256(_mm256_srli_epi16(_src, 4), F);
//sum precalculated value from T
_sum = _mm256_add_epi64(_sum, _mm256_sad_epu8(Z, _mm256_shuffle_epi8(T, hi)));
}
uint64_t sum[4];
_mm256_storeu_si256((__m256i*)sum, _sum);
return sum[0] + sum[1] + sum[2] + sum[3];
}

A fast C# solution using a pre-calculated table of Byte bit counts with branching on the input size.
public static class BitCount
{
public static uint GetSetBitsCount(uint n)
{
var counts = BYTE_BIT_COUNTS;
return n <= 0xff ? counts[n]
: n <= 0xffff ? counts[n & 0xff] + counts[n >> 8]
: n <= 0xffffff ? counts[n & 0xff] + counts[(n >> 8) & 0xff] + counts[(n >> 16) & 0xff]
: counts[n & 0xff] + counts[(n >> 8) & 0xff] + counts[(n >> 16) & 0xff] + counts[(n >> 24) & 0xff];
}
public static readonly uint[] BYTE_BIT_COUNTS =
{
0, 1, 1, 2, 1, 2, 2, 3, 1, 2, 2, 3, 2, 3, 3, 4,
1, 2, 2, 3, 2, 3, 3, 4, 2, 3, 3, 4, 3, 4, 4, 5,
1, 2, 2, 3, 2, 3, 3, 4, 2, 3, 3, 4, 3, 4, 4, 5,
2, 3, 3, 4, 3, 4, 4, 5, 3, 4, 4, 5, 4, 5, 5, 6,
1, 2, 2, 3, 2, 3, 3, 4, 2, 3, 3, 4, 3, 4, 4, 5,
2, 3, 3, 4, 3, 4, 4, 5, 3, 4, 4, 5, 4, 5, 5, 6,
2, 3, 3, 4, 3, 4, 4, 5, 3, 4, 4, 5, 4, 5, 5, 6,
3, 4, 4, 5, 4, 5, 5, 6, 4, 5, 5, 6, 5, 6, 6, 7,
1, 2, 2, 3, 2, 3, 3, 4, 2, 3, 3, 4, 3, 4, 4, 5,
2, 3, 3, 4, 3, 4, 4, 5, 3, 4, 4, 5, 4, 5, 5, 6,
2, 3, 3, 4, 3, 4, 4, 5, 3, 4, 4, 5, 4, 5, 5, 6,
3, 4, 4, 5, 4, 5, 5, 6, 4, 5, 5, 6, 5, 6, 6, 7,
2, 3, 3, 4, 3, 4, 4, 5, 3, 4, 4, 5, 4, 5, 5, 6,
3, 4, 4, 5, 4, 5, 5, 6, 4, 5, 5, 6, 5, 6, 6, 7,
3, 4, 4, 5, 4, 5, 5, 6, 4, 5, 5, 6, 5, 6, 6, 7,
4, 5, 5, 6, 5, 6, 6, 7, 5, 6, 6, 7, 6, 7, 7, 8
};
}

I always use this in competitive programming, and it's easy to write and is efficient:
#include <bits/stdc++.h>
using namespace std;
int countOnes(int n) {
bitset<32> b(n);
return b.count();
}

There are many algorithm to count the set bits; but i think the best one is the faster one!
You can see the detailed on this page:
Bit Twiddling Hacks
I suggest this one:
Counting bits set in 14, 24, or 32-bit words using 64-bit instructions
unsigned int v; // count the number of bits set in v
unsigned int c; // c accumulates the total bits set in v
// option 1, for at most 14-bit values in v:
c = (v * 0x200040008001ULL & 0x111111111111111ULL) % 0xf;
// option 2, for at most 24-bit values in v:
c = ((v & 0xfff) * 0x1001001001001ULL & 0x84210842108421ULL) % 0x1f;
c += (((v & 0xfff000) >> 12) * 0x1001001001001ULL & 0x84210842108421ULL)
% 0x1f;
// option 3, for at most 32-bit values in v:
c = ((v & 0xfff) * 0x1001001001001ULL & 0x84210842108421ULL) % 0x1f;
c += (((v & 0xfff000) >> 12) * 0x1001001001001ULL & 0x84210842108421ULL) %
0x1f;
c += ((v >> 24) * 0x1001001001001ULL & 0x84210842108421ULL) % 0x1f;
This method requires a 64-bit CPU with fast modulus division to be efficient. The first option takes only 3 operations; the second option takes 10; and the third option takes 15.

Converting short-list to byte array, but only using last X bits

I have to compress a list of short-values into a byte array, but only the last X bits of the value.
Given this method:
byte[] compress(int bitsPerWord, List<Short> input){
...
}
The BitsPerWorld will never be bigger than the given values in the input field.
Example: 10 bits per word => maximum value 1023
I also may not waste bits, I have to save X bits in the first Y bytes, and then append the next X bits directly to them.
Example:
Input(Short) [ 500, 150, 100 ]
Input(Binary):0000000111110100 0000000001101000 0000000001100100
Output (10 bits per short): 0111110100 0001101000 0001100100
Output (As byte array):0111 1101 0000 0110 1000 0001 1001 0000
What the result should look like
Any way to do this efficiently? BitSet seems not fitting for this task, because i would have to set every single bit explicit.
Thanks

Efficient in what way?
In terms of work required, extending BitSet adding a bulk put method and an index is super efficient; little work and thinking required.
The alternative, shifting and masking bits is moderately complicated in terms of programming effort if you know your ways with bitwise operations. It may be a major obstacle if you don't.
Considering you already use wrapper types and collections, indicating troughput is not your major concern, extending BitSet is probably all you need.

You need to perform some bit manipulations, and for that to work you need to find a repeatable pattern. In this case, you have a list of "short" values, but actually you just use the rightmost 10 bits. Since you want to pack those into bytes, the minimum repeatable pattern is 40 bits long (5 bytes, 4 10-bit values). That is the "block size" for processing.
You would then have a loop that would do the main parsing of full blocks, plus maybe a special case at the end for the final incomplete block.
byte[] pack10(List<Short> source) {
final int nBlk = source.size() / 4;
final int remBits = (source.size() % 4) * 10;
final int remBytes = (remBits / 8) + (remBits % 8 > 0 ? 1 : 0);
byte[] ret = new byte[nBlk*5 + remBytes];
final short bitPat = (short)0b0000001111111111;
for (int iBlk = 0; iBlk < nBlk; ++iBlk) {
// Parse full blocks
List<Short> curS = source.subList(iBlk*4, (iBlk+1)*4);
ret[iBlk*5 ] = (byte) ((curS.get(0) & bitPat) >> 2);
ret[iBlk*5+1] = (byte) ((curS.get(0) & bitPat) << 6
| (curS.get(1) & bitPat) >> 4);
ret[iBlk*5+2] = (byte) ((curS.get(1) & bitPat) << 4
| (curS.get(2) & bitPat) >> 6);
ret[iBlk*5+3] = (byte) ((curS.get(2) & bitPat) << 2
| (curS.get(3) & bitPat) >> 8);
ret[iBlk*5+4] = (byte) (curS.get(3) & bitPat);
}
// Parse final values
List<Short> remS = source.subList(nBlocks*4, source.size());
if (remS.size() >= 1) {
ret[nBlk*5 ] = (byte) ((remS.get(0) & bitPat) >> 2);
ret[nBlk*5+1] = (byte) ((remS.get(0) & bitPat) << 6);
}
if (remS.size() >= 2) { // The first byte is appended to
ret[nBlk*5+1] |= (byte) ((remS.get(1) & bitPat) >> 4);
ret[nBlk*5+2] = (byte) ((remS.get(1) & bitPat) << 4);
}
if (remS.size() == 3) { // The first byte is appended to
ret[iBlk*5+2] |= (byte) ((curS.get(2) & bitPat) >> 6);
ret[iBlk*5+3] = (byte) ((curS.get(2) & bitPat) << 2);
}
return ret;
}
That is a specific version for 10-bit values; if you want a version with a generic number of values you'd have to generalise from that. The bit pattern operations changes, and all the system becomes less efficient if the pattern is computed at runtime (i.e. if the number of bits is a variable like in your example).

There are several people who have already written a BitOutputStream in Java. Pick one of them, wrap it in a ByteArrayOutputStream, and you’re done.

Get n Least Significant Bits from an Int

This seems fairly straightforward, but I cant find an answer. If I have an int X, what is the best way to get N least significant bits from this int, in Java?

This should work for all non-negative N < 33 32:
x & ((1 << N) - 1)
It's worth elaborating on how this works for N == 31 and N == 32. For N == 31, we get 1 << N == Integer.MIN_VALUE. When you subtract 1 from that, Java silently wraps around to Integer.MAX_VALUE, which is exactly what you need. For N == 32, the 1 bit is shifted completely out, so 1 << N == 0; then (1 << N) - 1 == -1, which is all 32 bits set.
For N == 32, this unfortunately doesn't work because (thanks, #zstring!) the << operator only shifts by the right side mod 32. Instead, if you want to avoid testing for that case specially, you could use:
x & ((int)(1L << N) - 1)
By shifting a long, you get the full 32-bit shift, which, after casting back to an int, gets you 0. Subtracting 1 gives you -1 and x & -1 is just x for any int value x (and x is the value of the lower 32 bits of x).

Ted's approach is likely to be faster but here is another approach
x << -N >>> -N
This shift all the bit up and then down to chop off the top bits.
int i = -1;
System.out.println(Integer.toBinaryString(i));
i = i << -5 >>> -5;
System.out.println(Integer.toBinaryString(i));
prints
11111111111111111111111111111111
11111

You can also use a mask. If you use the & bitwise operator you can then remove whatever bit you would want to remove (say the highest x bits);
int mask = 0x7FFFFFFF //Example mask where you will remove the
// most significant bit
// (0x7 = 0111b and 0xF = 1111b).
int result = numberToProcess & mask; //And apply the mask with the &bitwise op.
The disadvantage to this is that you will need to make a mask for each bit, so perhaps this is better seen as another method of approach in general.

use of the bitwise operators to pack multiple values in one int

Low level bit manipulation has never been my strong point. I will appreciate some help in understanding the following use case of bitwise operators.Consider...
int age, gender, height, packed_info;
. . . // Assign values
// Pack as AAAAAAA G HHHHHHH using shifts and "or"
packed_info = (age << 8) | (gender << 7) | height;
// Unpack with shifts and masking using "and"
height = packed_info & 0x7F; // This constant is binary ...01111111
gender = (packed_info >> 7) & 1;
age = (packed_info >> 8);
I am not sure what this code is accomplishing and how? Why use the magic number 0x7F ? How is the packing and unpacking accomplished?
Source

As the comment says, we're going to pack the age, gender and height into 15 bits, of the format:
AAAAAAAGHHHHHHH
Let's start with this part:
(age << 8)
To start with, age has this format:
age = 00000000AAAAAAA
where each A can be 0 or 1.
<< 8 moves the bits 8 places to the left, and fills in the gaps with zeroes. So you get:
(age << 8) = AAAAAAA00000000
Similarly:
gender = 00000000000000G
(gender << 7) = 0000000G0000000
height = 00000000HHHHHHH
Now we want to combine these into one variable. The | operator works by looking at each bit, and returning 1 if the bit is 1 in either of the inputs. So:
0011 | 0101 = 0111
If a bit is 0 in one input, then you get the bit from the other input. Looking at (age << 8), (gender << 7) and height, you'll see that, if a bit is 1 for one of these, it's 0 for the others. So:
packed_info = (age << 8) | (gender << 7) | height = AAAAAAAGHHHHHHH
Now we want to unpack the bits. Let's start with the height. We want to get the last 7 bits, and ignore the first 8. To do this, we use the & operator, which returns 1 only if both of the input bits are 1. So:
0011 & 0101 = 0001
So:
packed_info = AAAAAAAGHHHHHHH
0x7F = 000000001111111
(packed_info & 0x7F) = 00000000HHHHHHH = height
To get the age, we can just push everything 8 places to the right, and we're left with 0000000AAAAAAAA. So age = (packed_info >> 8).
Finally, to get the gender, we push everything 7 places to the right to get rid of the height. We then only care about the last bit:
packed_info = AAAAAAAGHHHHHHH
(packed_info >> 7) = 0000000AAAAAAAG
1 = 000000000000001
(packed_info >> 7) & 1 = 00000000000000G

This could be a rather long lesson in bit manipulation but first let me point you too the bit masking article on Wikipedia.
packed_info = (age << 8) | (gender << 7) | height;
Take age and move it's value over 8 bits then take gender and move it over 7 bits and height will occupy the last bits.
age = 0b101
gender = 0b1
height = 0b1100
packed_info = 0b10100000000
| 0b00010000000
| 0b00000001100
/* which is */
packed_info = 0b10110001100
Unpacking does the reverse but uses masks like 0x7F (which is 0b 01111111) to trim out the other values in the field.
gender = (packed_info >> 7) & 1;
Would work like...
gender = 0b1011 /* shifted 7 here but still has age on the other side */
& 0b0001
/* which is */
gender = 0b1
Note that ANDing anything to 1 is the same as "keeping" that bit and ANDing with 0 is the same as "ignoring" that bit.

If you were going to store a date as a number, maybe you would accomplish it by multiplying the year by 10000, the month by 100 and adding the day. A date such as July, 2, 2011 would be encoded as the number 20110702:
year * 10000 + month * 100 + day -> yyyymmdd
2011 * 10000 + 7 * 100 + 2 -> 20110702
We can say that we encoded the date in a yyyymmdd mask. We could describe this operation as
Shift the year 4 positions to the left,
shift the month 2 positions to the left and
leave the day as is.
Then combine the three values together.
This is the same thing that is happenning with the age, gender and height encoding, only that the author is thinking in binary.
See the ranges that those values may have:
age: 0 to 127 years
gender: M or F
height: 0 to 127 inches
If we translate those values to binary, we would have this:
age: 0 to 1111111b (7 binary digits, or bits)
gender: 0 or 1 (1 bit)
height: 0 to 1111111b (7 bits also)
With this in mind, we can encode the age-gender-height data with the mask aaaaaaaghhhhhhh, only that here we are talking about binary digits, not decimal digits.
So,
Shift the age 8 bits to the left,
shift the gender 7 bits to the left and
leave the height as is.
Then combine all three values together.
In binary, the Shift-Left operator (<<) moves a value n positions to the left. The "Or" operator ("|" in many languages) combines values together. Therefore:
(age << 8) | (gender << 7) | height
Now, how to "decode" those values?
It's easier in binary than in decimal:
You "mask away" the height,
shift the gender 7 bits to the right and mask that away also, and finally
shift the age 8 bits to the right.
The Shift-Right operator (>>) moves a value n positions to the right (whatever digits shifted "out" of the rightmost position are lost). The "And" binary operator ("&" in many languages) masks bits. To do that it needs a mask, indicating which bits to preserve and which bits to destroy (1 bits are preserved). Therefore:
height = value & 1111111b (preserve the 7 rightmost bits)
gender = (value >> 1) & 1 (preserve just one bit)
age = (value >> 8)
Since 1111111b in hex is 0x7f in most languages, that's the reason for that magic number. You would have the same effect by using 127 (which is 1111111b in decimal).

Same requirement I have faced many times. It is very easy with the help of Bitwise AND operator. Just qualify your values with increasing powers of two(2). To store multiple values, ADD their relative number ( power of 2 ) and get the SUM. This SUM will consolidate your selected values. HOW ?
Just do Bitwise AND with every value and it will give zero (0) for values which were not selected and non-zero for which are selected.
Here is the explanation:
1) Values ( YES, NO, MAYBE )
2) Assignment to power of two(2)
YES = 2^0 = 1 = 00000001
NO = 2^1 = 2 = 00000010
MAYBE = 2^2 = 4 = 00000100
3) I choose YES and MAYBE hence SUM:
SUM = 1 + 4 = 5
SUM = 00000001 + 00000100 = 00000101
This value will store both YES as well as MAYBE. HOW?
1 & 5 = 1 ( non zero )
2 & 5 = 0 ( zero )
4 & 5 = 4 ( non zero )
Hence SUM consists of
1 = 2^0 = YES
4 = 2^2 = MAYBE.
For more detailed explanation and implementation visit my blog

A more condense answer:
AAAAAAA G HHHHHHH
Packing:
packed = age << 8 | gender << 7 | height
Alternatively you can just sum components if ie when used in MySQL SUM aggregate function
packed = age << 8 + gender << 7 + height
Unpacking:
age = packed >> 8 // no mask required
gender = packed >> 7 & ((1 << 1) - 1) // applying mask (for gender it is just 1)
height = packed & ((1 << 7) - 1) // applying mask
Another (longer) example:
Say you have an IP address you want to pack, however it is a fictional IP address eg
132.513.151.319. Note that some components greater then 256 which requires more then 8 bits unlike real ip addresses.
First we need to figure out what offset we need to use to be able to store the max number.
Lets say with our fictional IPs no component can be bigger then 999 that means we need 10 bits of storage per component (allows numbers up to 1014).
packed = (comp1 << 0 * 10) | (comp1 << 1 * 10) | (comp1 << 2 * 10) | (comp1 << 3 * 10)
Which gives dec 342682502276 or bin 100111111001001011110000000010010000100
Now lets unpack the value
comp1 = (packed >> 0 * 10) & ((1 << 10) - 1) // 132
comp2 = (packed >> 1 * 10) & ((1 << 10) - 1) // 513
comp3 = (packed >> 2 * 10) & ((1 << 10) - 1) // 151
comp4 = (packed >> 3 * 10) & ((1 << 10) - 1) // 319
Where (1 << 10) - 1 is a binary mask we use to hide bits on the left beyond the 10 right most bits we are interested in.
Same example using MySQL query
SELECT
(#offset := 10) AS `No of bits required for each component`,
(#packed := (132 << 0 * #offset) |
(513 << 1 * #offset) |
(151 << 2 * #offset) |
(319 << 3 * #offset)) AS `Packed value (132.513.151.319)`,
BIN(#packed) AS `Packed value (bin)`,
(#packed >> 0 * #offset) & ((1 << #offset) - 1) `Component 1`,
(#packed >> 1 * #offset) & ((1 << #offset) - 1) `Component 2`,
(#packed >> 2 * #offset) & ((1 << #offset) - 1) `Component 3`,
(#packed >> 3 * #offset) & ((1 << #offset) - 1) `Component 4`;

The left shift operator means "multiply by two, this many times". In binary, multiplying a number by two is the same as adding a zero to the right side.
The right shift operator is the reverse of the left shift operator.
The pipe operator is "or", meaning overlay two binary numbers on top of each other, and where there is a 1 in either number the result in that column is a 1.
So, let's extract the operation for packed_info:
// Create age, shifted left 8 times:
// AAAAAAA00000000
age_shifted = age << 8;
// Create gender, shifted left 7 times:
// 0000000G0000000
gender_shifted = gender << 7;
// "Or" them all together:
// AAAAAAA00000000
// 0000000G0000000
// 00000000HHHHHHH
// ---------------
// AAAAAAAGHHHHHHH
packed_info = age_shifted | gender_shifted | height;
And the unpacking is the reverse.
// Grab the lowest 7 bits:
// AAAAAAAGHHHHHHH &
// 000000001111111 =
// 00000000HHHHHHH
height = packed_info & 0x7F;
// right shift the 'height' bits into the bit bucket, and grab the lowest 1 bit:
// AAAAAAAGHHHHHHH
// >> 7
// 0000000AAAAAAAG &
// 000000000000001 =
// 00000000000000G
gender = (packed_info >> 7) & 1;
// right shift the 'height' and 'gender' bits into the bit bucket, and grab the result:
// AAAAAAAGHHHHHHH
// >> 8
// 00000000AAAAAAA
age = (packed_info >> 8);

You can see the expression x & mask as an operation that removes from x the bits that are not present (i.e., have value 0) in mask. That means, packed_info & 0x7F removes from packed_info all the bits that are above the seventh bit.
Example: if packed_info is 1110010100101010 in binary, then packed_info & 0x7f will be
1110010100101010
0000000001111111
----------------
0000000000101010
So, in height we get the lower 7 bits of packed_info.
Next, we are shifting the whole packed_info by 7, this way we remove the information which we have already read out. So we get (for the value from previous example) 111001010 The gender is stored at the next bit, so with the same trick: & 1 we are extracting only that bit from the information. The rest of the information is contained at offset 8.
Packing back is not complicated, too: you take age, shift it 8 bits (so you get 1110010100000000 from 11100101), shift the gender by 7 (so you get 00000000), and take the height (assuming it would fit lower 7 bits). Then, you are composing all of them together:
1110010100000000
0000000000000000
0000000000101010
----------------
1110010100101010

Binary representation in Java

I am finding it difficult to understand and work with this binary representation in java:
With the help of the user Jon Skeet, I understood that binary representation should be built this way.
Here's a code sample:
public class chack {
public static void main(String[] args) {
int num2=2;
int num3=3;
int num4=4;
int num1=1;
int nirbinary = (num1 << 24) | (num2 << 16) | (num3 << 8) | num4;
System.out.println(nirbinary);
String nir= Integer.toBinaryString(nirbinary);
System.out.println(nir);
}
}
Couple of question:
How does one get num1 (for example) back from an int who is already in this binary
why do I get 16909060 when I print nirbinary- what does it stands for?
How does one get num1 (for example) back from an int who is already in this binary
representation?
Thank you

I am not completely sure what you are missing, so I will just explain how you can convert integers to binary strings back and forth in java.
You can get a binary string from an integer like so:
int i = 1234;
String binString = Integer.toBinaryString(i);
and you can convert the string back to an integer this way:
int iNew = Integer.parseInt(binString, 2);
Note the second argument to Integer.parseInt() is the desired base of the number. 2 is binary, 8 is octal, 10 decimal, etc.

16909060 stands for the number 16909060.
It is (1 * 224) + (2 * 216) + (3 * 28) + 4.
To get num1 back out, just right-shift the result the same amount you left-shifted and mask out the other bytes (not always necessary for num1(*), but for the others):
int num1 = nirbinary >> 24 & 0xFF;
int num2 = nirbinary >> 16 & 0xFF;
int num3 = nirbinary >> 8 & 0xFF;
int num4 = nirbinary & 0xFF;
Note that nirbinary is not "a binary representation". Or more precisely: it's no more or less binary than num1, num2, num3 and num4: internally all numbers (and characters, and booleans, ...) are stored in binary.
(*) note that if num1 is > 127, then you either need to use >>> to do the right-shift or use the & 0xFF in order to ensure that the correct value is restored. The difference between >> and >>> are the "new" bits inserted on the "left" side of the value: With >> they will depend on the highest-value bit (known as sign-extension) and with >>> they will always be 0.

Every int is a number, it's not binary, hex or decimal, it's just a number. the statement (num1 << 24) | (num2 << 16) | (num3 << 8) | num4; is a binary manipulation of 4 ints into another int. It doesn't change the representation of nirbinary to binary, since nirbinary has no representation, because (again) it's just a number.
Integer.toBinaryString(nirbinary) returns the binary representation of nirbinary which means "how would nibinary look like in base-2".
If you have a String which is a binary representation of a number, you could get its value, by using Integer.parseint(yourbinaryrepresentation, yourbase); for example - Integer.parseint(nir, 2);
And another thing:
You can't always get back one of the numbers back from nirbinary, since you performed a bit manipulation that is not reversible, for example:
int i1 = 5; //binary 0101
int i2 = 4; //binary 0100
int i3 = i1 | i2; //binary 0101
you cannot recognize each of your variables (i1, i2) since they have a common bit, i3 could have been the result of or on two other numbers:
int i1 = 1; //binary 0101
int i2 = 4; //binary 0100
int i3 = i1 | i2; //binary 0101
in your case, if each number is smaller than 256, you can reverse it with the following operation:
int myoldnumber = (nirbinary >> previousShift) & 0xff;
for example, to retrieve num1 you can do:
int retrievedNum1 = (nirbinary >> 24) & 0xff;

Here no need to depend only on binary or any other format...
one flexible built in function is available
That prints whichever format you want in your program..
Integer.toString(int,representation);
Integer.toString(100,8) // prints 144 --octal representation
Integer.toString(100,2) // prints 1100100 --binary representation
Integer.toString(100,16) //prints 64 --Hex representation
Integer.toString(100,5) // prints 400 --Base 5

When working with bitshifting and integers I would recommend you think in hexadecimal numbers, that will usually make life a lot easier. Just keep in mind that 8 bits represent 1 byte and 1 byte covers the hex-range from 0x00 to 0xFF
Since num1 to num4 are smaller than 10, their decimal representation is equal to their hex representiation, ie 1 = 0x01, 2 = 0x02 etc..
As I told you: 1 Byte is 8 bits. In your bitshifting operation you always shift multiple of 8.
So 0x01 << 8 => 0x0100
0x01 << 16 => 0x010000
etc.
So you basically only add zero bytes, which of course increases the value.
What you do next is to | them, a bitwise or. This means that two bitfields get modified in such a way that the result has a 1 at one place if at least one of the input values as a 1 there. Since your shifted ints contain only zero at the back, a bitwise or is nothing else then to put the value in this spot.
E.g:
(0x01 << 8) | 0x02
0x01 << 8 will produce 0x0100. Now you simply have to replace the last 00 with 02, since you or them: 0x0102
If you want to recreate the original int, you have to mask the part that int represents (this is easy since the parts do not overlap in your example) and then shift it back.
E.g.
Say ou produced 0x010203 and want to have only 0x02. You now have to mask shift it back 0x010203 >> 8 which will put the 02 in the last part. Now simply mask this last part 0x0102 && 0xFF. This will set all but the last 8 bits to zero

it's basically 1 * 2^24 + 2 * 2^16 + 3 * 2^8 + 4 = 16909060
You can get num1 by doing num1 = nirbinary >> 24.

What did you expect instead?
To get the most significant byte from an int i:
(i >> 24) & 0xff