We Keep Coding

Converting short-list to byte array, but only using last X bits

I have to compress a list of short-values into a byte array, but only the last X bits of the value.
Given this method:
byte[] compress(int bitsPerWord, List<Short> input){
...
}
The BitsPerWorld will never be bigger than the given values in the input field.
Example: 10 bits per word => maximum value 1023
I also may not waste bits, I have to save X bits in the first Y bytes, and then append the next X bits directly to them.
Example:
Input(Short) [ 500, 150, 100 ]
Input(Binary):0000000111110100 0000000001101000 0000000001100100
Output (10 bits per short): 0111110100 0001101000 0001100100
Output (As byte array):0111 1101 0000 0110 1000 0001 1001 0000
What the result should look like
Any way to do this efficiently? BitSet seems not fitting for this task, because i would have to set every single bit explicit.
Thanks

Efficient in what way?
In terms of work required, extending BitSet adding a bulk put method and an index is super efficient; little work and thinking required.
The alternative, shifting and masking bits is moderately complicated in terms of programming effort if you know your ways with bitwise operations. It may be a major obstacle if you don't.
Considering you already use wrapper types and collections, indicating troughput is not your major concern, extending BitSet is probably all you need.

You need to perform some bit manipulations, and for that to work you need to find a repeatable pattern. In this case, you have a list of "short" values, but actually you just use the rightmost 10 bits. Since you want to pack those into bytes, the minimum repeatable pattern is 40 bits long (5 bytes, 4 10-bit values). That is the "block size" for processing.
You would then have a loop that would do the main parsing of full blocks, plus maybe a special case at the end for the final incomplete block.
byte[] pack10(List<Short> source) {
final int nBlk = source.size() / 4;
final int remBits = (source.size() % 4) * 10;
final int remBytes = (remBits / 8) + (remBits % 8 > 0 ? 1 : 0);
byte[] ret = new byte[nBlk*5 + remBytes];
final short bitPat = (short)0b0000001111111111;
for (int iBlk = 0; iBlk < nBlk; ++iBlk) {
// Parse full blocks
List<Short> curS = source.subList(iBlk*4, (iBlk+1)*4);
ret[iBlk*5 ] = (byte) ((curS.get(0) & bitPat) >> 2);
ret[iBlk*5+1] = (byte) ((curS.get(0) & bitPat) << 6
| (curS.get(1) & bitPat) >> 4);
ret[iBlk*5+2] = (byte) ((curS.get(1) & bitPat) << 4
| (curS.get(2) & bitPat) >> 6);
ret[iBlk*5+3] = (byte) ((curS.get(2) & bitPat) << 2
| (curS.get(3) & bitPat) >> 8);
ret[iBlk*5+4] = (byte) (curS.get(3) & bitPat);
}
// Parse final values
List<Short> remS = source.subList(nBlocks*4, source.size());
if (remS.size() >= 1) {
ret[nBlk*5 ] = (byte) ((remS.get(0) & bitPat) >> 2);
ret[nBlk*5+1] = (byte) ((remS.get(0) & bitPat) << 6);
}
if (remS.size() >= 2) { // The first byte is appended to
ret[nBlk*5+1] |= (byte) ((remS.get(1) & bitPat) >> 4);
ret[nBlk*5+2] = (byte) ((remS.get(1) & bitPat) << 4);
}
if (remS.size() == 3) { // The first byte is appended to
ret[iBlk*5+2] |= (byte) ((curS.get(2) & bitPat) >> 6);
ret[iBlk*5+3] = (byte) ((curS.get(2) & bitPat) << 2);
}
return ret;
}
That is a specific version for 10-bit values; if you want a version with a generic number of values you'd have to generalise from that. The bit pattern operations changes, and all the system becomes less efficient if the pattern is computed at runtime (i.e. if the number of bits is a variable like in your example).

There are several people who have already written a BitOutputStream in Java. Pick one of them, wrap it in a ByteArrayOutputStream, and you’re done.

Bit operations converting to an integer

I have some binary operations that are not working like I expect.
I have byte array with the first 2 bytes having these values : 0x5, and 0xE0.
I want to combine them into an integer value that should be 0x5E0.
I tried doing :
int val = (b[i]) << 8 | b[i+1];
but the value is coming out 0xFFFFFFEE0 and the first byte 0x5 is getting lost
I thought this would be easy? What am I doing wrong?

Try: int val = ((b[i] & 0xff) << 8) | (b[i + 1] & 0xff). Bytes are (unfortunately) signed in Java, so if the high bit is set, it gets sign-extended when converted to an integer.

The problem is that byte data type is signed. Therefore, b[i+1] gets sign-extended before performing the operation, becoming 0xFFFFFFE0. When it gets OR-ed with 0x0500 from b[i]<<8, the 0x0500 gets lost.
You can fix this by AND-ing with 0xFF before performing the operation:
public static int toInt16(byte high, byte low) {
int res = (high << 8);
res |= (low & 0xFF);
return res & 0xFFFF;
}
Demo.

Bit twiddling: interleave bytes in word with zero bytes using bitwise operators (<<, |, &, etc)

Given a long with bytes WXYZ (where each letter is a byte), I would like some fast bit twiddling code that will create two longs with the same bytes as the original, but interleaved with the 0 byte.
For example, given the long with value ABCDEFGH (each letter being one byte), produce the two longs:
0A0B0C0D
0E0F0G0H
Something equivalent to, but faster than:
long result1 = expand((int)(input >>> 32));
long result2 = expand((int)input);
long expand(int inputInt) {
long input = intputInt;
return
(input & 0x000000FF) |
(input & 0x0000FF00) << 8 |
(input & 0x00FF0000) << 16 |
(input & 0xFF000000) << 24;
}

The following is about 25% faster for me (Java 7, benchmarked using Google Caliper), YMMV may vary according to your compiler of course:
long a = (input | (input << 16));
long result = (a & 0xFF000000FFL) + ((a & 0xFF000000FF00L) <<8);
The idea is to use a bit of extra parallelism vs. the original approach.
The first line is a neat trick that produces garbage in bits 17-32, but you don't care as you are going to mask it out anyway. :-)

long expand(int inputInt) {
long input = intputInt;
return
(input & 0x000000FF) << 8 |
(input & 0x0000FF00) << 16 |
(input & 0x00FF0000) << 24 |
(input & 0xFF000000) << 32;
}

In C++ you could try to use a union:
typedef union
{
char bytes[8];
long value;
} PlatformSpecificSolution;
long expand(int valueInt)
{
PlatformSpecificSolution pss;
pss.value = valueInt;
pss.bytes[6] = pss.bytes[3]; pss.bytes[3] = 0;
pss.bytes[4] = pss.bytes[2]; pss.bytes[2] = 0;
pss.bytes[2] = pss.bytes[1]; pss.bytes[1] = 0;
// pss.bytes[0] = pss.bytes[0];
return pss.value;
}
I have no idea if this is faster (you will have to run benchmarks on the platforms that you want to support). This solution is definitely more error prone. You should always ask yourself, if the performance benefit outways the disadvantage of less maintainable code.

convert four 32 bits ints to IP address in java

Using the code found here: https://libbits.wordpress.com/2011/05/17/check-if-ip-is-within-range-specified-in-cidr-in-java/
// Step 1. Convert IPs into ints (32 bits).
// E.g. 157.166.224.26 becomes 10011101 10100110 11100000 00011010
int addr = (( 157 << 24 ) & 0xFF000000)
| (( 166 << 16 ) & 0xFF0000)
| (( 224 << 8 ) & 0xFF00)
| ( 26 & 0xFF);
// Step 2. Get CIDR mask
int mask = (-1) << (32 - 10);
// Step 3. Find lowest IP address
int lowest = addr & mask;
// Step 4. Find highest IP address
int highest = lowest + (~mask);
I'm able to split a string into four ints and create boundaries for my IP range.
Now I want to be able to generate an ip that is between the highest and lowest values. For example:
given the range: 157.166.224.26/10 I get an address of -1650008038 my lowest ip address is -1652555776 and highest ip address is -1648361473. Now I need to generate a number that is between my lowest and highest and convert it back to four integers, this last part is where I'm lost at, I'm not sure how to convert -1648361473 to an ip address

That's pretty easy. Let say the IPv4 address is in the ipaddr variable, you can write something like that:
byte[] addr = new byte[4];
addr[0] = (ipaddr >> 24) & 0xFF;
addr[1] = (ipaddr >> 16) & 0xFF;
addr[2] = (ipaddr >> 8 ) & 0xFF;
addr[3] = ipaddr & 0xFF;
InetAddress inetAddr = InetAddress.getByAddress(addr);

The answer that Teetoo made above deserves some explanation.
Lets start with the first value:
(ipaddr >> 24) & 0xFF
When this is shifted down, the 8 bits representing the 157 are in the rightmost position of the resulting integer. However, since the value was initially negative, you would have 1's in 24 most significant bits, which would end up giving you a negative number. What you want is to 0 out all but those last 8 bits, hence the "& 0xFF". Another way to do this would be to right shift using the >>> operator which forces 0's into the most significant bits.
(ipaddr >>> 24)
now we move on to:
(ipaddr >> 16) & 0xFF
When you shift, you'll have the 16 leftmost bits set to 1 (due to shifting a negative number). Then you'll have the 8 bits representing the 157 and then the 8 bits representing 166. In this case, the >>> operator wouldn't help us because we still have the 157 in there. So the "& 0xFF" will 0 out all but the 8 bits for the 166.
Similarly for the last two values.

an addition to #Teetoo's answer.
let's ByteBuffer do the int to byte array magic
ByteBuffer bb = ByteBuffer.wrap (addr);
bb.putInt (iIP);

You can use
int addr = (157 << 24) | (166 << 16) | (224 << 8) | 26;
to reverse this.
byte[] addrAsBytes = { (byte) (addr >> 24), (byte) (addr >> 16),
(byte) (addr >> 8), (byte) addr };

Binary representation in Java

I am finding it difficult to understand and work with this binary representation in java:
With the help of the user Jon Skeet, I understood that binary representation should be built this way.
Here's a code sample:
public class chack {
public static void main(String[] args) {
int num2=2;
int num3=3;
int num4=4;
int num1=1;
int nirbinary = (num1 << 24) | (num2 << 16) | (num3 << 8) | num4;
System.out.println(nirbinary);
String nir= Integer.toBinaryString(nirbinary);
System.out.println(nir);
}
}
Couple of question:
How does one get num1 (for example) back from an int who is already in this binary
why do I get 16909060 when I print nirbinary- what does it stands for?
How does one get num1 (for example) back from an int who is already in this binary
representation?
Thank you

I am not completely sure what you are missing, so I will just explain how you can convert integers to binary strings back and forth in java.
You can get a binary string from an integer like so:
int i = 1234;
String binString = Integer.toBinaryString(i);
and you can convert the string back to an integer this way:
int iNew = Integer.parseInt(binString, 2);
Note the second argument to Integer.parseInt() is the desired base of the number. 2 is binary, 8 is octal, 10 decimal, etc.

16909060 stands for the number 16909060.
It is (1 * 224) + (2 * 216) + (3 * 28) + 4.
To get num1 back out, just right-shift the result the same amount you left-shifted and mask out the other bytes (not always necessary for num1(*), but for the others):
int num1 = nirbinary >> 24 & 0xFF;
int num2 = nirbinary >> 16 & 0xFF;
int num3 = nirbinary >> 8 & 0xFF;
int num4 = nirbinary & 0xFF;
Note that nirbinary is not "a binary representation". Or more precisely: it's no more or less binary than num1, num2, num3 and num4: internally all numbers (and characters, and booleans, ...) are stored in binary.
(*) note that if num1 is > 127, then you either need to use >>> to do the right-shift or use the & 0xFF in order to ensure that the correct value is restored. The difference between >> and >>> are the "new" bits inserted on the "left" side of the value: With >> they will depend on the highest-value bit (known as sign-extension) and with >>> they will always be 0.

Every int is a number, it's not binary, hex or decimal, it's just a number. the statement (num1 << 24) | (num2 << 16) | (num3 << 8) | num4; is a binary manipulation of 4 ints into another int. It doesn't change the representation of nirbinary to binary, since nirbinary has no representation, because (again) it's just a number.
Integer.toBinaryString(nirbinary) returns the binary representation of nirbinary which means "how would nibinary look like in base-2".
If you have a String which is a binary representation of a number, you could get its value, by using Integer.parseint(yourbinaryrepresentation, yourbase); for example - Integer.parseint(nir, 2);
And another thing:
You can't always get back one of the numbers back from nirbinary, since you performed a bit manipulation that is not reversible, for example:
int i1 = 5; //binary 0101
int i2 = 4; //binary 0100
int i3 = i1 | i2; //binary 0101
you cannot recognize each of your variables (i1, i2) since they have a common bit, i3 could have been the result of or on two other numbers:
int i1 = 1; //binary 0101
int i2 = 4; //binary 0100
int i3 = i1 | i2; //binary 0101
in your case, if each number is smaller than 256, you can reverse it with the following operation:
int myoldnumber = (nirbinary >> previousShift) & 0xff;
for example, to retrieve num1 you can do:
int retrievedNum1 = (nirbinary >> 24) & 0xff;

Here no need to depend only on binary or any other format...
one flexible built in function is available
That prints whichever format you want in your program..
Integer.toString(int,representation);
Integer.toString(100,8) // prints 144 --octal representation
Integer.toString(100,2) // prints 1100100 --binary representation
Integer.toString(100,16) //prints 64 --Hex representation
Integer.toString(100,5) // prints 400 --Base 5

When working with bitshifting and integers I would recommend you think in hexadecimal numbers, that will usually make life a lot easier. Just keep in mind that 8 bits represent 1 byte and 1 byte covers the hex-range from 0x00 to 0xFF
Since num1 to num4 are smaller than 10, their decimal representation is equal to their hex representiation, ie 1 = 0x01, 2 = 0x02 etc..
As I told you: 1 Byte is 8 bits. In your bitshifting operation you always shift multiple of 8.
So 0x01 << 8 => 0x0100
0x01 << 16 => 0x010000
etc.
So you basically only add zero bytes, which of course increases the value.
What you do next is to | them, a bitwise or. This means that two bitfields get modified in such a way that the result has a 1 at one place if at least one of the input values as a 1 there. Since your shifted ints contain only zero at the back, a bitwise or is nothing else then to put the value in this spot.
E.g:
(0x01 << 8) | 0x02
0x01 << 8 will produce 0x0100. Now you simply have to replace the last 00 with 02, since you or them: 0x0102
If you want to recreate the original int, you have to mask the part that int represents (this is easy since the parts do not overlap in your example) and then shift it back.
E.g.
Say ou produced 0x010203 and want to have only 0x02. You now have to mask shift it back 0x010203 >> 8 which will put the 02 in the last part. Now simply mask this last part 0x0102 && 0xFF. This will set all but the last 8 bits to zero

it's basically 1 * 2^24 + 2 * 2^16 + 3 * 2^8 + 4 = 16909060
You can get num1 by doing num1 = nirbinary >> 24.

What did you expect instead?
To get the most significant byte from an int i:
(i >> 24) & 0xff