We Keep Coding

Bit shift operations on a byte array in Java

How do I shift a byte array n positions to the right? For instance shifting a 16 byte array right 29 positions? I read somewhere it can be done using a long? Would using a long work like this:
Long k1 = byte array from 0 to 7
Long k2 = byte array from 8 to 15
Then right rotating these two longs using Long.rotateRight(Long x, number of rotations).How would the two longs be joined back into a byte array?

I believe you can do this using java.math.BigInteger which supports shifts on arbitrarily large numbers. This has advantage of simplicity, but disadvantage of not padding into original byte array size, i.e. input could be 16 bytes but output might only be 10 etc, requiring additional logic.
BigInteger approach
byte [] array = new byte[]{0x7F,0x11,0x22,0x33,0x44,0x55,0x66,0x77};
// create from array
BigInteger bigInt = new BigInteger(array);
// shift
BigInteger shiftInt = bigInt.shiftRight(4);
// back to array
byte [] shifted = shiftInt.toByteArray();
// print it as hex
for (byte b : shifted) {
System.out.print(String.format("%x", b));
}
Output
7f1122334455667 <== shifted 4 to the right. Looks OK
Long manipulation
I don't know why you'd want to do this as rotateRight() as this makes life more difficult, you have to blank at the bits that appear at the left hand side in K1 etc. You'd be better with using shift IMO as describe below. I've used a shift of 20 as divisible by 4 so easier to see the nibbles move in the output.
1) Use ByteBuffer to form two longs from 16 byte array
byte[] array = { 0x00, 0x00, 0x11, 0x11, 0x22, 0x22, 0x33, 0x33, 0x44, 0x44, 0x55, 0x55, 0x66, 0x66, 0x77, 0x77 };
ByteBuffer buffer = ByteBuffer.wrap(array);
long k1 = buffer.getLong();
long k2 = buffer.getLong();
2) Shift each long n bits to the right
int n = 20;
long k1Shift = k1 >> n;
long k2Shift = k2 >> n;
System.out.println(String.format("%016x => %016x", k1, k1Shift));
System.out.println(String.format("%016x => %016x", k2, k2Shift));
0000111122223333 => 0000000001111222
4444555566667777 => 0000044445555666
Determine bits from k1 that "got pushed off the edge"
long k1CarryBits = (k1 << (64 - n));
System.out.println(String.format("%016x => %016x", k1, k1CarryBits));
0000111122223333 => 2333300000000000
Join the K1 carry bits onto K2 on right hand side
long k2WithCarray = k2Shift | k1CarryBits;
System.out.println(String.format("%016x => %016x", k2Shift, k2WithCarray));
0000044445555666 => 2333344445555666
Write the two longs back into a ByteBuffer and extract as a byte array
buffer.position(0);
buffer.putLong(k1Shift);
buffer.putLong(k2WithCarray);
for (byte each : buffer.array()) {
System.out.print(Long.toHexString(each));
}
000011112222333344445555666

Here is what I came up with to shift a byte array by some arbitrary number of bits left:
/**
* Shifts input byte array len bits left.This method will alter the input byte array.
*/
public static byte[] shiftLeft(byte[] data, int len) {
int word_size = (len / 8) + 1;
int shift = len % 8;
byte carry_mask = (byte) ((1 << shift) - 1);
int offset = word_size - 1;
for (int i = 0; i < data.length; i++) {
int src_index = i+offset;
if (src_index >= data.length) {
data[i] = 0;
} else {
byte src = data[src_index];
byte dst = (byte) (src << shift);
if (src_index+1 < data.length) {
dst |= data[src_index+1] >>> (8-shift) & carry_mask;
}
data[i] = dst;
}
}
return data;
}

1. Manually implemented
Here are left and right shift implementation without using BigInteger (ie. without creating a copy of the input array) and with unsigned right shift (BigInteger only supports arithmetic shifts of course)
Left Shift <<
/**
* Left shift of whole byte array by shiftBitCount bits.
* This method will alter the input byte array.
*/
static byte[] shiftLeft(byte[] byteArray, int shiftBitCount) {
final int shiftMod = shiftBitCount % 8;
final byte carryMask = (byte) ((1 << shiftMod) - 1);
final int offsetBytes = (shiftBitCount / 8);
int sourceIndex;
for (int i = 0; i < byteArray.length; i++) {
sourceIndex = i + offsetBytes;
if (sourceIndex >= byteArray.length) {
byteArray[i] = 0;
} else {
byte src = byteArray[sourceIndex];
byte dst = (byte) (src << shiftMod);
if (sourceIndex + 1 < byteArray.length) {
dst |= byteArray[sourceIndex + 1] >>> (8 - shiftMod) & carryMask;
}
byteArray[i] = dst;
}
}
return byteArray;
}
Unsigned Right Shift >>>
/**
* Unsigned/logical right shift of whole byte array by shiftBitCount bits.
* This method will alter the input byte array.
*/
static byte[] shiftRight(byte[] byteArray, int shiftBitCount) {
final int shiftMod = shiftBitCount % 8;
final byte carryMask = (byte) (0xFF << (8 - shiftMod));
final int offsetBytes = (shiftBitCount / 8);
int sourceIndex;
for (int i = byteArray.length - 1; i >= 0; i--) {
sourceIndex = i - offsetBytes;
if (sourceIndex < 0) {
byteArray[i] = 0;
} else {
byte src = byteArray[sourceIndex];
byte dst = (byte) ((0xff & src) >>> shiftMod);
if (sourceIndex - 1 >= 0) {
dst |= byteArray[sourceIndex - 1] << (8 - shiftMod) & carryMask;
}
byteArray[i] = dst;
}
}
return byteArray;
}
Used in this class by this Project.
2. Using BigInteger
Be aware that BigInteger internally converts the byte array into an int[] array so this may not be the most optimized solution:
Arithmetic Left Shift <<:
byte[] result = new BigInteger(byteArray).shiftLeft(3).toByteArray();
Arithmetic Right Shift >>:
byte[] result = new BigInteger(byteArray).shiftRight(2).toByteArray();
3. External Library
Using the Bytes java library*:
Add to pom.xml:
<dependency>
<groupId>at.favre.lib</groupId>
<artifactId>bytes</artifactId>
<version>{latest-version}</version>
</dependency>
Code example:
Bytes b = Bytes.wrap(someByteArray);
b.leftShift(3);
b.rightShift(3);
byte[] result = b.array();
*Full Disclaimer: I am the developer.

The is an old post, but I want to update Adam's answer.
The long solution works with a few tweak.
In order to rotate, use >>> instead of >>, because >> will pad with significant bit, changing the original value.
second, the printbyte function seems to miss leading 00 when it prints.
use this instead.
private String getHexString(byte[] b) {
StringBuilder result = new StringBuilder();
for (int i = 0; i < b.length; i++)
result.append(Integer.toString((b[i] & 0xff) + 0x100, 16)
.substring(1));
return result.toString();
}

Sending Java int to C over TCP

I'm trying to send Java's signed integers over TCP to a C client.
At the Java side, I write the integers to the outputstream like so:
static ByteBuffer wrapped = ByteBuffer.allocateDirect(4); // big-endian by default
public static void putInt(OutputStream out, int nr) throws IOException {
wrapped.rewind();
wrapped.putInt(nr);
wrapped.rewind();
for (int i = 0; i < 4; i++)
out.write(wrapped.get());
}
At the C side, I read the integers like so:
int cnt = 0;
char buf[1];
char sizebuf[4];
while(cnt < 4) {
iResult = recv(ConnectSocket, buf, 1, 0);
if (iResult <= 0) continue;
sizebuf[cnt] = buf[0];
cnt++;
}
However, how do I convert the char array to an integer in C?
Edit
I have tried the following (and the reverse):
int charsToInt(char* array) {
return (array[3] << 24) | (array[2] << 16) | (array[1] << 8) | array[0];
}
Edited again, because I forgot the tags.
Data
For example of what happens currently:
I receive:
char 0
char 0
char 12
char -64
the int becomes 2448
and use the function for creating the int from the char array:
int charsToInt(char* array) {
return ntohl(*((int*) array));
}
I expect the signed integer: 3264
Update
I will investigate more after some sleep..
Update
I have a Java client which interprets the integers correctly and receives the exact same bytes:
0
0
12
-64

That depends on endianness, but you want either:
int x = sizebuf[0] +
(sizebuf[1] << 8) +
(sizebuf[2] << 16) +
(sizebuf[3] << 24);
or:
int x = sizebuf[3] +
(sizebuf[2] << 8) +
(sizebuf[1] << 16) +
(sizebuf[0] << 24);
Note that sizebuf needs to have an unsigned type for this to work correctly. Otherwise you need to mask off any sign-extended values you don't want:
int x = (sizebuf[3] & 0x000000ff) +
((sizebuf[2] << 8) & 0x0000ff00) +
((sizebuf[1] << 16) & 0x00ff0000) +
((sizebuf[0] << 24) & 0xff000000);

The classical C library has the method you want already, and it is independent from the machine endianness: ntohl!
// buf is a char */uint8_t *
uint32_t from_network = *((uint32_t *) buf);
uint32_t ret = ntohl(from_network);
This, and htonl for the reverse etc expect that the "network order" is big endian.
(the code above presupposes that buf has at least 4 bytes; the return type, and argument type, of ntohl and htonl are uint32_t; the JLS defines an int as 4 bytes so you are guaranteed the result)

To convert you char array, one possibility is to cast it to int* and to store the result :
int result = *((int*) sizebuf)
This is valid and one line. Other possibility is to compute integer from chars.
for (i = 0 ; i < 4; i++)
result = result << sizeof(char) + buf[0]
Choose the one that you prefer.
Alexis.
Edit :
sizeof(char) is 1 because sizeof return a Byte result. So the right line is :
result = result << (sizeof(char) * 8) + buf[0]

How to correctly convert from rgb565 to rgb888

I'm trying to convert a value from rgb565 to rgb888 but I can't seem to get it right.
I've also checked a few existing answers like this one but there are values that don't seem to be properly convert.
Here's what I've tried so far:
PImage picker;
int c;
void setup(){
size(285,240);
picker = loadImage("hsv.gif");
int c = color(255,255,255);
byte[] word = colorToWord(c);
int[] rgb = wordToRGB(word);
noStroke();
fill(c);
rect(0,0,50,100);
fill(rgb[0],rgb[1],rgb[2]);
rect(50,0,50,100);
}
void draw(){
image(picker,0,0);
if((mouseX >= 0 && mouseX <= picker.width)&&
(mouseY >= 0 && mouseY <= picker.height)) c = picker.get(mouseX,mouseY);
byte[] word = colorToWord(c);
int[] rgb = wordToRGB(word);
// int[] rgb = rgbWordToRGB(word);
int cc = color(rgb[0],rgb[1],rgb[2]);
fill(c);
rect(0,159,142,80);
fill(cc);
rect(142,159,143,80);
fill(0);
text("original\n"+hex(c),10,185);
text("converted\n"+hex(cc),152,185);
}
byte[] colorToWord(int c){
int r = (c >> 16) & 0xFF;
int g = (c >> 8) & 0xFF;
int b = c & 0xFF;
return new byte[]{(byte)((r&248)|g>>5),(byte)((g&28)<<3|b>>3)};
}
int[] wordToRGB3(byte[] data){
// Reconstruct 16 bit rgb565 value from two bytes
int rgb565 = (data[0] & 255) | ((data[1] & 255) << 8);
// Extract raw component values (range 0..31 for g and b, 0..63 for g)
int b5 = rgb565 & 0x1f;
int g6 = (rgb565 >> 5) & 0x3f;
int r5 = (rgb565 >> 11) & 0x1f;
// Scale components up to 8 bit:
// Shift left and fill empty bits at the end with the highest bits,
// so 00000 is extended to 000000000 but 11111 is extended to 11111111
int b = (b5 << 3) | (b5 >> 2);
int g = (g6 << 2) | (g6 >> 4);
int r = (r5 << 3) | (r5 >> 2);
return new int[]{r,g,b};
}
int[] wordToRGB2(byte[] word){
int r = word[0]&248;
int g = (word[0]<<5) | ((word[1]&28)>>3);
int b = word[1] << 3;
r <<= 3;
g <<= 2;
b <<= 3;
return new int[]{r,g,b};
}
int[] wordToRGB(byte[] word){
int c = (word[0] << 8) | (word[1]);
//RGB565
int r = (c >> (6+5)) & 0x01F;
int g = (c >> 5) & 0x03F;
int b = (c) & 0x01F;
//RGB888 - amplify
r <<= 3;
g <<= 2;
b <<= 3;
return new int[]{r,g,b};
}
final int RGB565_MASK_RED = 0xF800;
final int RGB565_MASK_GREEN = 0x07E0;
final int RGB565_MASK_BLUE = 0x001F;
int[] rgbWordToRGB(byte[] word){
int rgb565 = (word[0] << 8) | (word[1]);
int[] rgb24 = new int[3];
rgb24[2] = (rgb565 & RGB565_MASK_RED) >> 11;
rgb24[1] = (rgb565 & RGB565_MASK_GREEN) >> 5;
rgb24[0] = (rgb565 & RGB565_MASK_BLUE);
//amplify the image
rgb24[2] <<= 3;
rgb24[1] <<= 2;
rgb24[0] <<= 3;
return rgb24;
}
using this image as a colour palette:
The problem is with certain values:
Why does that happen ? Is there a way to fix that ?

ok so here's the fix if I understand correctly:
in your wordToRGB() method replace the first line:
int c = (word[0] << 8) | (word[1]);
with
int c = (word[0] << 8) | (word[1] & 0xFF);
The problem happens when you convert from byte to int. Example 1:
int a = 255;
println(binary(a));
int b = (byte) 255;
println(binary(b));
prints:
00000000000000000000000011111111
11111111111111111111111111111111
Example 2:
int a = 127;
println(binary(a));
int b = (byte) 127;
println(binary(b));
prints:
00000000000000000000000001111111
00000000000000000000000001111111
Processing pads with to the left with whatever your last bit was, so essentially after that, your | distorts all the values.

>> is arithmetic shift, so it will sign extends the value. Use logic shift >>> instead
In C, there's no >>> operators, the right shift depends on implementation but most will do a logical shift if the variable is unsigned, so just cast or define it as unsigned

Interval Partitioning on IP address range

I'm dealing with a problem to partition specific IP address ranges (in Java). Suppose I have:
startIP endIP
1.2.3.4 1.2.5.6
and I need cut this range into every [0, 255] interval so we have:
startIP endIP
1.2.3.4 1.2.3.255
1.2.4.0 1.2.4.255
1.2.5.0 1.2.5.6
I'm looking at this problem more or less like partitioning a decimal range for instance from 7 to 26 and we have [7, 9], [10, 19], [20, 26]. The only difference is we are dealing with 256-simal numbers. Any ideas, folks? Thank you!

You can use InetAddress for this, and run a straightforward loop to do the increments:
InetAddress from = InetAddress.getByName("1.2.3.4");
InetAddress to = InetAddress.getByName("1.2.5.6");
byte[] partsTo = to.getAddress();
for (;;) {
System.out.print(from+" - ");
byte[] parts = from.getAddress();
boolean sameUpperPart = true;
for (int i = 0 ; sameUpperPart && i < parts.length-1 ; i++) {
sameUpperPart &= (partsTo[i] == parts[i]);
}
if (sameUpperPart) {
System.out.println(to);
break;
}
int last = parts.length-1;
parts[last] = (byte)0xFF;
System.out.println(InetAddress.getByAddress(parts));
parts[last] = 0x00;
for (int i = last-1 ; i >= 0 ; i--) {
if (++parts[i] != 0) {
break;
}
}
from = InetAddress.getByAddress(parts);
}
This code produces the following output on ideone:
/1.2.3.4 - /1.2.3.255
/1.2.4.0 - /1.2.4.255
/1.2.5.0 - /1.2.5.6

Some sample code (assumes the initial translation to a int[] is done outside) and which doesn't really return useful data (just prints). But the idea is there.
public class Convertor {
public static void convert(int []a, int []b) {
int left = (a[0] << 24) | (a[1] << 16) | (a[2] << 8) | a[3];
int right = left | 0xff;
int end = (b[0] << 24) | (b[1] << 16) | (b[2] << 8) | b[3];
while ( right < end ) {
System.out.printf("%s -> %s\n", toIp(left), toIp(right));
left = right + 1; right += 256;
}
System.out.printf("%s -> %s\n", toIp(left), toIp(end));
}
private static String toIp(int value) {
return String.format("%d.%d.%d.%d",
value >> 24 & 0xFF,
value >> 16 & 0xFF,
value >> 8 & 0xFF,
value & 0xFF);
}
}
call site:
Convertor.convert(new int[]{1, 2, 3, 4}, new int[]{1, 2, 5, 6});
output:
1.2.3.4 -> 1.2.3.255
1.2.4.0 -> 1.2.4.255
1.2.5.0 -> 1.2.5.6

First convert the IP address into a single integer, then partition by just adding 256 repeatedly, and converting back to an IP address.
public class IPPartition {
public static void main(String[] args) {
String startIP = "1.2.3.4";
String endIP = "1.2.5.6";
long start = toLong(startIP);
long end = toLong(endIP);
long last = start;
long part = (start / 256 + 1) * 256;
for (; part < end; last = part, part += 256) {
System.out.println(toIP(last) + " " + toIP(part));
}
System.out.println(toIP(last) + " " + toIP(end));
}
private static long toLong(String ip) {
String[] tokens = ip.split("\\.");
long sum = 0;
for (int i = 0; i < 4; i++) {
sum *= 256;
sum += Integer.parseInt(tokens[i]);
}
return sum;
}
private static String toIP(long num) {
String result = "";
for (int i = 0; i < 4; i++) {
long section = num % 256;
result = section + result;
if (i < 3) result = "." + result;
num /= 256;
}
return result;
}
}
I used longs here just to be safe, but you might be able to use integers, I haven't tried it. There could also be some off-by-one errors or something, I haven't tested it thoroughly.

I wouldn't know the specific solution for Java, but there is an algorithm in C. It will partition the IP range to largest possible subnets and print them. The resulting ranges will be aligned on subnet boundaries.
#define NETWORK_ZEROS(ip, prefix) ((ip) & (0xffffffff << (32 - (prefix))))
#define NETWORK_ONES(ip, prefix) ((ip) | (~ (0xffffffff << (32 - (prefix)))))
void print_ip(unsigned long ip)
{
unsigned char bytes[4];
bytes[0] = ip & 0xFF;
bytes[1] = (ip >> 8) & 0xFF;
bytes[2] = (ip >> 16) & 0xFF;
bytes[3] = (ip >> 24) & 0xFF;
printf("%d.%d.%d.%d", bytes[3], bytes[2], bytes[1], bytes[0]);
}
void partition_ip(unsigned long start, unsigned long stop)
{
int i;
while (start < stop)
{
// Change the start of the loop to 24 if you
// need to align on /24 boundaries
for (i=0; i <= 32; i++)
{
if (NETWORK_ZEROS(start, i) == start &&
NETWORK_ONES(start, i) <= stop)
{
print_ip(NETWORK_ZEROS(start, i));
printf(" - ");
print_ip(NETWORK_ONES(start, i));
printf("\n");
start = NETWORK_ONES(start, i) + 1;
break;
}
}
}
}
To convert IP to binary number just do something like this
a.b.c.d => ((((a*256)+b)*256)+c)*256+d
or
ip = (a << 24) | (b << 16) | (c << 8) | d
if you want it to be more efficient by utilizing bit shifting.

how to read signed int from bytes in java?

I have a spec which reads the next two bytes are signed int.
To read that in java i have the following
When i read a signed int in java using the following code i get a value of 65449
Logic for calculation of unsigned
int a =(byte[1] & 0xff) <<8
int b =(byte[0] & 0xff) <<0
int c = a+b
I believe this is wrong because if i and with 0xff i get an unsigned equivalent
so i removed the & 0xff and the logic as given below
int a = byte[1] <<8
int b = byte[0] << 0
int c = a+b
which gives me the value -343
byte[1] =-1
byte[0]=-87
I tried to offset these values with the way the spec reads but this looks wrong.Since the size of the heap doesnt fall under this.
Which is the right way to do for signed int calculation in java?
Here is how the spec goes
somespec() { xtype 8 uint8 xStyle 16 int16 }
xStyle :A signed integer that represents an offset (in bytes) from the start of this Widget() structure to the start of an xStyle() structure that expresses inherited styles for defined by page widget as well as styles that apply specifically to this widget.

If you value is a signed 16-bit you want a short and int is 32-bit which can also hold the same values but not so naturally.
It appears you wants a signed little endian 16-bit value.
byte[] bytes =
short s = ByteBuffer.wrap(bytes).order(ByteOrder.LITTLE_ENDIAN).getShort();
or
short s = (short) ((bytes[0] & 0xff) | (bytes[1] << 8));
BTW: You can use an int but its not so simple.
// to get a sign extension.
int i = ((bytes[0] & 0xff) | (bytes[1] << 8)) << 16 >> 16;
or
int i = (bytes[0] & 0xff) | (short) (bytes[1] << 8));

Assuming that bytes[1] is the MSB, and bytes[0] is the LSB, and that you want the answer to be a 16 bit signed integer:
short res16 = ((bytes[1] << 8) | bytes[0]);
Then to get a 32 bit signed integer:
int res32 = res16; // sign extends.
By the way, the specification should say which of the two bytes is the MSB, and which is the LSB. If it doesn't and if there aren't any examples, you can't implement it!
Somewhere in the spec it will say how an "int16" is represented. Paste THAT part. Or paste a link to the spec so that we can read it ourselves.

Take a look on DataInputStream.readInt(). You can either steel code from there or just use DataInputStream: wrap your input stream with it and then read typed data easily.
For your convenience this is the code:
public final int readInt() throws IOException {
int ch1 = in.read();
int ch2 = in.read();
int ch3 = in.read();
int ch4 = in.read();
if ((ch1 | ch2 | ch3 | ch4) < 0)
throw new EOFException();
return ((ch1 << 24) + (ch2 << 16) + (ch3 << 8) + (ch4 << 0));
}

I can't compile it right now, but I would do (assuming byte1 and byte0 are realling of byte type).
int result = byte1;
result = result << 8;
result = result | byte0; //(binary OR)
if (result & 0x8000 == 0x8000) { //sign extension
result = result | 0xFFFF0000;
}
if byte1 and byte0 are ints, you will need to make the `&0xFF
UPDATE because Java forces the expression of an if to be a boolean

do you have a way of finding a correct output for a given input?
technically, an int size is 4 bytes, so with just 2 bytes you can't reach the sign bit.

I ran across this same problem reading a MIDI file. A MIDI file has signed 16 bit as well as signed 32 bit integers. In a MIDI file, the most significant bytes come first (big-endian).
Here's what I did. It might be crude, but it maintains the sign. If the least significant bytes come first (little-endian), reverse the order of the indexes.
pos is the position in the byte array where the number starts.
length is the length of the integer, either 2 or 4. Yes, a 2 byte integer is a short, but we all work with ints.
private int convertBytes(byte[] number, int pos, int length) {
int output = 0;
if (length == 2) {
output = ((int) number[pos]) << 24;
output |= convertByte(number[pos + 1]) << 16;
output >>= 16;
} else if (length == 4) {
output = ((int) number[pos]) << 24;
output |= convertByte(number[pos + 1]) << 16;
output |= convertByte(number[pos + 2]) << 8;
output |= convertByte(number[pos + 3]);
}
return output;
}
private int convertByte(byte number) {
return (int) number & 0xff;
}