We Keep Coding

Java - Assign Value to Fixed Byte Array

I'm am attempting to assign an integer value to an abnormal fixed size byte array (3). I saw about ByteBuffers allocate feature, however putInt attempts to put in 4 bytes, which then breaks due to overflow
For Example:
byte[] messageLength = ByteBuffer.allocate(3).putInt(Integer.parseUnsignedInt("300")).array();
Results in the following exception
Exception in thread "main" java.nio.BufferOverflowException
at java.nio.Buffer.nextPutIndex(Buffer.java:527)
at java.nio.HeapByteBuffer.putInt(HeapByteBuffer.java:372)
Obviously, 300 can fit into 3 bytes since in binary it is 0001 0010 1100.
What can I do to put a perfectly legal sized integer value into a non 4 byte array?

You need 4 bytes to allocate an Integer inside a byte buffer.
You can read from the byte buffer manually if you need a special unpacking rule.
Here's an example:
public static byte[] convertInts(int[] source) {
ByteBuffer buffer = ByteBuffer.allocate(4 * source.length);
for (int data : source) {
buffer.putInt(data);
}
buffer.flip();
byte[] destination = new byte[3 * source.length];
for (int i = 0; i < source.length; i++) {
buffer.get();
destination[i * 3] = buffer.get();
destination[i * 3 + 1] = buffer.get();
destination[i * 3 + 2] = buffer.get();
}
return destination;
}
Example usage:
int[] source = {
Integer.parseUnsignedInt("30"),
Integer.parseUnsignedInt("300"),
Integer.parseUnsignedInt("3000"),
Integer.parseUnsignedInt("300000"),
};
byte[] data = convertInts(source);

A simple solution is to convert the Integer value into a byte[] that contains only the necessary bits. The following code works with integers that fit in 1, 2, 3 and 4 bytes:
private static byte[] compressInteger(int value) {
if (value >= Byte.MIN_VALUE && value <= Byte.MAX_VALUE) {
return new byte[] { (byte) value };
} else if (value >= Short.MIN_VALUE && value <= Short.MAX_VALUE) {
return new byte[] { (byte) (value >>> 8), (byte) (value) };
} else if ((byte)(value >>> 24) == 0) {
return new byte[] { (byte) (value >>> 16), (byte) (value >>> 8), (byte) (value) };
} else {
return new byte[] { (byte) (value >>> 24), (byte) (value >>> 16), (byte) (value >>> 8), (byte) (value) };
}
}
If you want to convert the byte[] back to an integer value, you can do:
private static int decompressInteger(byte[] bytes) {
int value = 0;
for (int i = bytes.length - 1; i >= 0; i--) {
for (int bit = 0; bit <= 7; bit++) {
boolean isSet = ((bytes[i] >>> bit) & 1) == 1;
if (isSet) {
int shift = 8 * (bytes.length - 1 - i) + bit;
int mask = 1 << shift;
value |= mask;
}
}
}
return value;
}

Reading bytes in a file :java.lang.NegativeArraySizeException

I have gone through the similar posts but none of them really answered my question. Hence I post my question in a separate thread.
I need to skip some bytes in the file which I am reading in the byte array.I am trying to achieve this through code below
1. byte [] readBytesToSKip = null;
2. readBytesToSKip = new byte[(int)bytesToSkip];
3. bytesReadToSkip = System.in.read(readBytesToSKip) ;
4. if(bytesReadToSkip > 0)
5. {
6. baos_.write(readBytesToSKip, 0, bytesReadToSkip);
7. }
But I get a NegativeArraySizeException at line 2 where the size exceeds Integer.MAX_VALUE. I am not sure how to achieve this otherwise.
bytesToSkip is long as I calculate in function below:
public static long bytesToLong1(byte[] bytes) {
long value = 0;
for (int i = 0; i < bytes.length; i++)
{
value = (value << 8) + (bytes[i] & 0xff);
}
return value;
}

Starting from your posted bytes->long function
public static long bytesToLong1(byte[] bytes) {
long value = 0;
for (int i = 0; i < bytes.length; i++)
{
value = (value << 8) + (bytes[i] & 0xff);
}
return value;
}
Whenever bytes.length==8 and the first byte is >= 128 in absolute value (in Java specific: it's a negative byte value), you'll end having a negative value for your long.
Example:
byte vals[]={0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF};
System.out.println(bytesToLong1(vals));
will result it -1 being printed. Which will convert quite fine to an int with a -1 value, but will still be improper for the length of an array.

Bit shift operations on a byte array in Java

How do I shift a byte array n positions to the right? For instance shifting a 16 byte array right 29 positions? I read somewhere it can be done using a long? Would using a long work like this:
Long k1 = byte array from 0 to 7
Long k2 = byte array from 8 to 15
Then right rotating these two longs using Long.rotateRight(Long x, number of rotations).How would the two longs be joined back into a byte array?

I believe you can do this using java.math.BigInteger which supports shifts on arbitrarily large numbers. This has advantage of simplicity, but disadvantage of not padding into original byte array size, i.e. input could be 16 bytes but output might only be 10 etc, requiring additional logic.
BigInteger approach
byte [] array = new byte[]{0x7F,0x11,0x22,0x33,0x44,0x55,0x66,0x77};
// create from array
BigInteger bigInt = new BigInteger(array);
// shift
BigInteger shiftInt = bigInt.shiftRight(4);
// back to array
byte [] shifted = shiftInt.toByteArray();
// print it as hex
for (byte b : shifted) {
System.out.print(String.format("%x", b));
}
Output
7f1122334455667 <== shifted 4 to the right. Looks OK
Long manipulation
I don't know why you'd want to do this as rotateRight() as this makes life more difficult, you have to blank at the bits that appear at the left hand side in K1 etc. You'd be better with using shift IMO as describe below. I've used a shift of 20 as divisible by 4 so easier to see the nibbles move in the output.
1) Use ByteBuffer to form two longs from 16 byte array
byte[] array = { 0x00, 0x00, 0x11, 0x11, 0x22, 0x22, 0x33, 0x33, 0x44, 0x44, 0x55, 0x55, 0x66, 0x66, 0x77, 0x77 };
ByteBuffer buffer = ByteBuffer.wrap(array);
long k1 = buffer.getLong();
long k2 = buffer.getLong();
2) Shift each long n bits to the right
int n = 20;
long k1Shift = k1 >> n;
long k2Shift = k2 >> n;
System.out.println(String.format("%016x => %016x", k1, k1Shift));
System.out.println(String.format("%016x => %016x", k2, k2Shift));
0000111122223333 => 0000000001111222
4444555566667777 => 0000044445555666
Determine bits from k1 that "got pushed off the edge"
long k1CarryBits = (k1 << (64 - n));
System.out.println(String.format("%016x => %016x", k1, k1CarryBits));
0000111122223333 => 2333300000000000
Join the K1 carry bits onto K2 on right hand side
long k2WithCarray = k2Shift | k1CarryBits;
System.out.println(String.format("%016x => %016x", k2Shift, k2WithCarray));
0000044445555666 => 2333344445555666
Write the two longs back into a ByteBuffer and extract as a byte array
buffer.position(0);
buffer.putLong(k1Shift);
buffer.putLong(k2WithCarray);
for (byte each : buffer.array()) {
System.out.print(Long.toHexString(each));
}
000011112222333344445555666

Here is what I came up with to shift a byte array by some arbitrary number of bits left:
/**
* Shifts input byte array len bits left.This method will alter the input byte array.
*/
public static byte[] shiftLeft(byte[] data, int len) {
int word_size = (len / 8) + 1;
int shift = len % 8;
byte carry_mask = (byte) ((1 << shift) - 1);
int offset = word_size - 1;
for (int i = 0; i < data.length; i++) {
int src_index = i+offset;
if (src_index >= data.length) {
data[i] = 0;
} else {
byte src = data[src_index];
byte dst = (byte) (src << shift);
if (src_index+1 < data.length) {
dst |= data[src_index+1] >>> (8-shift) & carry_mask;
}
data[i] = dst;
}
}
return data;
}

1. Manually implemented
Here are left and right shift implementation without using BigInteger (ie. without creating a copy of the input array) and with unsigned right shift (BigInteger only supports arithmetic shifts of course)
Left Shift <<
/**
* Left shift of whole byte array by shiftBitCount bits.
* This method will alter the input byte array.
*/
static byte[] shiftLeft(byte[] byteArray, int shiftBitCount) {
final int shiftMod = shiftBitCount % 8;
final byte carryMask = (byte) ((1 << shiftMod) - 1);
final int offsetBytes = (shiftBitCount / 8);
int sourceIndex;
for (int i = 0; i < byteArray.length; i++) {
sourceIndex = i + offsetBytes;
if (sourceIndex >= byteArray.length) {
byteArray[i] = 0;
} else {
byte src = byteArray[sourceIndex];
byte dst = (byte) (src << shiftMod);
if (sourceIndex + 1 < byteArray.length) {
dst |= byteArray[sourceIndex + 1] >>> (8 - shiftMod) & carryMask;
}
byteArray[i] = dst;
}
}
return byteArray;
}
Unsigned Right Shift >>>
/**
* Unsigned/logical right shift of whole byte array by shiftBitCount bits.
* This method will alter the input byte array.
*/
static byte[] shiftRight(byte[] byteArray, int shiftBitCount) {
final int shiftMod = shiftBitCount % 8;
final byte carryMask = (byte) (0xFF << (8 - shiftMod));
final int offsetBytes = (shiftBitCount / 8);
int sourceIndex;
for (int i = byteArray.length - 1; i >= 0; i--) {
sourceIndex = i - offsetBytes;
if (sourceIndex < 0) {
byteArray[i] = 0;
} else {
byte src = byteArray[sourceIndex];
byte dst = (byte) ((0xff & src) >>> shiftMod);
if (sourceIndex - 1 >= 0) {
dst |= byteArray[sourceIndex - 1] << (8 - shiftMod) & carryMask;
}
byteArray[i] = dst;
}
}
return byteArray;
}
Used in this class by this Project.
2. Using BigInteger
Be aware that BigInteger internally converts the byte array into an int[] array so this may not be the most optimized solution:
Arithmetic Left Shift <<:
byte[] result = new BigInteger(byteArray).shiftLeft(3).toByteArray();
Arithmetic Right Shift >>:
byte[] result = new BigInteger(byteArray).shiftRight(2).toByteArray();
3. External Library
Using the Bytes java library*:
Add to pom.xml:
<dependency>
<groupId>at.favre.lib</groupId>
<artifactId>bytes</artifactId>
<version>{latest-version}</version>
</dependency>
Code example:
Bytes b = Bytes.wrap(someByteArray);
b.leftShift(3);
b.rightShift(3);
byte[] result = b.array();
*Full Disclaimer: I am the developer.

The is an old post, but I want to update Adam's answer.
The long solution works with a few tweak.
In order to rotate, use >>> instead of >>, because >> will pad with significant bit, changing the original value.
second, the printbyte function seems to miss leading 00 when it prints.
use this instead.
private String getHexString(byte[] b) {
StringBuilder result = new StringBuilder();
for (int i = 0; i < b.length; i++)
result.append(Integer.toString((b[i] & 0xff) + 0x100, 16)
.substring(1));
return result.toString();
}

Getting the CRC checksum of a byte array and adding it to that byte array

I have this byte array:
static byte[] buf = new byte[] { (byte) 0x01, (byte) 0x04, (byte)0x00, (byte)0x01,(byte)0x00, (byte) 0x01};
Now, the CRC checksum of this byte array is supposed to be 0x60, 0x0A. I want the Java code to recreate this checksum, however I cant seem to recreate it. I have tried crc16:
static int crc16(final byte[] buffer) {
int crc = 0xFFFF;
for (int j = 0; j < buffer.length ; j++) {
crc = ((crc >>> 8) | (crc << 8) )& 0xffff;
crc ^= (buffer[j] & 0xff);//byte to int, trunc sign
crc ^= ((crc & 0xff) >> 4);
crc ^= (crc << 12) & 0xffff;
crc ^= ((crc & 0xFF) << 5) & 0xffff;
}
crc &= 0xffff;
return crc;
}
and convert them using Integer.toHexString(), but none of the results match the correct CRC. Could someone please point me in the right direction in terms of CRC formula.

Use the following code instead:
// Compute the MODBUS RTU CRC
private static int ModRTU_CRC(byte[] buf, int len)
{
int crc = 0xFFFF;
for (int pos = 0; pos < len; pos++) {
crc ^= (int)buf[pos] & 0xFF; // XOR byte into least sig. byte of crc
for (int i = 8; i != 0; i--) { // Loop over each bit
if ((crc & 0x0001) != 0) { // If the LSB is set
crc >>= 1; // Shift right and XOR 0xA001
crc ^= 0xA001;
}
else // Else LSB is not set
crc >>= 1; // Just shift right
}
}
// Note, this number has low and high bytes swapped, so use it accordingly (or swap bytes)
return crc;
}
You may have to reverse your return CRC to get the right endianness, though. I even tested it here:
http://ideone.com/PrBXVh
Using windows calculator or something you can see that the first result (from the above function call) gives the expected value (albeit reversed).

Would CRC32 do, or does it have to be CRC16? If 32 is okay, have you tried using the CRC32 in java.util.zip?
import java.util.zip.CRC32;
byte[] buf = new byte[] { (byte) 0x01, (byte) 0x04, (byte)0x00, (byte)0x01,(byte)0x00, (byte) 0x01};
CRC32 crc32 = new CRC32();
crc32.update(buf);
System.out.printf("%X\n", crc32.getValue());
The output is:
F9DB8E67
Then you can do whatever additional calculation you want on top of that.

I was working on modbus using Java 1.6, tried the above code and it only partially worked? Agreed on some CRCs, wrong on others. I researched it a bit more, and saw I had a problem with sign extension. I masked off the high bits (see FIX HERE below) and now it works great.
NOTE: All CRC calcs are not the same, MODBUS is a bit different:
public static int getCRC(byte[] buf, int len ) {
int crc = 0xFFFF;
int val = 0;
for (int pos = 0; pos < len; pos++) {
crc ^= (int)(0x00ff & buf[pos]); // FIX HERE -- XOR byte into least sig. byte of crc
for (int i = 8; i != 0; i--) { // Loop over each bit
if ((crc & 0x0001) != 0) { // If the LSB is set
crc >>= 1; // Shift right and XOR 0xA001
crc ^= 0xA001;
}
else // Else LSB is not set
crc >>= 1; // Just shift right
}
}
// Note, crc has low and high bytes swapped, so use it accordingly (or swap bytes)
val = (crc & 0xff) << 8;
val = val + ((crc >> 8) & 0xff);
System.out.printf("Calculated a CRC of 0x%x, swapped: 0x%x\n", crc, val);
return val;
} // end GetCRC

How can I convert a 4-byte array to an integer?

I want to perform a conversion without resorting to some implementation-dependent trick. Any tips?

You need to know the endianness of your bytes.
Assuming (like #WhiteFang34) that bytes is a byte[] of length 4, then...
Big-endian:
int x = java.nio.ByteBuffer.wrap(bytes).getInt();
Little-endian:
int x = java.nio.ByteBuffer.wrap(bytes).order(java.nio.ByteOrder.LITTLE_ENDIAN).getInt();

Assuming bytes is a byte[4] of an integer in big-endian order, typically used in networking:
int value = ((bytes[0] & 0xFF) << 24) | ((bytes[1] & 0xFF) << 16)
| ((bytes[2] & 0xFF) << 8) | (bytes[3] & 0xFF);
The & 0xFF are necessary because byte is signed in Java and you need to retain the signed bit here. You can reverse the process with this:
bytes[0] = (byte) ((value >> 24) & 0xFF);
bytes[1] = (byte) ((value >> 16) & 0xFF);
bytes[2] = (byte) ((value >> 8) & 0xFF);
bytes[3] = (byte) (value & 0xFF);

Not sure if this is correct java syntax, but how about:
int value = 0;
for (i = 0; i <= 3; i++)
value = (value << 8) + (bytes[i] & 0xFF);

You need to specify the byte order of the array, but assuming that the bytes[0] is the most significant byte then:
int res = ((bytes[0] & 0xff) << 24) | ((bytes[1] & 0xff) << 16) |
((bytes[2] & 0xff) << 8) | (bytes[3] & 0xff);
This code is 100% portable, assuming that you use the reverse algorithm to create the byte array in the first place.
Byte order problems arise in languages where you can cast between a native integer type and byte array type ... and then discover that different architectures store the bytes of an integer in different orders.
You can't do that cast in Java. So for Java to Java communication, this should not be an issue.
However, if you are sending or receiving packets to some remote application that is implemented in (say) C or C++, you need to "know" what byte order is being used in the network packets. Some alternative strategies for knowing / figuring this out are:
Everyone uses "network order" (big-endian) for stuff on the wire as per the example code above. Non-java applications on little-endian machines need to flip the bytes.
The sender finds out what order the receiver expects and uses that order when assembling the data.
The receiver figures out what order the sender used (e.g. via a flag in the packet) and decodes accordingly.
The first approach is simplest and most widely used, though it does result in 2 unnecessary endian-ness conversions if both the sender and receiver are little-endian.
See http://en.wikipedia.org/wiki/Endianness

Assuming your byte[] come from somewhere e.g. a stream you can use
DataInputStream dis = ... // can wrap a new ByteArrayInputStream(bytes)
int num = dis.readInt(); // assume big-endian.
or
ByteChannel bc = ... // can be a SocketChannel
ByteBuffer bb = ByteBuffer.allocate(64*1024);
bc.read(bb);
bb.flip();
if (bb.remaining()<4) // not enough data
int num = bb.getInt();
When you send data, you should know if you are sending big-endian or little endian. You have to assume other things such as whether you are sending a 4-byte signed integer. A binary protocol is full of assumptions. (Which makes it more compact and faster, but more brittle than text)
If you don't want to be making as many assumptions, send text.

WE can also use following to make it more dynamic byte array size
BigEndian Format:
public static int pareAsBigEndianByteArray(byte[] bytes) {
int factor = bytes.length - 1;
int result = 0;
for (int i = 0; i < bytes.length; i++) {
if (i == 0) {
result |= bytes[i] << (8 * factor--);
} else {
result |= bytes[i] << (8 * factor--);
}
}
return result;
}
Little Endian Format :
public static int pareAsLittleEndianByteArray(byte[] bytes) {
int result = 0;
for (int i = 0; i < bytes.length; i++) {
if (i == 0) {
result |= bytes[i] << (8 * i);
} else {
result |= bytes[i] << (8 * i);
}
}
return result;
}
This will helps you lot for converting bytes to int values

public static int toInt( byte[] bytes ) {
int result = 0;
for (int i=0; i<3; i++) {
result = ( result << 8 ) - Byte.MIN_VALUE + (int) bytes[i];
}
return result;
}