I'm am attempting to assign an integer value to an abnormal fixed size byte array (3). I saw about ByteBuffers allocate feature, however putInt attempts to put in 4 bytes, which then breaks due to overflow
For Example:
byte[] messageLength = ByteBuffer.allocate(3).putInt(Integer.parseUnsignedInt("300")).array();
Results in the following exception
Exception in thread "main" java.nio.BufferOverflowException
at java.nio.Buffer.nextPutIndex(Buffer.java:527)
at java.nio.HeapByteBuffer.putInt(HeapByteBuffer.java:372)
Obviously, 300 can fit into 3 bytes since in binary it is 0001 0010 1100.
What can I do to put a perfectly legal sized integer value into a non 4 byte array?
You need 4 bytes to allocate an Integer inside a byte buffer.
You can read from the byte buffer manually if you need a special unpacking rule.
Here's an example:
public static byte[] convertInts(int[] source) {
ByteBuffer buffer = ByteBuffer.allocate(4 * source.length);
for (int data : source) {
buffer.putInt(data);
}
buffer.flip();
byte[] destination = new byte[3 * source.length];
for (int i = 0; i < source.length; i++) {
buffer.get();
destination[i * 3] = buffer.get();
destination[i * 3 + 1] = buffer.get();
destination[i * 3 + 2] = buffer.get();
}
return destination;
}
Example usage:
int[] source = {
Integer.parseUnsignedInt("30"),
Integer.parseUnsignedInt("300"),
Integer.parseUnsignedInt("3000"),
Integer.parseUnsignedInt("300000"),
};
byte[] data = convertInts(source);
A simple solution is to convert the Integer value into a byte[] that contains only the necessary bits. The following code works with integers that fit in 1, 2, 3 and 4 bytes:
private static byte[] compressInteger(int value) {
if (value >= Byte.MIN_VALUE && value <= Byte.MAX_VALUE) {
return new byte[] { (byte) value };
} else if (value >= Short.MIN_VALUE && value <= Short.MAX_VALUE) {
return new byte[] { (byte) (value >>> 8), (byte) (value) };
} else if ((byte)(value >>> 24) == 0) {
return new byte[] { (byte) (value >>> 16), (byte) (value >>> 8), (byte) (value) };
} else {
return new byte[] { (byte) (value >>> 24), (byte) (value >>> 16), (byte) (value >>> 8), (byte) (value) };
}
}
If you want to convert the byte[] back to an integer value, you can do:
private static int decompressInteger(byte[] bytes) {
int value = 0;
for (int i = bytes.length - 1; i >= 0; i--) {
for (int bit = 0; bit <= 7; bit++) {
boolean isSet = ((bytes[i] >>> bit) & 1) == 1;
if (isSet) {
int shift = 8 * (bytes.length - 1 - i) + bit;
int mask = 1 << shift;
value |= mask;
}
}
}
return value;
}
Related
I am not getting the expected results when I tried to re-assemble the two bytes in Java (from C++ via JNI). Example below:
C++:
uint8_t originalValue = 0x38;
uint8_t high = originalValue & 0x0f0; //equals to 0x30
uint8_t low = originalValue << 4; //equals to 0x80
sendByte(high);
delayNs(1);
sendByte(low);
Java:
private int[] instructionRegister = new int[2];
private void processData(byte[] data) {
if (data.length <= 0)
return;
if (data.length == 1) {
int tmp = data[0];
String hexTmp = ByteUtils.toHexString(false, (byte) tmp).toUpperCase();
switch (tmp) {
case CMD_INSTRUCTION: {
log.info("\t>> INSTRUCTION: {}", hexTmp);
processingInstruction = true;
break;
}
case CMD_DATA: {
log.info("\t>> DATA: {}", hexTmp);
break;
}
default: {
log.info("\t\t- {}", hexTmp);
if (processingInstruction) {
if (instructionCount == 0) {
instructionRegister[0] = tmp;
instructionCount++;
} else {
instructionRegister[1] = tmp;
int combined = instructionRegister[0] | ((instructionRegister[1] >> 4);
log.info("\t>> Combined: ({}, {}) = {}",
ByteUtils.toHexString(false, (byte) instructionRegister[0]),
ByteUtils.toHexString(false, (byte) instructionRegister[1]),
ByteUtils.toHexString(false, (byte) combined));
instructionRegister = new int[2];
instructionCount = 0;
processingInstruction = false;
}
}
}
}
Result:
>> Combined: (30 , 80 ) = f8
I expected this to be 0x38 but got 0xF8. When I manually did the calculation in speedcrunch, I get the expected results.
Like I said in my comment, the problem is converting a signed byte holding 0x80 to an int (using a widening conversion), because the byte is negative, and the sign then gets extended, so you get ffffff80 in the int. If you then shift that to the right by 4 and convert back to a byte, you get: 0xf8, and if you binary-or that with 0x30 it's still 0xf8.
It looks like you're only doing such a conversion here:
int tmp = data[0];
You can change this line to:
int tmp = Byte.toUnsignedInt(data[0]);
To do a conversion as if the byte was unsigned.
To demonstrate the difference:
byte b = (byte) 0x80; // force overflow, making -128
int i1 = b;
int i2 = Byte.toUnsignedInt(b);
System.out.println(String.format("%x", b)); // 80
System.out.println(String.format("%x", i1)); // ffffff80
System.out.println(String.format("%x", i2)); // 80
I'm only starting to go into Swift development. I have the following method in Java:
public static byte[] addChecksum(byte[]command, boolean isDeviceSendFormat) {
int checksum = 0;
int l = command.length;
for (int i=0; i<l-2; i++) {
if (i==1 && isDeviceSendFormat==true) {
continue;
}
int val = command[i];
if (val < 0) {
val = 0x100 + val;
}
checksum += val;
}
if (l > 2) {
if (isDeviceSendFormat == false) {
command[l - 1] = (byte) (checksum % 0x100); // LSB
command[l - 2] = (byte) (checksum / 0x100); // MSB
}
else {
command[l - 2] = (byte) (checksum % 0x100); // LSB
command[l - 1] = (byte) (checksum / 0x100); // MSB
}
}
return command;
}
I need to translate to Swift and I'm having some problems, here is what I got so far:
func addCheckSum(bufferInput:[UInt8], isDeviceSendFormat: Bool) -> [UInt8]{
var checksum: UInt8 = 0
var length: Int = 0
var iIndex: Int
var bufferOutput: [UInt8]
length = bufferInput.count
for (index, value) in bufferInput.enumerated() {
if index < bufferInput.count - 2 {
if value == 1 && isDeviceSendFormat {
continue
}
var val:UInt8 = bufferInput[index]
if (val < 0) {
val = 0x100 + val //Error line
}
checksum = checksum + val
}
}
}
But I'm getting the following error: Integer literal '256' overflows when stored into 'UInt8' on the commented line in the code above. How to translate this method from Java to Swift?
This is my translation from your Java code into Swift:
public static func addChecksum(_ command: inout [UInt8], isDeviceSendFormat: Bool) -> [UInt8] {
var checksum: UInt32 = 0
let l: Int = command.count
for i in 0..<l-2 {
if i == 1 && isDeviceSendFormat {
continue
}
let val = UInt32(command[i])
//No need to modify `val` as it takes non-negative value when `command` is `[UInt8]`.
checksum += val
}
if l > 2 {
if !isDeviceSendFormat {
command[l - 1] = UInt8(checksum % 0x100) // LSB
command[l - 2] = UInt8(truncatingIfNeeded: checksum / 0x100) // Next to LSB
} else {
command[l - 2] = UInt8(checksum % 0x100) // LSB
command[l - 1] = UInt8(truncatingIfNeeded: checksum / 0x100) // Next to LSB
}
}
return command
}
//Assuming `command` is not too long as to make integer overflow in `checksum += val`.
Some notes:
These 3 lines of your Java code:
if (val < 0) {
val = 0x100 + val;
}
convert the value -128...127, to 0...255 by adding 0x100(=256) when the val is negative. So, val takes any value in 0...255, so, I choose [UInt8] for command. When you choose UInt8, the 3 lines above in Java is not needed in Swift.
In your Swift code, you have chosen UInt8 for checksum and val, but int in Java is 32-bit long and I choose UInt32 for them. Assuming integer overflow may never happen, they take only non-negative values, so non-negative 32-bit long integer would be appropriate.
There's no direct equivalent of byte[] of Java in Swift. So, in some cases [UInt8] is more appropriate than [Int8]. And you can find many cases that Java's byte[] are translated into Data of Swift.
I have gone through the similar posts but none of them really answered my question. Hence I post my question in a separate thread.
I need to skip some bytes in the file which I am reading in the byte array.I am trying to achieve this through code below
1. byte [] readBytesToSKip = null;
2. readBytesToSKip = new byte[(int)bytesToSkip];
3. bytesReadToSkip = System.in.read(readBytesToSKip) ;
4. if(bytesReadToSkip > 0)
5. {
6. baos_.write(readBytesToSKip, 0, bytesReadToSkip);
7. }
But I get a NegativeArraySizeException at line 2 where the size exceeds Integer.MAX_VALUE. I am not sure how to achieve this otherwise.
bytesToSkip is long as I calculate in function below:
public static long bytesToLong1(byte[] bytes) {
long value = 0;
for (int i = 0; i < bytes.length; i++)
{
value = (value << 8) + (bytes[i] & 0xff);
}
return value;
}
Starting from your posted bytes->long function
public static long bytesToLong1(byte[] bytes) {
long value = 0;
for (int i = 0; i < bytes.length; i++)
{
value = (value << 8) + (bytes[i] & 0xff);
}
return value;
}
Whenever bytes.length==8 and the first byte is >= 128 in absolute value (in Java specific: it's a negative byte value), you'll end having a negative value for your long.
Example:
byte vals[]={0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF};
System.out.println(bytesToLong1(vals));
will result it -1 being printed. Which will convert quite fine to an int with a -1 value, but will still be improper for the length of an array.
I encountered with the issue with CRC16 algorithm. There is a string of hex 80 01 F0, after CRC16 I get the low byte = 23 and the high one = 80. So, the question is how to calculate these two bytes? I tried the CRC calculators but there was no result. Also, it would be perfect if there is an example of this method in Java.
In manual there is additional information:
Low and high byte of a forward CRC-16 algorithm using the Polynomial (X16 + X15 + X2 + 1) calculated on all bytes. It is initialised using the seed 0xFFFF.
Thank you for responses. I am confident my answer will be useful for others. Tested and working code.
private static byte[] getCRC16LowHighBytes(byte[] byteSequence) {
// Create a byte array for Low and High bytes
byte[] returnBytes = new byte[2];
int crc = CRC16_SEED;
for (int i = 0; i < byteSequence.length; ++i) {
crc ^= (byteSequence[i] << 8);
for (int j = 0; j < 8; ++j) {
if ((crc & 0x8000) != 0) {
crc = (crc << 1) ^ CRC16_POLINOM;
} else {
crc <<= 1;
}
}
}
byte[] crcBytes = getBytes(crc);
// The first two bytes of crcBytes are low and high bytes respectively.
for (int i = 0; i < returnBytes.length; i++) {
returnBytes[i] = crcBytes[i];
}
return returnBytes;
}
private static byte[] getBytes(int v) {
byte[] writeBuffer = new byte[4];
writeBuffer[3] = (byte) ((v >>> 24) & 0xFF);
writeBuffer[2] = (byte) ((v >>> 16) & 0xFF);
writeBuffer[1] = (byte) ((v >>> 8) & 0xFF);
writeBuffer[0] = (byte) ((v >>> 0) & 0xFF);
return writeBuffer;
}
How do I shift a byte array n positions to the right? For instance shifting a 16 byte array right 29 positions? I read somewhere it can be done using a long? Would using a long work like this:
Long k1 = byte array from 0 to 7
Long k2 = byte array from 8 to 15
Then right rotating these two longs using Long.rotateRight(Long x, number of rotations).How would the two longs be joined back into a byte array?
I believe you can do this using java.math.BigInteger which supports shifts on arbitrarily large numbers. This has advantage of simplicity, but disadvantage of not padding into original byte array size, i.e. input could be 16 bytes but output might only be 10 etc, requiring additional logic.
BigInteger approach
byte [] array = new byte[]{0x7F,0x11,0x22,0x33,0x44,0x55,0x66,0x77};
// create from array
BigInteger bigInt = new BigInteger(array);
// shift
BigInteger shiftInt = bigInt.shiftRight(4);
// back to array
byte [] shifted = shiftInt.toByteArray();
// print it as hex
for (byte b : shifted) {
System.out.print(String.format("%x", b));
}
Output
7f1122334455667 <== shifted 4 to the right. Looks OK
Long manipulation
I don't know why you'd want to do this as rotateRight() as this makes life more difficult, you have to blank at the bits that appear at the left hand side in K1 etc. You'd be better with using shift IMO as describe below. I've used a shift of 20 as divisible by 4 so easier to see the nibbles move in the output.
1) Use ByteBuffer to form two longs from 16 byte array
byte[] array = { 0x00, 0x00, 0x11, 0x11, 0x22, 0x22, 0x33, 0x33, 0x44, 0x44, 0x55, 0x55, 0x66, 0x66, 0x77, 0x77 };
ByteBuffer buffer = ByteBuffer.wrap(array);
long k1 = buffer.getLong();
long k2 = buffer.getLong();
2) Shift each long n bits to the right
int n = 20;
long k1Shift = k1 >> n;
long k2Shift = k2 >> n;
System.out.println(String.format("%016x => %016x", k1, k1Shift));
System.out.println(String.format("%016x => %016x", k2, k2Shift));
0000111122223333 => 0000000001111222
4444555566667777 => 0000044445555666
Determine bits from k1 that "got pushed off the edge"
long k1CarryBits = (k1 << (64 - n));
System.out.println(String.format("%016x => %016x", k1, k1CarryBits));
0000111122223333 => 2333300000000000
Join the K1 carry bits onto K2 on right hand side
long k2WithCarray = k2Shift | k1CarryBits;
System.out.println(String.format("%016x => %016x", k2Shift, k2WithCarray));
0000044445555666 => 2333344445555666
Write the two longs back into a ByteBuffer and extract as a byte array
buffer.position(0);
buffer.putLong(k1Shift);
buffer.putLong(k2WithCarray);
for (byte each : buffer.array()) {
System.out.print(Long.toHexString(each));
}
000011112222333344445555666
Here is what I came up with to shift a byte array by some arbitrary number of bits left:
/**
* Shifts input byte array len bits left.This method will alter the input byte array.
*/
public static byte[] shiftLeft(byte[] data, int len) {
int word_size = (len / 8) + 1;
int shift = len % 8;
byte carry_mask = (byte) ((1 << shift) - 1);
int offset = word_size - 1;
for (int i = 0; i < data.length; i++) {
int src_index = i+offset;
if (src_index >= data.length) {
data[i] = 0;
} else {
byte src = data[src_index];
byte dst = (byte) (src << shift);
if (src_index+1 < data.length) {
dst |= data[src_index+1] >>> (8-shift) & carry_mask;
}
data[i] = dst;
}
}
return data;
}
1. Manually implemented
Here are left and right shift implementation without using BigInteger (ie. without creating a copy of the input array) and with unsigned right shift (BigInteger only supports arithmetic shifts of course)
Left Shift <<
/**
* Left shift of whole byte array by shiftBitCount bits.
* This method will alter the input byte array.
*/
static byte[] shiftLeft(byte[] byteArray, int shiftBitCount) {
final int shiftMod = shiftBitCount % 8;
final byte carryMask = (byte) ((1 << shiftMod) - 1);
final int offsetBytes = (shiftBitCount / 8);
int sourceIndex;
for (int i = 0; i < byteArray.length; i++) {
sourceIndex = i + offsetBytes;
if (sourceIndex >= byteArray.length) {
byteArray[i] = 0;
} else {
byte src = byteArray[sourceIndex];
byte dst = (byte) (src << shiftMod);
if (sourceIndex + 1 < byteArray.length) {
dst |= byteArray[sourceIndex + 1] >>> (8 - shiftMod) & carryMask;
}
byteArray[i] = dst;
}
}
return byteArray;
}
Unsigned Right Shift >>>
/**
* Unsigned/logical right shift of whole byte array by shiftBitCount bits.
* This method will alter the input byte array.
*/
static byte[] shiftRight(byte[] byteArray, int shiftBitCount) {
final int shiftMod = shiftBitCount % 8;
final byte carryMask = (byte) (0xFF << (8 - shiftMod));
final int offsetBytes = (shiftBitCount / 8);
int sourceIndex;
for (int i = byteArray.length - 1; i >= 0; i--) {
sourceIndex = i - offsetBytes;
if (sourceIndex < 0) {
byteArray[i] = 0;
} else {
byte src = byteArray[sourceIndex];
byte dst = (byte) ((0xff & src) >>> shiftMod);
if (sourceIndex - 1 >= 0) {
dst |= byteArray[sourceIndex - 1] << (8 - shiftMod) & carryMask;
}
byteArray[i] = dst;
}
}
return byteArray;
}
Used in this class by this Project.
2. Using BigInteger
Be aware that BigInteger internally converts the byte array into an int[] array so this may not be the most optimized solution:
Arithmetic Left Shift <<:
byte[] result = new BigInteger(byteArray).shiftLeft(3).toByteArray();
Arithmetic Right Shift >>:
byte[] result = new BigInteger(byteArray).shiftRight(2).toByteArray();
3. External Library
Using the Bytes java library*:
Add to pom.xml:
<dependency>
<groupId>at.favre.lib</groupId>
<artifactId>bytes</artifactId>
<version>{latest-version}</version>
</dependency>
Code example:
Bytes b = Bytes.wrap(someByteArray);
b.leftShift(3);
b.rightShift(3);
byte[] result = b.array();
*Full Disclaimer: I am the developer.
The is an old post, but I want to update Adam's answer.
The long solution works with a few tweak.
In order to rotate, use >>> instead of >>, because >> will pad with significant bit, changing the original value.
second, the printbyte function seems to miss leading 00 when it prints.
use this instead.
private String getHexString(byte[] b) {
StringBuilder result = new StringBuilder();
for (int i = 0; i < b.length; i++)
result.append(Integer.toString((b[i] & 0xff) + 0x100, 16)
.substring(1));
return result.toString();
}