I never write here, but as almost everyone I often read questions and answers... now it's my time to ask :)
So, I was reading a long piece of code that works on a stream of bytes to decode tags and values of ePassports (ICAO Doc 9303 defines the International Technical Specifications)... it wasn't difficult until I found this part (I placed comments before and after the cryptic part)
int len = s.read(); // s is an InputStream
readPos++;
if ((len > 0x80)) {
int lenlen = len - 0x80;
len = 0;
for (int i = 0; (i < lenlen); i++) {
if ((readPos == length)) {
throw new ParseException();
}
// wtf begin
len = (len << 8) | ((byte) (s.read()));
// wtf end
readPos++;
}
}
size = readPos + len;
I understand that it reads from an InputStream, byte by byte, and increases the position in the byte stream, but why does it perform the shift and save the result in the variable len? If I'm not wrong, len gets overwritten over and over, so how could this variable have a meaningful value after the for loop?
It is processing lenlen number of bytes, and making out of those bytes a big endian number:
case len was <= 128
if is not entered, and len represents a single byte number
case len == 128+1, +2, +3, +4
Respectively 1, 2, 3, 4 bytes are used: every byte is placed at the least
significant position and where the older bytes are shifted a byte position to the left (<< 8). So 129 for a 1 byte length (for numbers between 128 and 255),
and 130 for a 2 bytes number.
at the end: len bytes are skipped.
Data integrity would require to check for the original len >= 0, and len <= 128 + 4 and for the calculated len being >= 0 too.
The writing process The inverse algorithm you might encounter:
if (len <= 128) {
out.write((byte)len);
} else {
int bytes = 4 - Integer.numberOfLeadingZeroes(len) / 8;
out.write(128 + bytes);
for (int i = 0; i < bytes; ++i) {
int b = len >>>(bytes - 1);
out.write((byte)b); // Truncates the int to just 1 byte.
}
}
(Assuming len is positive.)
Related
I was going through a piece of code in the Apache commons library and was wondering what these conditions do exactly.
public static byte[] decodeHex(final char[] data) throws DecoderException {
final int len = data.length;
if ((len & 0x01) != 0) { // what does this condition do
throw new DecoderException("Odd number of characters.");
}
final byte[] out = new byte[len >> 1];
// two characters form the hex value.
for (int i = 0, j = 0; j < len; i++) {
int f = toDigit(data[j], j) << 4;
j++;
f = f | toDigit(data[j], j);
j++;
out[i] = (byte) (f & 0xFF); // what is happening here.
}
return out;
}
thanks in advance.
This checks if the last digit in the binary writing of len is a 1.
xxxxxxxy
& 00000001
gives 1 if y is 1, 0 if y is 0, ignoring the other digits.
If y is 1, the length of the char array is odd, which shouldn't happen in this hex writing, hence the exception.
Another solution would have been
if (len%2 != 0) {
which would have been clearer in my opinion. I doubt the slight performance increase just before a loop really matters.
It's a 1337 (high performance) way of coding:
if (len % 2 == 1)
i.e. is len odd. It works because the binary representation of every odd integer has its least significant (ie last) bit set. Performaning a bitwise AND with 1 masks all other bits, leaving a result of either 1 if it's odd or 0 if even.
It's a carryover from C, where you can code simply:
if (len & 1)
This line checks if len is an odd number or not.
If len isn't odd, len & 1 will be equal to 0. (1 and 0x01 are the same value, 0x01 is just the hexadecimal notation)
I have the following array of bytes:
01010110 01110100 00100101 01001011
These bytes are broken into two groups to encode seven integers. I know that the first group consists of 3 values 4 bits each (0101 0110 0111) that represent numbers 5,6,7. The second group consists of 4 values 5 bits each (01000 01001 01010 01011), which represent integers 8,9,10, and 11.
To extract the integers, I am currently using the following approach. Convert the array into a binary string:
public static String byteArrayToBinaryString(byte[] byteArray)
{
String[] arrayOfStrings = new String[byteArray.length];
for(int i=0; i<byteArray.length; i++)
{
arrayOfStrings[i] = byteToBinaryString(byteArray[i]);
}
String bitsetString = "";
for(String testArrayStringElement : arrayOfStrings)
{
bitsetString += testArrayStringElement;
}
return bitsetString;
}
// Taken from here: http://helpdesk.objects.com.au/java/converting-large-byte-array-to-binary-string
public static String byteToBinaryString(byte byteIn)
{
StringBuilder sb = new StringBuilder("00000000");
for (int bit = 0; bit < 8; bit++)
{
if (((byteIn >> bit) & 1) > 0)
{
sb.setCharAt(7 - bit, '1');
}
}
return sb.toString();
}
Then, I split the binary string into 2 substrings: 12 characters and 20 characters. Then I split each substring into new substrings, each of which has length that equals the number of bits. Then I convert each sub-substring into an integer.
It works but a byte array representing thousands of integers takes 30 seconds to a minute to extract.
I am a bit at a loss here. How do I do this using bitwise operators?
Thanks a lot!
I assume you have an understanding of the basic bit operations and how to express them in Java.
Use a pencil to draw a synthetic picture of the problem
byte 0 byte 1 byte 2 byte 3
01010110 01110100 00100101 01001011
\__/\__/ \__/\______/\___/\______/\___/
a b c d e f g
To extract a, b and c we need to do the following
a b c
byte 0 byte 0 byte 1
01010110 01010110 01110100
\. \. |||||||| \. \.
'\ '\ XXXX|||| '\ '\
0.. 0101 0.. 0110 0.. 0111
Shift And Shift
In Java
int a = byteArray[0] >>> 4, b = byteArray[0] & 0xf, c = byteArray[1] >>> 4;
The other values d, e, f and g are computed similarly but some of them require to read two bytes from the array (d and f actually).
d e
byte 1 byte 2 byte 2
01110100 00100101 00100101
||||\\\\ | |\\\\\
XXXX \\\\ | X \\\\\
\\\\| \\\\\
0.. 01000 01001
To compute d we need to isolate the least four bits of byte 1 with byteArray[1] & 0xf then make space for the bit from byte 2 with (byteArray[1] & 0xf) << 1, extract that bit with byteArray[1] >>> 7 and finally merge together the result.
int d = (byteArray[1] & 0xf) << 1 | byteArray[2] >>> 7;
int e = (byteArray[2] & 0x7c) >>> 2;
int f = (byteArray[2] & 0x3) << 3 | byteArray[3] >>> 5;
int g = byteArray[3] & 0x1f;
When you are comfortable with handling bits operations you may consider generalizing the function that extract the integers.
I made function int extract(byte[] bits, int[] sizes, int[] res), that given an array of bytes bits, an array of sizes sizes, where the even indices hold the size of the integers to extract in bits and the odd indices the number of integers to extract, and an output array res large enough to hold all the integers in output, extracts from bits all the integers expressed by sizes.
It returns the number of integers extracted.
For example the original problem can be solved as
int res[] = new int[8];
byte bits[] = new byte[]{0x56, 0x74, 0x25, 0x4b};
//Extract 3 integers of 4 bits and 4 integers of 5 bits
int ints = BitsExtractor.extract(bits, new int[]{4, 3, 5, 4}, res);
public class BitsExtractor
{
public static int extract(byte[] bits, int[] sizes, int[] res)
{
int currentByte = 0; //Index into the bits array
int intProduced = 0; //Number of ints produced so far
int bitsLeftInByte = 8; //How many bits left in the current byte
int howManyInts = 0; //Number of integers to extract
//Scan the sizes array two items at a time
for (int currentSize = 0; currentSize < sizes.length - 1; currentSize += 2)
{
//Size, in bits, of the integers to extract
int intSize = sizes[currentSize];
howManyInts += sizes[currentSize+1];
int temp = 0; //Temporary value of an integer
int sizeLeft = intSize; //How many bits left to extract
//Do until we have enough integer or we exhaust the bits array
while (intProduced < howManyInts && currentByte <= bits.length)
{
//How many bit we can extract from the current byte
int bitSize = Math.min(sizeLeft, bitsLeftInByte); //sizeLeft <= bitsLeftInByte ? sizeLeft : bitsLeftInByte;
//The value to mask out the number of bit extracted from
//The current byte (e.g. for 3 it is 7)
int byteMask = (1 << bitSize) - 1;
//Extract the new bits (Note that we extract starting from the
//RIGHT so we need to consider the bits left in the byte)
int newBits = (bits[currentByte] >>> (bitsLeftInByte - bitSize)) & byteMask;
//Create the new temporary value of the current integer by
//inserting the bits in the lowest positions
temp = temp << bitSize | newBits;
//"Remove" the bits processed from the byte
bitsLeftInByte -= bitSize;
//Is the byte has been exhausted, move to the next
if (bitsLeftInByte == 0)
{
bitsLeftInByte = 8;
currentByte++;
}
//"Remove" the bits processed from the size
sizeLeft -= bitSize;
//If we have extracted all the bits, save the integer
if (sizeLeft == 0)
{
res[intProduced++] = temp;
temp = 0;
sizeLeft = intSize;
}
}
}
return intProduced;
}
}
Well I did the first group , the second can be done in similar fashion
public static void main(String args[]) {
//an example 32 bits like your example
byte[] bytes = new byte[4];
bytes[0] = 31;//0001 1111
bytes[1] = 54;//0011 0110
bytes[2] = 67;
bytes[3] = 19;
//System.out.println(bytes[0]);
int x = 0;
int j = -1; // the byte number
int k = 0; // the bit number in that byte
int n = 0; // the place of the bit in the integer we are trying to read
for (int i = 0; i < 32; i++) {
if (i < 12) { //first group
if (i % 8 == 0) {
j++;
k = 0;
}
if (i % 4 == 0) {
x = 0;
n = 0;
}
byte bit = (byte) ((bytes[j] & (1 << (7 - k))) >> (7 - k));
System.out.println("j is :" + j + " k is :" + k + " " + bit);
x = x | bit << (3 - n);
if ((i + 1) % 4 == 0) {
System.out.println(x);
}
k++;
n++;
} else {
}
}
}
It's a bit tricky because you are trying to encode an integer on less than what java allocates (8 bits). So I had to take each bit and "construct" the int from them
To get each bit
byte bit = (byte) ((bytes[j] & (1 << (7 - k))) >> (7 - k));
this takes the byte we are at and does And operation. For example I want the 3rd bit of the 1st byte, I do
bytes[0] & 1 << (7 - 3)
but this gives me an integer encoded over 8 bits, so I still have to shift it to get that single bit with >> (7 - 3)
Then I just Or it with x (the int we are trying to decode). All while putting it at the right position with << (3 - n) . 3 because your integer is encoded over 4 bits
Try running the code and reading the output.
I am honestly not sure if this is the best way, but I believe it's at least faster than dealing with Strings
Working with BitSets I have a failing test:
BitSet bitSet = new BitSet();
bitSet.set(1);
bitSet.set(100);
logger.info("BitSet: " + BitSetHelper.toString(bitSet));
BitSet fromByteArray = BitSetHelper.fromByteArray(bitSet.toByteArray());
logger.info("fromByteArray: " + BitSetHelper.toString(bitSet));
Assert.assertEquals(2, fromByteArray.cardinality());
Assert.assertTrue(fromByteArray.get(1)); <--Assertion fail!!!
Assert.assertTrue(fromByteArray.get(100)); <--Assertion fail!!!
To be more weird I can see my String representation of both BitSets:
17:34:39.194 [main] INFO c.i.uniques.helper.BitSetHelperTest - BitSet: 00000010000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000010000
17:34:39.220 [main] INFO c.i.uniques.helper.BitSetHelperTest - fromByteArray: 00000010000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000010000
Are equals! What's happening here??
The used methods on this example are:
public static BitSet fromByteArray(byte[] bytes) {
BitSet bits = new BitSet();
for (int i = 0; i < bytes.length * 8; i++) {
if ((bytes[bytes.length - i / 8 - 1] & (1 << (i % 8))) > 0) {
bits.set(i);
}
}
return bits;
}
And the method used to get the String representation:
public static String toString(BitSet bitSet) {
StringBuffer buffer = new StringBuffer();
for (byte b : bitSet.toByteArray()) {
buffer.append(String.format("%8s", Integer.toBinaryString(b & 0xFF)).replace(' ', '0'));
}
return buffer.toString();
}
Some one could explain what's going on here?
Note that BitSet has a valueOf(byte[]) that already does this for you.
Inside your fromByteArray method
for (int i = 0; i < bytes.length * 8; i++) {
if ((bytes[bytes.length - i / 8 - 1] & (1 << (i % 8))) > 0) {
bits.set(i);
}
}
you're traversing your byte[] in reverse. On the first iteration,
bytes.length - i / 8 - 1
will evaluate to
8 - (0 / 8) - 1
which is 7, which will access the most significant byte. This is the one containing the 100th bit from your original bitset. Viewed from the reverse side, this is the fourth bit. And if you check the bits set in your generated BitSet, you'll notice the 5th and 98th (there might be an off by one bug here) bits are set.
But the byte[] returned by toByteArray() contains
a little-endian representation of all the bits in this bit set
You need to read the byte[] in the appropriate order
for (int i = 0; i < bytes.length * 8; i++) {
if ((bytes[i / 8] & (1 << (i % 8))) > 0) {
bits.set(i);
}
}
So say I have a byte array, and I have a function that checks whether the n-th least significant bit index of the byte array is a 1 or a 0. The function returns true if the bit is a 1 and false if the bit is a 0. The least significant bit of the byte array is defined as the last significant bit in the 0th index of the byte array, and the most significant bit of the byte array is defined as the most significant bit in the (byte array.length - 1)th index of the byte array.
For instance,
byte[] myArray = new byte[2];
byte[0] = 0b01111111;
byte[1] = 0b00001010;
Calling:
myFunction(0) = true;
myFunction(1) = true;
myFunction(7) = false;
myFunction(8) = false;
myFunction(9) = true;
myFunction(10) = false;
myFunction(11) = true;
What is the best way to do this?
Thanks!
You can use this method:
public boolean isSet(byte[] arr, int bit) {
int index = bit / 8; // Get the index of the array for the byte with this bit
int bitPosition = bit % 8; // Position of this bit in a byte
return (arr[index] >> bitPosition & 1) == 1;
}
bit % 8 is the bit position relative to a byte.
arr[index] >> bit % 8 moves the bit at index to bit 0 position.
I'm trying to make a reliable UDP system and I need to convert byte[] to int and back (JCreator 5.0 LE)
DatagramPacket requires its data in byte[] form, so I have to convert the int information into byte[] so I can send it:
byte[] data = new byte[48];
System.arraycopy(task.getProtocol().byteValue(), 0, data, 0, task.getProtocol().byteValue().length);
System.arraycopy(task.getMessage().byteValue(), 0, data, 4, task.getMessage().byteValue().length);
System.arraycopy(task.getSequence().byteValue(), 0, data, 8, task.getSequence().byteValue().length);
System.arraycopy(task.getAcknowledge().byteValue(), 0, data, 12, task.getAcknowledge().byteValue().length);
for (int i = task.getAcknowledge(); i >= 0 && i > task.getAcknowledge() - 33; i--) {
for (Packet j: tasks) {
if (j.getSequence() == i) {
data[i] = 1;
break;
}
}
}
out = new DatagramPacket(data, data.length, ds.getInetAddress(), portNum);
ds.send(out);
Protocol is the protocolID
Message is the "information" that is being sent
Sequence is the packet's sequence number; the first packet sent has a sequence of 0, the next is 1, and so on
Acknowledge is the acknowledgement of a packet being sent back
The next part is the 32 other acknowledgements. For the sake of saving memory, they are compressed into 1 byte each instead of 4 bytes (int)
Now, when I receive the packet I need to unpackage it. First I need to check the first 4 bytes (the protocol) to see if I will ignore the packet or not, but I don't know how to convert the byte array into an int.
You can use
ByteBuffer bb = ByteBuffer.wrap(bytes).order(ByteOrder.LITTLE_ENDIAN or BIG_ENDIAN);
bb.position(pos);
int n = bb.getInt();
byte array - > String -> Integer
Byte array - 4 bytes;
String - new String(byte array);
Integer -
Integer.parseInt(String);
Well, here's a general way to do it; it it's a 32-bit Big-Endian (the usual 'network order'), starting at position 'pos' in your array:
int out = 0;
out += 0xFF & data[pos++];
out <<= 8;
out += 0xFF & data[pos++];
out <<= 8;
out += 0xFF & data[pos++];
out <<= 8;
out += 0xFF & data[pos++];
But this can be adapted to the number of bytes used for your integers. I'd make methods to call, returning 'out'. Look out for bugs due to sign. The "0xFF &" is there to avoid those. Also, not sure I got the <<= thing right.
If they're little-endian, well a bit harder:
int out = 0;
pos += 4;
out += 0xFF & data[--pos];
out <<= 8;
out += 0xFF & data[--pos];
out <<= 8;
out += 0xFF & data[--pos];
out <<= 8;
out += 0xFF & data[--pos];
These are just one way to do it. (Disclaimer again: untested.)