We Keep Coding

Do you understand what this code is aiming to do?

I never write here, but as almost everyone I often read questions and answers... now it's my time to ask :)
So, I was reading a long piece of code that works on a stream of bytes to decode tags and values of ePassports (ICAO Doc 9303 defines the International Technical Specifications)... it wasn't difficult until I found this part (I placed comments before and after the cryptic part)
int len = s.read(); // s is an InputStream
readPos++;
if ((len > 0x80)) {
int lenlen = len - 0x80;
len = 0;
for (int i = 0; (i < lenlen); i++) {
if ((readPos == length)) {
throw new ParseException();
}
// wtf begin
len = (len << 8) | ((byte) (s.read()));
// wtf end
readPos++;
}
}
size = readPos + len;
I understand that it reads from an InputStream, byte by byte, and increases the position in the byte stream, but why does it perform the shift and save the result in the variable len? If I'm not wrong, len gets overwritten over and over, so how could this variable have a meaningful value after the for loop?

It is processing lenlen number of bytes, and making out of those bytes a big endian number:
case len was <= 128
if is not entered, and len represents a single byte number
case len == 128+1, +2, +3, +4
Respectively 1, 2, 3, 4 bytes are used: every byte is placed at the least
significant position and where the older bytes are shifted a byte position to the left (<< 8). So 129 for a 1 byte length (for numbers between 128 and 255),
and 130 for a 2 bytes number.
at the end: len bytes are skipped.
Data integrity would require to check for the original len >= 0, and len <= 128 + 4 and for the calculated len being >= 0 too.
The writing process The inverse algorithm you might encounter:
if (len <= 128) {
out.write((byte)len);
} else {
int bytes = 4 - Integer.numberOfLeadingZeroes(len) / 8;
out.write(128 + bytes);
for (int i = 0; i < bytes; ++i) {
int b = len >>>(bytes - 1);
out.write((byte)b); // Truncates the int to just 1 byte.
}
}
(Assuming len is positive.)

Byte Operations and Datagrams in Java

I am trying to program a handshake type message as follows where C=Client S=Server:
C-->S: "I'd like to talk" //session initiation
S-->C: "80" //(n, randomly generated number)
C-->S: "81" //(n+1)
S: "does n= (n+1)-1?" //confirms the connection.
For the purposes of this question assume that the logic above is correct. I would like the random number I generated to be a 32 bit number (i.e. 4 bytes sent in a UDP datagram). Since an int is 32 bits, I would prefer to use this data type, but I seem to run into one of two problems:
When using an array of bytes, it is easy to send in a datagram but difficult to perform a simple math operation (such as subtract 1) for a 32 bit number.
When using an int it is easy to perform a simple math operation, but it is difficult to convert between ints and bytes when sending back and forth between the client and server.
I found a method that can convert from an int to bytes. I found some information regarding using a Bytebuffer to convert to an int, but I'm not sure it's even correct. Is there an easier way to go about a process of sending an int in a datagram? It seems like an extraordinary amount of work to keep converting back and forth between bytes and ints.

Nothing hard about any of those operations. DataInputStream and DataOutputStream take care of the stream->int->stream conversions, and ByteArrayInputStream and ByteArrayOutputStream take care of the stream->byte[]->stream conversions.

There are two options:
the above mentioned bytebuffer
converting via bitshift:
//int to byte[]
int val = someval;
byte[] bytes = new byte[4];
for(int i = 0 ; i < 4 ; i++)
bytes[i] = (byte) (val >>> (i * 8));
//byte[] to int
int val = 0;
byte[] bytes = input();
for(int i = 0 ; i < 4 ; i++)
val |= ((int)(bytes[i])) << i * 8;

If you are defining your own format of the datagram, it's easy enough to establish that the nth 4 bytes of content represent an integer.
You then can use some simple conversion functions to go from int to byte[] and vice-versa.
A small class implementing this two methods should do:
public static byte[] toByteArray(int value) {
byte[] b = new byte[4];
// MSB to LSB
b[0] = (byte) (value >> 24);
b[1] = (byte) (value >> 16);
b[2] = (byte) (value >> 8);
b[3] = (byte) (value);
return b;
}
public static int fromByteArray(byte[] value) {
int i = ((((int) value[0]) & 0xFF) << 24) |
((((int) value[1]) & 0xFF) << 16) |
((((int) value[2]) & 0xFF) << 8) |
((((int) value[3] & 0xFF)));
return i;
}

Bit shift operations on a byte array in Java

How do I shift a byte array n positions to the right? For instance shifting a 16 byte array right 29 positions? I read somewhere it can be done using a long? Would using a long work like this:
Long k1 = byte array from 0 to 7
Long k2 = byte array from 8 to 15
Then right rotating these two longs using Long.rotateRight(Long x, number of rotations).How would the two longs be joined back into a byte array?

I believe you can do this using java.math.BigInteger which supports shifts on arbitrarily large numbers. This has advantage of simplicity, but disadvantage of not padding into original byte array size, i.e. input could be 16 bytes but output might only be 10 etc, requiring additional logic.
BigInteger approach
byte [] array = new byte[]{0x7F,0x11,0x22,0x33,0x44,0x55,0x66,0x77};
// create from array
BigInteger bigInt = new BigInteger(array);
// shift
BigInteger shiftInt = bigInt.shiftRight(4);
// back to array
byte [] shifted = shiftInt.toByteArray();
// print it as hex
for (byte b : shifted) {
System.out.print(String.format("%x", b));
}
Output
7f1122334455667 <== shifted 4 to the right. Looks OK
Long manipulation
I don't know why you'd want to do this as rotateRight() as this makes life more difficult, you have to blank at the bits that appear at the left hand side in K1 etc. You'd be better with using shift IMO as describe below. I've used a shift of 20 as divisible by 4 so easier to see the nibbles move in the output.
1) Use ByteBuffer to form two longs from 16 byte array
byte[] array = { 0x00, 0x00, 0x11, 0x11, 0x22, 0x22, 0x33, 0x33, 0x44, 0x44, 0x55, 0x55, 0x66, 0x66, 0x77, 0x77 };
ByteBuffer buffer = ByteBuffer.wrap(array);
long k1 = buffer.getLong();
long k2 = buffer.getLong();
2) Shift each long n bits to the right
int n = 20;
long k1Shift = k1 >> n;
long k2Shift = k2 >> n;
System.out.println(String.format("%016x => %016x", k1, k1Shift));
System.out.println(String.format("%016x => %016x", k2, k2Shift));
0000111122223333 => 0000000001111222
4444555566667777 => 0000044445555666
Determine bits from k1 that "got pushed off the edge"
long k1CarryBits = (k1 << (64 - n));
System.out.println(String.format("%016x => %016x", k1, k1CarryBits));
0000111122223333 => 2333300000000000
Join the K1 carry bits onto K2 on right hand side
long k2WithCarray = k2Shift | k1CarryBits;
System.out.println(String.format("%016x => %016x", k2Shift, k2WithCarray));
0000044445555666 => 2333344445555666
Write the two longs back into a ByteBuffer and extract as a byte array
buffer.position(0);
buffer.putLong(k1Shift);
buffer.putLong(k2WithCarray);
for (byte each : buffer.array()) {
System.out.print(Long.toHexString(each));
}
000011112222333344445555666

Here is what I came up with to shift a byte array by some arbitrary number of bits left:
/**
* Shifts input byte array len bits left.This method will alter the input byte array.
*/
public static byte[] shiftLeft(byte[] data, int len) {
int word_size = (len / 8) + 1;
int shift = len % 8;
byte carry_mask = (byte) ((1 << shift) - 1);
int offset = word_size - 1;
for (int i = 0; i < data.length; i++) {
int src_index = i+offset;
if (src_index >= data.length) {
data[i] = 0;
} else {
byte src = data[src_index];
byte dst = (byte) (src << shift);
if (src_index+1 < data.length) {
dst |= data[src_index+1] >>> (8-shift) & carry_mask;
}
data[i] = dst;
}
}
return data;
}

1. Manually implemented
Here are left and right shift implementation without using BigInteger (ie. without creating a copy of the input array) and with unsigned right shift (BigInteger only supports arithmetic shifts of course)
Left Shift <<
/**
* Left shift of whole byte array by shiftBitCount bits.
* This method will alter the input byte array.
*/
static byte[] shiftLeft(byte[] byteArray, int shiftBitCount) {
final int shiftMod = shiftBitCount % 8;
final byte carryMask = (byte) ((1 << shiftMod) - 1);
final int offsetBytes = (shiftBitCount / 8);
int sourceIndex;
for (int i = 0; i < byteArray.length; i++) {
sourceIndex = i + offsetBytes;
if (sourceIndex >= byteArray.length) {
byteArray[i] = 0;
} else {
byte src = byteArray[sourceIndex];
byte dst = (byte) (src << shiftMod);
if (sourceIndex + 1 < byteArray.length) {
dst |= byteArray[sourceIndex + 1] >>> (8 - shiftMod) & carryMask;
}
byteArray[i] = dst;
}
}
return byteArray;
}
Unsigned Right Shift >>>
/**
* Unsigned/logical right shift of whole byte array by shiftBitCount bits.
* This method will alter the input byte array.
*/
static byte[] shiftRight(byte[] byteArray, int shiftBitCount) {
final int shiftMod = shiftBitCount % 8;
final byte carryMask = (byte) (0xFF << (8 - shiftMod));
final int offsetBytes = (shiftBitCount / 8);
int sourceIndex;
for (int i = byteArray.length - 1; i >= 0; i--) {
sourceIndex = i - offsetBytes;
if (sourceIndex < 0) {
byteArray[i] = 0;
} else {
byte src = byteArray[sourceIndex];
byte dst = (byte) ((0xff & src) >>> shiftMod);
if (sourceIndex - 1 >= 0) {
dst |= byteArray[sourceIndex - 1] << (8 - shiftMod) & carryMask;
}
byteArray[i] = dst;
}
}
return byteArray;
}
Used in this class by this Project.
2. Using BigInteger
Be aware that BigInteger internally converts the byte array into an int[] array so this may not be the most optimized solution:
Arithmetic Left Shift <<:
byte[] result = new BigInteger(byteArray).shiftLeft(3).toByteArray();
Arithmetic Right Shift >>:
byte[] result = new BigInteger(byteArray).shiftRight(2).toByteArray();
3. External Library
Using the Bytes java library*:
Add to pom.xml:
<dependency>
<groupId>at.favre.lib</groupId>
<artifactId>bytes</artifactId>
<version>{latest-version}</version>
</dependency>
Code example:
Bytes b = Bytes.wrap(someByteArray);
b.leftShift(3);
b.rightShift(3);
byte[] result = b.array();
*Full Disclaimer: I am the developer.

The is an old post, but I want to update Adam's answer.
The long solution works with a few tweak.
In order to rotate, use >>> instead of >>, because >> will pad with significant bit, changing the original value.
second, the printbyte function seems to miss leading 00 when it prints.
use this instead.
private String getHexString(byte[] b) {
StringBuilder result = new StringBuilder();
for (int i = 0; i < b.length; i++)
result.append(Integer.toString((b[i] & 0xff) + 0x100, 16)
.substring(1));
return result.toString();
}

How can I convert a 4-byte array to an integer?

I want to perform a conversion without resorting to some implementation-dependent trick. Any tips?

You need to know the endianness of your bytes.
Assuming (like #WhiteFang34) that bytes is a byte[] of length 4, then...
Big-endian:
int x = java.nio.ByteBuffer.wrap(bytes).getInt();
Little-endian:
int x = java.nio.ByteBuffer.wrap(bytes).order(java.nio.ByteOrder.LITTLE_ENDIAN).getInt();

Assuming bytes is a byte[4] of an integer in big-endian order, typically used in networking:
int value = ((bytes[0] & 0xFF) << 24) | ((bytes[1] & 0xFF) << 16)
| ((bytes[2] & 0xFF) << 8) | (bytes[3] & 0xFF);
The & 0xFF are necessary because byte is signed in Java and you need to retain the signed bit here. You can reverse the process with this:
bytes[0] = (byte) ((value >> 24) & 0xFF);
bytes[1] = (byte) ((value >> 16) & 0xFF);
bytes[2] = (byte) ((value >> 8) & 0xFF);
bytes[3] = (byte) (value & 0xFF);

Not sure if this is correct java syntax, but how about:
int value = 0;
for (i = 0; i <= 3; i++)
value = (value << 8) + (bytes[i] & 0xFF);

You need to specify the byte order of the array, but assuming that the bytes[0] is the most significant byte then:
int res = ((bytes[0] & 0xff) << 24) | ((bytes[1] & 0xff) << 16) |
((bytes[2] & 0xff) << 8) | (bytes[3] & 0xff);
This code is 100% portable, assuming that you use the reverse algorithm to create the byte array in the first place.
Byte order problems arise in languages where you can cast between a native integer type and byte array type ... and then discover that different architectures store the bytes of an integer in different orders.
You can't do that cast in Java. So for Java to Java communication, this should not be an issue.
However, if you are sending or receiving packets to some remote application that is implemented in (say) C or C++, you need to "know" what byte order is being used in the network packets. Some alternative strategies for knowing / figuring this out are:
Everyone uses "network order" (big-endian) for stuff on the wire as per the example code above. Non-java applications on little-endian machines need to flip the bytes.
The sender finds out what order the receiver expects and uses that order when assembling the data.
The receiver figures out what order the sender used (e.g. via a flag in the packet) and decodes accordingly.
The first approach is simplest and most widely used, though it does result in 2 unnecessary endian-ness conversions if both the sender and receiver are little-endian.
See http://en.wikipedia.org/wiki/Endianness

Assuming your byte[] come from somewhere e.g. a stream you can use
DataInputStream dis = ... // can wrap a new ByteArrayInputStream(bytes)
int num = dis.readInt(); // assume big-endian.
or
ByteChannel bc = ... // can be a SocketChannel
ByteBuffer bb = ByteBuffer.allocate(64*1024);
bc.read(bb);
bb.flip();
if (bb.remaining()<4) // not enough data
int num = bb.getInt();
When you send data, you should know if you are sending big-endian or little endian. You have to assume other things such as whether you are sending a 4-byte signed integer. A binary protocol is full of assumptions. (Which makes it more compact and faster, but more brittle than text)
If you don't want to be making as many assumptions, send text.

WE can also use following to make it more dynamic byte array size
BigEndian Format:
public static int pareAsBigEndianByteArray(byte[] bytes) {
int factor = bytes.length - 1;
int result = 0;
for (int i = 0; i < bytes.length; i++) {
if (i == 0) {
result |= bytes[i] << (8 * factor--);
} else {
result |= bytes[i] << (8 * factor--);
}
}
return result;
}
Little Endian Format :
public static int pareAsLittleEndianByteArray(byte[] bytes) {
int result = 0;
for (int i = 0; i < bytes.length; i++) {
if (i == 0) {
result |= bytes[i] << (8 * i);
} else {
result |= bytes[i] << (8 * i);
}
}
return result;
}
This will helps you lot for converting bytes to int values

public static int toInt( byte[] bytes ) {
int result = 0;
for (int i=0; i<3; i++) {
result = ( result << 8 ) - Byte.MIN_VALUE + (int) bytes[i];
}
return result;
}

Correct way to Convert 16bit PCM Wave data to float

I have a wave file in 16bit PCM form. I've got the raw data in a byte[] and a method for extracting samples, and I need them in float format, i.e. a float[] to do a Fourier Transform. Here's my code, does this look right? I'm working on Android so javax.sound.sampled etc. is not available.
private static short getSample(byte[] buffer, int position) {
return (short) (((buffer[position + 1] & 0xff) << 8) | (buffer[position] & 0xff));
}
...
float[] samples = new float[samplesLength];
for (int i = 0;i<input.length/2;i+=2){
samples[i/2] = (float)getSample(input,i) / (float)Short.MAX_VALUE;
}

I had a similar solution, but IMHO a little cleaner. Unfortunately, there's no good library method as far as I'm aware: *This assumes the even bytes are the lower bytes
private static float[] bytesToFloats(byte[] bytes) {
float[] floats = new float[bytes.length / 2];
for(int i=0; i < bytes.length; i+=2) {
floats[i/2] = bytes[i] | (bytes[i+1] << 8);
}
return floats;
}

You may try using the ByteBuffer API.
http://developer.android.com/reference/java/nio/ByteBuffer.html#asFloatBuffer()

As indicated by hertzsprung the answer by jk. only works for unsigned PCM. On Android PCM16 is big-endian signed, so you need to account for the potentially negative value, encoded in two's complement. This means we need to check whether the high byte is greater than 127 and if so subtract 256 from it first before multiplying it by 256.
private static float[] bytesToFloats(byte[] bytes) {
float[] floats = new float[bytes.length / 2];
for(int i=0; i < bytes.length; i+=2) {
floats[i/2] = bytes[i] | (bytes[i+1] < 128 ? (bytes[i+1] << 8) : ((bytes[i+1] - 256) << 8));
}
return floats;
}