We Keep Coding

Variable length-encoding of int to 2 bytes

I'm implementing variable lenght encoding and reading wikipedia about it. Here is what I found:
0x00000080 0x81 0x00
It mean 0x80 int is encoded as 0x81 0x00 2 bytes. That what I cannot understand. Okay, following the algorithm listed there we have.
Binary 0x80: 00000000 00000000 00000000 10000000
We move the sign bit to the next octet so we have and set to 1 (indicating that we have more octets):
00000000 00000000 00000001 10000000 which is not equals to 0x81 0x00. I tried to write a program for that:
byte[] ba = new byte[]{(byte) 0x81, (byte) 0x00};
int first = (ba[0] & 0xFF) & 0x7F;
int second = ((ba[1] & 0xFF) & 0x7F) << 7;
int result = first | second;
System.out.println(result); //prints 1, not 0x80
ideone
What did I miss?

Let's review the algorithm from the Wikipedia page:
Take the binary representation of the integer
Split it into groups of 7 bits, the group with the highest value will have less
Take these seven bits as a byte, setting the MSB (most significant bit) to 1 for all but the last; leave it 0 for the last one
We can implement the algorithm like this:
public static byte[] variableLengthInteger(int input) {
// first find out how many bytes we need to represent the integer
int numBytes = ((32 - Integer.numberOfLeadingZeros(input)) + 6) / 7;
// if the integer is 0, we still need 1 byte
numBytes = numBytes > 0 ? numBytes : 1;
byte[] output = new byte[numBytes];
// for each byte of output ...
for(int i = 0; i < numBytes; i++) {
// ... take the least significant 7 bits of input and set the MSB to 1 ...
output[i] = (byte) ((input & 0b1111111) | 0b10000000);
// ... shift the input right by 7 places, discarding the 7 bits we just used
input >>= 7;
}
// finally reset the MSB on the last byte
output[0] &= 0b01111111;
return output;
}
You can see it working for the examples from the Wikipedia page here, you can also plug in your own values and try it online.

Another Variable length encoding of integers exists and are widely used. For example ASN.1 from 1984 does define "length" field as:
The encoding of length can take two forms: short or long. The short
form is a single byte, between 0 and 127.
The long form is at least two bytes long, and has bit 8 of the first
byte set to 1. Bits 7-1 of the first byte indicate how many more bytes
are in the length field itself. Then the remaining bytes specify the
length itself, as a multi-byte integer.
This encoding is used for example in DLMS COSEM protocol or https certificates. For simple code, you can have a look at ASN.1 java library.

binary value comes wrong after & with 0x000000FF [duplicate]

I'm reading a file into a byte array in chunks and sending it over the network via a POST request to a webserver. It's not anything complicated, I've done it before using this exact same code. This time, I noticed that my images are looking really odd when they get to the server, so I decided to look at the byte array being sent and the one being received just to make sure it was the same. It's not. On the java sending side the byte array contains negative numbers. On the C# receiving side, there are no negative numbers.
The first 15 bytes on the receiving side (C#)
137
80
78
71
13
10
26
10
0
0
0
13
73
72
68
Those same bytes but on the sending side (java)
-119
80
78
71
13
10
26
10
0
0
0
13
73
72
68
All of the non-negative numbers are the same, and the -119 isn't the only negative number, they are all over. I did notice that -119 and 137 are 256 apart and wondered if that has something to do with it.
The code I'm using to read the image (java)
public static byte[] readPart(String fileName, long offset, int length) throws FileNotFoundException, Exception
{
byte[] data = new byte[length];
File file = new File(fileName);
InputStream is = new FileInputStream(file);
is.skip(offset);
is.read(data,0,data.length);
is.close();
return data;
}
The code I'm using to write the data (c#)
private void writeFile(string fileName, Stream contents)
{
using (FileStream fs = new FileStream(fileName, FileMode.Append, FileAccess.Write, FileShare.ReadWrite))
{
int bufferLen = 65000;
byte[] buffer = new byte[bufferLen];
int count = 0;
while ((count = contents.Read(buffer, 0, bufferLen)) > 0)
{
fs.Write(buffer, 0, count);
}
fs.Close();
}
contents.Close();
}
I don't know if that is something that always happens and I just never noticed it before or if it is something that decided to go horribly wrong. What I do know is that this code worked before for something very similar and that it's not working now.
If anyone has any suggestions or an explanation I would really appreciate it.
EDIT:
The reason my images were looking odd is how I was calling the readPart method.
byte[] data = FileUtilities.readPart(fileName,counter,maxFileSize);//counter is the current chunk number
How I should have been calling it
byte[] data = FileUtilities.readPart(fileName,counter*maxFileSize,maxFileSize);//the current chunk * cuhnksize for the offset...
Thanks everyone, I'm significantly less confused now :)

In Java, byte is a signed value (using two's complement to encode negative values), so what you see it correct if unexpected by most people.
To convert a byte to an unsigned int value, use b & 0xff

Java doesn't have unsigned bytes; all bytes are treated as signed. That's all.
All that really matters is how you think of the bytes, since you rarely ever actually need to do comparisons on bytes. The only significant difference is that they print out as signed, as you've discovered.
If you like, you can use e.g. Guava's UnsignedBytes utilities to view Java bytes as unsigned, but there's really not much practical difference.

As a further explanation, assume you have 137 as an unsigned byte. That is represented as:
1000 1001
This binary value, when expressed as a signed two's complement number, turns out to be -119. (-128 + 9)
Any unsigned byte values over 128 will be affected by the difference since the left-most bit is used in this way by the two's complement scheme.

Maybe it has something to do with the fact that Java's byte is signed (range -128 to 127) while C#'s is unsigned (0 to 255) :). The information is the same in binary, it's just interpreted differently.

The range of byte is from -128 to 127, so if you try to assign a byte 128, it will cycle around and the result will be -128.
System.out.println("Max val = " + Byte.MAX_VALUE); //prints: Max val = 127
System.out.println("Min val = " + Byte.MIN_VALUE); //prints: Min val = -128
System.out.println("(byte)137 = " + (byte)137); //prints: (byte)137 = -119
System.out.println("(byte)128 = " + (byte)128); //prints: (byte)128 = -128
System.out.println("(byte)-129 = " + (byte)-129); //prints: (byte)-129 = 127

Byte arrays in java

Hey I need to store the following hex value in a byte array - 0xCAFEBABEDEADBEEF.
So I tried to store it like so.
byte[] v0 = {11001010,11111110,10111010,10111110,11011110,10101101,10111110,11101111};
where 11001010 is CA in binary, 11111110 is FE in binary etc.
But I get an error saying 11001010 is an int, so I presume this is because bytes are signed bytes in java, and we can only have values between +127 and -128.
So is there way I can do this in java(maybe using unsigned bytes...if they exist!?)
Thanks guys.

Put 0b in front of the number. You may also have to cast to byte:
byte[] v0 = {(byte)0b11001010,(byte)0b11111110,...};
The 0b prefix means it is a binary number.
If you want it to be easier to read, you can use 0x for hexadecimal:
byte[] v0 = {(byte)0xCA,(byte)0xFE,(byte)0xBA,(byte)0xBE,...};
Here's a way to do it (binary form) if you're using a Java version less than 7:
byte[] v0 = {Byte.parseByte("11001010", 2),...);

The literal 11001010 represents a decimal of int type and of value 11,001,010 - that is 11 milions and something.
If you're using Java 7, you can define binary literals using the 0b prefix, such as 0b11001010. To improve readability, you can put underscores in the values, such as 0b_1100_1010.
However, note that the type of even such binary (or hexadecimal) literal is still int. This, together with the fact that bytes are (unfortunately) signed in Java (thus their value is in range -128 to 127) results in the problem that a literal with a value larger than 127 has to be manually cast to byte:
// 0b_1001_0001 or 0x91 is 145 in decimal
byte b1 = (byte) 0b_1001_0001;
byte b2 = (byte) 0x91;
However, the value of such byte will be -111 (145 - 256). To get back the unsigned value, you need to manually add the module (256) to the value:
int i1 = b1 + 256;
int i2 = b1 & 0xff;
// both i1 and i2 are 145
For more information see this question

If write byte to byte, you can use:
byte[] v0 = {0b11001010, 0b11111110, 0b10111010, ... }
or
byte[] v0 = {0xCA, 0xFE, ... }

Sending 0xFF and Calculating CRC with signed bytes - WriteSingleCoil & ModBUS & Java & Android -

EDITED & SOVLED (below)
I'm using Java for Android trying to send the byte 255 (0xFF in WriteSingleCoil function) to a ModBUS server device.
Device is not runnig, I don't know if because of not able to interpretate the signed byte -1 or because of I'm wrong calculating the CRC.
I don't know how to calculate CRC for negative bytes.
Summarizing: I don't know how to send function 05 Write Single Coil with 0xFF value for switch on the coil for from Java to ModBUS server.
Can anyone help me please?
SOLUTION:
" iIndex = ucCRCLo ^ b: operations like this must be written as iIndex
= (ucCRCLo ^ b)&0xff because the & will cast ucCRCLo, b and the result to int, which is 32 bits while short is 16 so you will have a lot of
extra bits set to 1 "
This answer helped me. Thanks a lot to TheDayOfcondor
But also my huge problem was the usual problem in Java with signed bytes. My CRC calculating function is doing it right for unsigned bytes, but it give errors if I pass inside signed bytes.
The trick for work with bytes for ModBUS comunication is work in the whole App with shorts as bytes, for have the range 0-255, even calculating trames and CRC. And only in the last step, when sending trame to ModBUS sever, cast them to bytes again. This is running.
Hope it will helps to someone in future.
EXPLAINING PROBLEM:
I'm trying to set ON a coil to ModBUS with function 05, this is explaining of function:
Request
I'm tryiing to set ON the coil on address 1:
This hex: 0A 05 00 01 ff 00 DC 81
This byte array: 10 5 0 1 255 0 220 129
10: The Slave Address (10 = 0A hex)
05: The Function Code (Force Single Coil)
0001: The Data Address of the coil. (coil# 1 = 01 hex)
FF00: The status to write ( FF00 = ON, 0000 = OFF )
DC81: The CRC (cyclic redundancy check) for error checking.
The thing is that Java is using signed bytes, so I can't put 255 on my byte array.
I understand I should put -1, but then I can't calculate CRC correctly, because of I have a couple of precalculated array of bytes for get the CRC but the function send a negative index.
So: I don't know if I'm doing right trying to send -1, if I have an alternative for sending 255, neither how to calculate CRC for -1.
This is function for calculate CRC:
public short[] GenerateCRC (byte[] pMsg) {
short ucCRCHi = 0xFF;
short ucCRCLo = 0xFF;
int iIndex;
for (byte b : pMsg)
{
iIndex = ucCRCLo ^ b;
try {
ucCRCLo = (short)(ucCRCHi ^ aucCRCHi[ ( iIndex ) ]);
ucCRCHi = aucCRCLo[ ( iIndex ) ];
} catch (Exception e) {
Log.e(LOGTAG, "GenerateCRC: " + e.toString(), e);
e.printStackTrace();
}
}
short[]result= new short[2];
result0]= ucCRCHi;
result1]= ucCRCLo;
return result;
}

The question is not very clear - however, the most common problem dealing with bytes is the fact Java does not have unsigned bytes, and boolean operation are always between int
The best way to deal with bytes is to use integers, and-ing every operation with 0xff. Also use >>> for the shift right (it is the unsigned version)
Example:
byte b= (byte)(255 & 0xff) // gives you the "unsigned byte"
byte b= (byte) ((b<<2)0xff ) // shift left must be truncated
If you post your code to calculate the CRC I can have a look into it
The best way to define a byte array without using negative numbers is like this:
byte[]={ (byte)0xff, (byte)0xff, (byte)0xff };

Bit manipulation of chars in Java or C?

I am a student trying to implement the DES algorithm.
I have a choice of 2 languages: C & Java.
I did understand the algorithm, but am stuck at the very beginning as to manipulation of the key.
Here's the problem.
In DES, we have a 64-bit key (8 chars in C and 4 in Java, although I can cast the char to byte to get only the ASCII part), of which every 8th bit is a parity bit and needs to be stripped to make it a 56-bit key and do further processing. I have thought about this for long, but cannot find a way to strip every 8th bit and store the result in another char array (in Java as well as C).
I tried using the java.util.BitSet class, but got confused.
Any suggestions as to how can I remove every 8th bit and concat adjacent bytes(Java) or chars(C) to get the 56 bit key?
I am aware of the bit operations and shifting, but for the specific example:
Suppose I have an 16 bit key: 1100 1001 1101 1000.
I need to remove the 8th and 16th bit, making the key: 1100 100 1101 100.
If I declare 2 bytes, how do I truncate the 8th bit and append the 9th bit to it, making the first byte: 1100 1001
So, what I need help with is how do I, replace 8th bit with 9th bit, replace 16th bit with 17th bit and so on to derive a 56-bit key from 64-bit key?
If someone can explain it to me, I might probably be able to implement it regardless of language.

Be careful of 16-bit chars in Java. Many methods only convert the lower 8 bits. Read the documentation carefully. It is more usual to treat a cryptographic key as a a byte[] in Java due to the stronger typing than in C.
As to the parity bits, check through the DES algorithm carefully and see where they are used. That should give you a hint as to what you need to do with them.

In C, you can manipulate bits with the bitwise operators, such as & and |, as well as the bitshift operators << and >>.
For instance, to turn off the high bit of a given byte, you can do this.
char c = 0xBF; // initial value is bit pattern 10111111
c &= 0x7F; // perform AND against the bit pattern 01111111
// final value is bit pattern 00111111 (0x3F)
Does that make sense?
Obviously, you need to be able to convert from a bit pattern to hex, but that's not too hard.
You can use similar masking to extract the bits you want, and put them in an output buffer.
Update:
You have 64 bits (8 bytes) of input, and want 56 bits (7 bytes) of output.
Let's represent your input as the following, where each letter represents a single bit
The 'x' bits are the ones you want to throw away.
xAAAAAAA xBBBBBBB xCCCCCCC xDDDDDDD xEEEEEEE xFFFFFFF xGGGGGGG xHHHHHHH
So you want your final answer to be:
AAAAAAAB BBBBBBCC CCCCCDDD DDDDEEEE EEEFFFFF FFGGGGGG GHHHHHHH
So in C, we might have code like this:
unsigned char data[8] = {/* put data here */};
// chop off the top bit of the first byte
data[0] <<= 1;
// the bottom bit of data[0] needs to come from the top data bit of data[1]
data[0] |= (data[1] >> 6) & 0x01;
// use similar transformations to fill in data[1], data[2], ... data[6]
// At the end, data[7] will be useless
Of course this is not optimized at all, but hopefully you get the idea.

I can briefly tell about a way....i will explain further if required...
Right shift all the 8 chars by 1 ie c1 = c1>>1 etc
Multiplyc1 with the total number of bytes (ie56) ie c1 * 0x0000000000 (not sure how many zeros)
Then, add 0x0000+ to next chars ie c2 = c2 + 0x0000; c3 = c3 + 0x000000 so on (Keep adding 2 0s for proceeding chars)
Now, start adding c1 + c2 + c3.......
The idea here is that, first fill in the number with zeros and start adding teh other chars so that they keep lying properly
00 00 00 00 00 00 00 00 00
34 00 00 00 00 00 00 00 00 (c1 = c1>>1) . Here c1=0x34, c2=0x67
00 67 00 00 00 00 00 00 00 (c2 = c2>>1)
so on...............
Add teh above;
I hope this will help.

#jwd, #jscode
Thanks a lot for your help.
To jwd: I got the idea from your code. Seemed pretty simple logic after I read it.. :-) Wonder why I didnt think of that.
Well, I did polish your idea a little bit and it works fine now in Java.
If anyone has any suggestions, please let me know.
THANKS..
P.S.: The testing part is very primitive. I print the bit values. I did it manually for a couple of examples and used the same as input and it works fine.
=============================================
public static void main(String[] args) {
BufferedReader br = new BufferedReader(new InputStreamReader(System.in));
System.out.println("Enter an 8 char key: ");
String input;
try {
// get key, key.length()>=8 chars
input = br.readLine();
if (input.length() < 8) {
System.out.println("Key < 8B. Exiting. . .");
System.exit(1);
}
// java char has 16 bits instead of 8 bits as in C,
// so convert it to 8 bit char by getting lower order byte &
// discarding higher order byte
char[] inputKey = input.toCharArray();
byte[] key64 = new byte[8];
byte[] key56 = new byte[7];
// consider only first 8 chars even if input > 8
for (int counter = 0; counter < 8; counter++)
key64[counter] = (byte) inputKey[counter];
System.out.print("\n$$ " + new String(key64) + " $$\n");
// converting 64bit key to 56 bit key
for (int counter = 0; counter < KEY_LENGTH - 1; counter++) {
key64[counter] = (byte) (key64[counter] >>> 1);
key64[counter] = (byte) (key64[counter] << 1);
}
for (int counter = 0; counter < KEY_LENGTH - 1; counter++) {
key56[counter] = (byte) (key64[counter] << counter);
key56[counter] = (byte) (key56[counter] | (key64[counter + 1] >>> (KEY_LENGTH - 1 - counter)));
}
/*Conversion from 64 to 56 bit testing code
System.out.println(new String(key56));
System.out.println();
for (int counter1 = 0; counter1 < 7; counter1++) {
for (int counter2 = 7; counter2 >= 0; counter2--) {
System.out.println(key56[counter1] & (1 << counter2));
}
System.out.println();
}*/
} catch (IOException e) {
e.printStackTrace();
}
}