We Keep Coding

Java - why does readAllBytes return incorrect byte codes? [duplicate]

This question already has answers here:
Java byte array contains negative numbers
(5 answers)
Closed 6 years ago.
I have the following problem. I'm using Java to create a byte array from a file. So I do the following:
byte[] myByteArray = Files.readAllBytes(filename);
However, for many of the bytes it is returning incorrect/negative values.
For instance, if I test using javascript, to read every byte of a file e.g.
function readbytes(s){
var f = new File(s);
var i,a,c;
var d = [];
if (f.isopen) {
c = f.eof;
for(i=0;i<c ;i++){
a = f.readbytes(1);
d.push(a);
}
f.close();
return d;
} else {
post("could not open file: " + s + "n");
}
}
(readbytes is a function in the program Im using that gives the byte at a specific position).
This returns the correct bytes
So Im wondering, why does java return incorrect codes? Is this something to do with unsigned values?

Java doesn't know unsigned bytes. For instance the unsigned byte 255 would be printed as its signed version -1. In memory however, the actual value would be the same and represented as 255.
If you'd like to convert a byte to its unsigned representation, you may use the bitwise AND operator.
For instance:
bytes[x] & 0xff
Java doesn't know about bytes at runtime either for any operand that may be pushed onto the Java virtual machine's stack. In fact every operation you apply to an integral value results in an integer. That's why ((byte)-1) & 0xff) results in an integer and its value is 255. If you would like to store that value back into a byte, you'd have to cast it to byte again, which of course, is -1.
byte x = -1; // java is friendly enough to insert the implicit cast here
System.out.println(x); // -1
System.out.println(x & 0xff); // 255
byte y = (byte)(x & 0xff); // must add (byte) cast
System.out.println(y); // -1
Also keep in mind that technically the output you see is different but the content is still the same since you can map from Java's signed byte always to its unsigned representation. Ideally, you'd use something like DataInputStream which offers you int readUnsignedByte().

consolidating bytes in java

As part of a program I'm writing I need to consolidate two bytes into a long from an array of bytes.
So assuming this:
byte a[] = new byte[2]
a[0] = 0b1000111
a[1] = 0b1111000
how can I consolidate them such that
long b = 0b10001111111000
EDIT: The program will attempt to consolidate anywhere between 2 bytes and 100, just for reference.

java.nio can do that:
ByteBuffer.wrap(a).getShort()

BigInteger can do this: http://docs.oracle.com/javase/6/docs/api/java/math/BigInteger.html
BigInteger buffer = new BigInteger(yourByteArray);
long result = buffer.longValue();
Note that if your byte array has a value larger than a long (eg, you mentioned that you might go up to 100 bytes), toLong() will only return the value of the lower value bytes. BigInteger, however, will handle any arbitrary number of bytes.

In order to combine those two bytes you will only need 16 bits. A short will suffice:
short result = (short)(((a[0] & 0xff) << 8) | (a[1] & 0xff))
If for some reason, you need the long data type, you need only replace the cast with a long.

Convert char[] to byte[] without losing 'bits'

I'm developing an Android 2.3.3 application with Java.
This app is an iOS code with unsigned data types port to Java.
On iOS it works with UInt16 and UInt8. In one case instead using byte[] I'm using char[].
But know I have to send that char[] as a byte[] using a DatagramPacket.
If one element of char[] is 128, how can I do to insert into byte[] and the receiver gets 128. I don't know what happens if I do this:
char c = (char)128;
byte b = (byte)c;
Which will be b value?
128 = 10000000. b = -1 or b = 127?
How can I convert char[] to byte[] without losing any bits?

In Java char is an unsigned 16-bit quantity. So you can directly convert your uint16 to char without doing anything else.
For unsigned 8-bit quantity you have 2 options:
Use byte. It also holds 8 bits. You don't lose any bits just because it is signed. However, if you do arithmetic with it you need to remember that Java will scale byte up automatically to an int and sign-extend it. To prevent this just always mask it like this:
byte b;
int foo = 5 * (b & 0xFF);
Use char. It is unsigned and can hold 16 bits so the 8 bits will fit in there quite nicely. To put a byte into a char just do this:
byte b;
char c = (char)(b & 0xFF); // Mask off the low-order 8 bits
To put a char into a byte just do:
char c;
byte b = (byte)c; // Truncates to 8 bits
Be aware that byte in Java is signed, so that whenever you do arithmetic with it you need to mask the low-order 8 bits only (to prevent sign-extension). Like this:
byte b;
int foo = (b & 0xFF);
You can do all the normal bitwise operations you want with a byte without having to mask:
byte b;
if (b & 0x80) ... // Test a flag
b |= 0x40; // Set a flag
b ^= 0x20; // Flip a flag from 0 to 1 or 1 to 0
b ^= ~0x10; // Clear a flag
byte x = b << 3; // Shift left 3 bits and assign
byte x = b >>> 4; // Shift right 4 bits and assign (WITHOUT sign extension)

I think you need to rethink your approach so you don't end up needing to convert char[] to byte[].
If your data really is characters, then you want to look at various serialization techniques, such as using new String(char[]) to create a string and then using getBytes(Charset) to get the bytes as encoded by a given Charset (because, of course, the same characters result in different bytes when encoded in ASCII or UTF-8 or UTF-16, etc.).
But from your question, it sounds like you're not really using characters, you're just using char as a 16-bit type. If so, doing the conversion isn't difficult, something along these lines:
byte[] toBytes(char[] chars) {
byte[] bytes = new byte[chars.length * 2];
int ci, bi;
char ch;
bi = 0;
for (ci = 0; ci < chars.length; ++ci) {
ch = chars[ci];
bytes[bi++] = (byte)((ch & 0xFF00) >> 8);
bytes[bi++] = (byte)(ch & 0x00FF);
}
return bytes;
}
Reverse the masks if you want the result to be small-endian instead.
But again, I would look at your overall approach and try to avoid this.

Parse a DatagramPacket after converting it to a byte array in Java

I am trying to parse a DatagramPacket that I will receive at a socket. I know the format of the packet I will receive, which is a DHCPREQUEST packet, but I don't think that really matters. For simplicity's sake, let's just consider the first six fields:
First field is the "opcode", which is 1 byte.
Second field is the "hardware type" which is 1 byte.
Third, "hardware address length", 1 byte.
Fourth, "hops", 1 byte.
Fifth, "transaction identifier xid", 4 bytes.
Sixth, "seconds", 2 bytes.
After I receive the packet, my approach is to convert it to a byte array.
DatagramPacket request = new DatagramPacket(new byte[1024], 1024);
socket.receive(request);
byte[] buf = request.getData();
At this point, the packet is stored in the byte array buf as a series of bytes. Since I know what the structure of this byte sequence is, how can I parse it? The one-byte fields are simple enough, but how about the multiple-bit fields? For example, how can I extract bytes 4 to 7, and store them in a variable named xid?
I could manually put each byte into an array:
byte[] xid = new byte[4];
xid[0] = buf[4];
xid[1] = buf[5];
xid[2] = buf[6];
xid[3] = buf[7];
But that's just tedious, and impractical for fields that are hundreds of bytes in length. The String class can parse substrings given an offset and length; is there a similar method for byte arrays in Java?
Or am I somehow making things difficult for myself?

Wrap the byte array in a ByteArrayOutputStream; wrap a DataInputStream around that; then use the methods of DataInputStream.

The cleanest way to do something like this is probably to use the utility method Arrays.copyOfRange.

What you do is write yourself some helper methods to extract 2 byte, 4 byte, etc values from the packet, reading the bytes and assembling them into Java short, int or whatever values.
For example
public short getShort(byte[] buffer, int offset) {
return (short) ((buffer[offset] << 8) | buffer[offset + 1]);
}
Then you use these helper methods as often as you need to. (If you want to be fancy, you could have the methods update an attribute that holds the current position, so that you don't have to pass an offset argument.)
Alternatively, if you were not worried by the overheads, you could wrap the byte array in ByteArrayInputStream and a DataInputStream, and use the latter's API to read bytes, shorts, ints, and so on. IIRC, DataInputStream assumes that numbers are represented in the stream in "network byte order" ... which is almost certainly what the DHCP spec mandates.

I'm a bit late to this, but there's a ByteBuffer class:
ByteBuffer b = ByteBuffer.wrap(request.getData());
byte opcode = b.get();
byte hwtype = b.get();
byte hw_addr_len = b.get();
byte hops = b.get();
int xid = b.getInt();
short seconds = b.getShort();
Or, if you only need a single field:
ByteBuffer b = ByteBuffer.wrap(request.getData());
int xid = b.getInt(4);

Java - converting int to byte array without considering sign

To convert an int into a byte array, I'm using the following code:
int a = 128;
byte[] b = convertIntValueToByteArray(a);
private static byte[] convertIntValueToByteArray(int intValue){
BigInteger bigInteger = BigInteger.valueOf(intValue);
byte[] origByteArray = bigInteger.toByteArray();
byte[] noSignByteArray = new byte[bigInteger.bitLength()/8];
if(bigInteger.bitLength()%8!=0){
noSignByteArray = origByteArray;
}else{
System.arraycopy(origByteArray,1,noSignByteArray,0,noSignByteArray.length);
}
return noSignByteArray;
}
There are two things which I'm attempting to do.
1)I need to know the number of bytes (rounded up to the closes byte) of the original integer. However, I don't need the additional bit that is added for the sign bit when I call the toByteArray() method. This is the reason why I have the helper method. So in this example, if I don't have the helper method, when I convert 128 to a byte array I get the length to be 2 octets because of the sign bit but I'm only expecting it to be one octet.
2)I need the positive representation of the number. In this example, if I attempt to print the first element in array b, I get -128. However, the numbers I will be using will be positive numbers only so what I actually want is 128. I'm limited to using a byte array. Is there a way to accomplish this?
Updated Post
Thank you for the responses. I haven't found the exact answer I was looking for so I'll attempt to give more details. Ultimately, I want to write values of different types over a data output stream. In this post, I'd like to clarify what happens when ints are written to a data output stream. I've come across two scenarios.
1)
DataOutputStream os = new DataOutputStream(this.socket.getOutputStream());
byte[] b = BigInteger.valueOf(128).toByteArray();
os.write(b);
2)
DataOutputStream os = new DataOutputStream(this.socket.getOutputStream());
os.write(128);
In the first scenario, when the bytes are read from a data input stream, it seems that the first element in the byte array is a 0 to represent the msb and the second element in the array contains the number -128. However, since the msb is 0 we would be able to determine that it is intended to be a positive number. In the second scenario, there is no msb and the only element present in the byte array read from the input stream is -128. I was expecting the write() method of the data output stream to convert the int into the byte array in the same manner as the toByteArray() method does on a BigInteger object. However, this doesn't seem to be the case as the msb is not present. So my question is, how in the second scenario are we supposed to know that 128 is supposed to be a positive number and not a negative one if there is no msb.

As you probably already know
In an octet, the pattern 10000000 can be interpreted as either 128 or -128, depending on the, um, outside interpretation
Java's byte type interprets octects as values in -128...127 only.
If you are building an application in which the entire world consists of nonnegative integers only, then you could simply do all of your work under the assumption that the byte value -128 will mean 128 and -127 will mean 129 and ... and -1 will mean 255. This is certainly doable but it takes work.
Dealing with the notion of an "unsigned byte" like this is normally done by expanding the byte into a short or int with the higher order bits all set to zero and then performing arithmetic or displaying your values. You will need to decide whether such an approach is more to your liking than just representing 128 as two octets in your array.

I think the following code might be sufficient.
In java int is a twos-complements binary number:
-1 = 111...111
ones complement = 000...000; + 1 =
1 = 000...001
So that about the sign bit I do not understand. Be it, that you could do Math.abs(n).
A byte ranges from -128 to 127, but the interpretation is a matter of masking, as below.
public static void main(String[] args) {
int n = 128;
byte[] bytes = intToFlexBytes(n);
for (byte b: bytes)
System.out.println("byte " + (((int)b) & 0xFF));
}
public static byte[] intToFlexBytes(int n) {
// Convert int to byte[4], via a ByteBuffer:
byte[] bytes = new byte[4];
ByteBuffer bb = ByteBuffer.allocateDirect(4);
bb.asIntBuffer().put(n);
bb.position(0);
bb.get(bytes);
// Leading bytes with 0:
int i = 0;
while (i < 4 && bytes[i] == 0)
++i;
// Shorten bytes array if needed:
if (i != 0) {
byte[] shortenedBytes = new byte[4 - i];
for (int j = i; j < 4; ++j) {
shortenedBytes[j - i] = bytes[j]; // System.arrayCopy not needed.
}
bytes = shortenedBytes;
}
return bytes;
}

To answer your first question—how many bytes are required to represent a nonnegative integer using an unsigned representation—consider the following functions I wrote in Common Lisp.
(defconstant +bits-per-byte+ 8)
(defun bit-length (n)
(check-type n (integer 0) "a nonnegative integer")
(if (zerop n)
1
(1+ (floor (log n 2)))))
(defun bytes-for-bits (n)
(check-type n (integer 1) "a positive integer")
(values (ceiling n +bits-per-byte+)))
These highlight the mathematical underpinnings of the problem: namely, the logarithm tells you how many powers of two (as provided by bits) it takes to dominate a given nonnegative integer, adjusted to be a step function with floor, and the number of bytes it takes to hold that number of bits again as a step function, this time adjusted with ceiling.
Note that the number zero is intolerable as input to a logarithm function, so we avoid it explicitly. You may observe that the bit-length function could also be written with a slight transformation of the core expression:
(defun bit-length-alt (n)
(check-type n (integer 0) "a nonnegative integer")
(values (ceiling (log (1+ n) 2))))
Unfortunately, as the logarithm of one is always zero, regardless of the base, this version says that the integer zero can be represented by zero bits, which isn't the answer we want.
For your second goal, you can use the functions I've defined above to allocate the required number of bytes, and incrementally set the bits you need, ignoring sign. It's hard to tell if you're having trouble getting the proper bits set in the byte vector, or whether your problem is in interpreting the bits in way that avoids treating the high bit as a sign bit (that is, two's complement representation). Please elaborate what kind of push you need to get you moving again.