We Keep Coding

Java 11 Compact Strings magic behind char[] to byte[]

I been reading about encoding Unicode Java 9 compact Strings in the last two days i am getting quite well. But there is something that i dont understand.
About byte data type
1). Is a 8-bit storage ranges from -128 to 127
Questions
1). Why Java didn't implement it like char unsigned 16 bits? i mean it would be in a range of 0.256 because from 0 to 127 only can i hold a Ascii value but what would happen if i set the value 200 a extended ascii would overflow to -56.
2). Does the negative value mean something i mean i have try a simple example using Java 11
final char value = (char)200;//in byte would overflow
final String stringValue = new String(new char[]{value});
System.out.println(stringValue);//THE SAME VALUE OF JAVA 8
I have checked the String.value variable and i see a byte array of
System.out.println(value[0]);//-56
The same questions like before arise does the -56 mean something i mean the (negative value) in other languages this overflow is detected to return to the value 200? How can Java know that -56 value is the same as 200 in char.
I have try hardest examples like codepoint 128048 and i see in String.value variable a array of bytes like this.
0 = 61
1 = -40
2 = 48
3 = -36
I know this codepoint takes 4 bytes but i get it how is transformed char[] to byte[] but i dont know how String handle this byte[] data.
Sorry if this question is simple and sorry any typing english is not my natural language thanks a lot.

Why Java didn't implement it like char unsigned 16 bits? i mean it would be in a range of 0.256 because from 0 to 127 only can i hold a Ascii value but what would happen if i set the value 200 a extended ascii would overflow to -56.
Java’s primitive data types were settled with Java 1.0 a quarter century ago. The compact strings were introduced in Java 9, less than two years ago. This new feature, which is merely an implementation detail, did not justify fundamental changes at Java’s type system.
Besides that, you are looking at one interpretation of the data stored in a byte. For the sake of representing iso-latin-1 units, it is entirely irrelevant whether interpreting the same data as Java’s built-in signed byte would result in a positive or negative number.
Likewise Java’s I/O API allows reading a file into a byte[] array and write byte[] arrays back to files and these two operations are already sufficient to copy a file losslessly, regardless of its file format which would be relevant when interpreting its content.
So the following works since Java 1.1:
byte[] bytes = "È".getBytes("iso-8859-1");
System.out.println(bytes[0]);
System.out.println(bytes[0] & 0xff);
-56
200
The two numbers, -56 and 200 are just different interpretations of the bit pattern 11001000 whereas the iso-latin-1 interpretation of a byte containing the bit pattern 11001000 is the character È.
A char value is also just an interpretation of a two byte quantity, i.e. as UTF-16 code unit. Likewise, a char[] array is a sequence of bytes in the computer’s memory with a standard interpretation.
We can also interpret other byte sequences this way.
StringBuilder sb = new StringBuilder().appendCodePoint(128048);
byte[] array = new byte[4];
StandardCharsets.UTF_16LE.newEncoder()
.encode(CharBuffer.wrap(sb), ByteBuffer.wrap(array), true);
System.out.println(Arrays.toString(array));
will print the value you’ve seen, [61, -40, 48, -36].
The advantage of using a byte[] array inside the String class is, that now, the interpretation can be chosen, to use iso-latin-1 when all characters are representable with this encoding or utf-16 otherwise.
The possible numeric interpretations are irrelevant to the string. However, when you ask “How can Java know that -56 value is the same as 200”, you should ask yourself, how does it know that the bit pattern 11001000 of a byte is -56 in the first place?
System.out.println(value[0]);
bears an actually expensive operation, compared to ordinary computer arithmetic, the conversion of a byte (or an int) to a String. This conversion operation is often overlooked as it has been defined as the default way of printing a byte, but is not more natural than a conversion to a String interpreting the value as an unsigned quantity. For further reading, I recommend Two's complement.

This is because not all bytes in a string are interpreted the same. This depends to the string's character encoding.
Example:
if a string is an UTF-8 string, its characters will be 8-bits in size.
in an UTF-16 string, its characters will be 16-bits in size.
etc...
This means, if the string is to be represented as UTF-8, the characters will be made by reading 1 byte at a time; if 16-bits, the characters will made by reading 2 bytes at a time.
Look at this code: a single byte array data is transformed to string using UTF-8 and UTF-16.
byte[] data = new byte[] {97, 98, 99, 100};
System.out.println(new String(data, StandardCharsets.UTF_8));
System.out.println(new String(data, StandardCharsets.UTF_16));
The output of this code is:
abcd // 4 bytes = 4 chars, 1 byte per char
慢捤 // 4 bytes = 2 chars, 2 byte per char
Going back to the question, what motivated the developers to do so is to reduce memory footprint on strings. Not all strings uses all the 16-bits a char offers.
EDIT: Code here

Converting String to UTF-8 byte array returns a negative value in Java

Let's say I have a byte array and I try to encode it to UTF_8 using the following
String tekst = new String(result2, StandardCharsets.UTF_8);
System.out.println(tekst);
//where result2 is the byte array
Then, I get the bytes using getBytes() with values from 0 to 128
byte[] orig = tekst.getBytes();
And then, I wish to do a frequency count of my byte[] orig using the ff:
int frequencies = new int[256];
for (byte b: orig){
frequencies[b]++;
}
Everything goes well till I encounter an error which states
java.lang.ArrayIndexOutOfBoundsException: -61
Does that mean that my byte still contains negative values despite converting it to UTF-8? Is there something wrong that I'm doing? Can someone please give me clarity on this cause I'm still a beginner on the subject. Thank you.

Answering the specific question
Does that mean that my byte still contains negative values despite converting it to UTF-8?
Yes, absolutely. That's because byte is signed in Java. A byte value of -61 would be 195 as an unsigned value. You should expect to get bytes which aren't in the range 0-127 when you encode any non-ASCII text with UTF-8.
The fix is easy: just clamp the range to 0-255 with a bit mask:
frequencies[b & 0xff]++;
Addressing what you're attempting to do
This line:
String tekst = new String(result2, StandardCharsets.UTF_8);
... is only appropriate if result2 is genuinely UTF-8-encoded text. It's not appropriate if result2 is some arbitrary binary data such as an image, compressed data, or even text encoded in some other encoding.
If you want to preserve arbitrary binary data as a string, you should use something like Base64 or hex. Basically, you need to determine whether your data is inherently textual (in which case, you should use strings for as much of the time as possible, and use an appropriate Charset to convert to binary where necessary) or inherently binary (in which case you should use bytes for as much of the time as possible, and use base64 or hex to convert to text where necessary).
This line:
byte[] orig = tekst.getBytes();
... is almost always a bad idea. It uses the platform-default encoding to convert a string to bytes. If you really, really want to use the platform-default encoding, I would make that explicit:
byte[] orig = tekst.getBytes(Charset.defaultCharset());
... but this is an extremely unusual requirement these days. It's almost always better to stick to UTF-8 everywhere.

DataOutputStream: purpose of the "encoded string too long" restriction

There is a strange restriction in java.io.DataOutputStream.writeUTF(String str) method, which limits the size of an UTF-8 encoded string to 65535 bytes:
if (utflen > 65535)
throw new UTFDataFormatException(
"encoded string too long: " + utflen + " bytes");
It is strange, because:
there is no any information about this restriction in JavaDoc of this method
this restriction can be easily solved by copying and modifying an internal static int writeUTF(String str, DataOutput out) method of this class
there is no such restriction in the opposite method java.io.DataInputStream.readUTF().
According to the said above I can not understand the purpose of a such restriction in the writeUTF method. What have I missed or misunderstood?

The Javadoc of DataOutputStream.writeUTF states:
First, two bytes are written to the output stream as if by the
writeShort method giving the number of bytes to follow. This value
is the number of bytes actually written out, not the length of the
string.
Two bytes means 16 bits: in 16 bits the maximum integer one can encode is 2^16 == 65535.
DataInputStream.readUTF has the exact same restriction, because it first reads the number of UTF-8 bytes to consume, in the form of a 2-byte integer, which again can only have a maximum value of 65535.
writeUTF first writes two bytes with the length, which has the same result as calling writeShort with the length and then writing the UTF-encoded bytes. writeUTF doesn't actually call writeShort - it builds up a single byte[] with both the 2-byte length and the UTF bytes. But that is why the Javadoc says "as if by the writeShort method" rather than just "by the writeShort method".

Java- gzip member trailer

This is part of a larger assignment that I've mostly got done except for this one part, which is a bit embarrassing because it sounds really simply on paper.
So basically, I've got a large amount of compressed data. I've been keeping track of the length using a CRC32
CRC32 checksum = new CRC32();
...
//read input into buffer
checksum.update(buff, 0, bytesRead);
So it updates everytime more info is read in. I've also kept track of the uncompress length using
uncompressedLength += manage.read(buff);
So it is an int value that has the number of bytes of the original file. This is a little Endian machine.
From what I can tell, what I need is four byte CRC, which I used
public byte[] longToBytes(long x) {
ByteBuffer buffer = ByteBuffer.allocate(8);
buffer.putLong(x);
return buffer.array();
}
byte[] c = longToBytes(checksum.getValue());
BUT this is 8 bytes. CRC32.getValue returns a long. Can I convert it to an int in this case without losing information I need?
And then the ISIZE is supposed to be...the four byte compressed length modulo 2^32. I've got the variable uncompresedLength which is an int. I think I just have to convert it to bytes and that's all?
I've been hexdumping the result from gzip and the result from my program and my header and data are right, I'm just missing my trailer.
As for why I'm doing this manually, it's because of an assignment. Trust me, I'd love to just use GZIPOoutputStream if I could.

CRC32 has 32 bits... the class returns long because of the super interface.
uncompressed length should be long, since nowadays files larger than 2G isn't uncommon.
so in both cases, you need to convert the lowest 32 bits of a long to 4 bytes.
static byte[] lower4bytes(long v)
{
return new byte[] {
(byte)(v ),
(byte)(v>> 8),
(byte)(v>>16),
(byte)(v>>24)
};
}

To write an integer in little-endian form, simply write the low byte of the integer (i.e. modulo 256 or anded with 0xff), then shift it down eight bits or divide by 256, then write the resulting low byte, and repeat that two more times. You'll write four bytes. Since you only write four, you will automatically be writing the length modulo 232.

How to read file created by C++ program in java?

I have one file created by c++ program which is in encrypted format. I want to read it in my java program. In case of decryption of file contents, decryption algorithm is performing operations on byte[which is unsigned char-BYTE in c/c++]. I used same decryption algorithm which I have used in my c/c++ program. This algorithm contains ^, %, * and - operations on byte. But byte datatype of java is signed because of which I am facing problems in decryption. How can I read file or process read data with 1byte at a time which is unsigned?
thanks in advance.

byte b = <as read from file>;
int i = b & 0xFF;
Perform operations on i as required

The standard method InputStream.read() reads one byte and fits it into a int, so in practice it is an unsinged byte. There are no unsigned primitive data types in java, so the only approach is to fit it in an upper primitive.
That being said you should have no trouble performing encryption/decryption over data bytes read from the file, since the bytes are the same, no matter if they are interpreted as signed or unsigned (0xFF can be 255 or -1). You say the alghorithm contains "^, %, *", etc. That is an interpretation of raw bytes, taking into account a character encoding (that fits 8 bit per character I suppose). You should not perform encryption/decryption operations over other than raw bytes.

First, InputStream.read() returns an int but it holds a byte; it uses an int so -1 can be returned if the EOF is reached. If the int is not -1, you can cast it to byte.
Second, there are read() metods that allow storing the bytes directly in a byte[]
And last, if you are going to use the file as a byte[] (and it is not too big) maybe it would be interesting copying the data from FileInputStream and write it into a ByteArrayOutputStream. You can get the resulting byte[] from the late object (note: do not use the .read() method, use .read(byte[], int, int) for performance).

Since there is no unsigned primitive type in Java, I think what you can do is to convert signed byte into integer (which will virtually be unsigned because the integer will always be positive). You can follow the code in here: Can we make unsigned byte in Java for the conversion.