I want to create an app that can open binary file, read and edit it's content in decimal values, and then save it back to binary file.
This file that I want to open have multiple parameters written in 2 bytes. e.g. 00 06 00 50 01 AB = 6, 80, 427
So far I managed to open bin file, put all data from file into byte[].
Here is the code to open and read file:
String path = getFilesDir().getAbsolutePath();
File file = new File(path + "/file.bin");
int size = (int) file.length();
byte[] bytes = new byte[size];
FileInputStream fis = new FileInputStream(file);
try {
fis.read(bytes);
...
} finally {
fis.close();
}
Here is first attempt to convert byte[] into deciaml values:
for (int i = 0; i < size / 2; i = i + 2) {
intArray[i] = ((bytes[i] & 0xff) << 8) | (bytes[i + 1] & 0xff);
}
But in this code only every second value is correct. It's shows 6,0,80,0,1 and should be 6,80,1.
The second idea was to convert byte[] into String[] and then into e.g. int[] but It doesn't work as intended. In this fragment I didn't parse whole array, but only hexArray[4] which was 50 in hex and 80 in dec, but the result was 50.
for (int i = 0; i < size; i = i + 2) {
num1 = String.format("%02X", bytes[i] & 0xFF);
num2 = String.format("%02X", bytes[i + 1] & 0xFF);
String twoByte = num1 + num2;
hexArrayWithNull[i] = twoByte;
}
String[] hexArray = Arrays.stream(hexArrayWithNull).filter(Objects::nonNull).toArray(String[]::new);
num = String.format("%d", Integer.parseInt(hexArray[4]));
I'm quite stuck here and need some guidance. I must say also that I'm new in programming.
Thanks
You increment i in each iteration by 2 and you store in intArray[i]. So, you store in index [0] then index [2] then index [4] and so on. This explains the zeros at odd indices [1], [3], ...
Also, you must continue the loop until size, not size/2.
Here is the correct loop:
for (int i = 0; i < size; i = i + 2) {
intArray[i/2] = ((bytes[i] & 0xff) << 8) | (bytes[i + 1] & 0xff);
}
Here is the another possibility:
for (int i = 0; i < size/2; i = i + 1) {
intArray[i] = ((bytes[2*i] & 0xff) << 8) | (bytes[2*i + 1] & 0xff);
}
[EDIT]
The reverse operation would be:
for (int i = 0; i < size/2; i = i + 1) {
bytes[2*i] = (intArray[i] & 0xff00) / 256; //or >>8 instead of /256
bytes[2*i+1] = intArray[i] & 0xff;
}
Related
Python Code
decoded = base64.b64decode(base64input)
resultBytes = b""
i = 0
while i < len(decoded):
c = decoded[i + 0] * 256 + decoded[i + 1]
d = decoded[i + 2] * 256 + decoded[i + 3]
lenRead = 0
gzchunk = (bytes((31,139,8,0)) + decoded[i:i+c])
try:
with gzip.GzipFile(fileobj=io.BytesIO(gzchunk)) as gf:
while True:
readSize = min(16384, d - lenRead)
readBytes = gf.read(size=readSize)
lenRead += len(readBytes)
resultBytes += readBytes
if len(readBytes) == 0 or (d - lenRead) <= 0:
break
except IOError as err:
pass # provide error message later
i += c + 4
i tried it with this Java Code, but it fails
// read file-content into byte array
byte[] decoded = null;
try {
decoded = IOUtils.toByteArray(new FileReader(fullFilePath), org.apache.commons.codec.Charsets.UTF_8);
} catch (Exception e) {
e.printStackTrace();
}
// Decode
byte[] fb = null;
try {
fb = StringUtils.newStringUtf8(Base64.decodeBase64(decoded)).getBytes("UTF-8");
} catch (Exception e1) {
e1.printStackTrace();
}
byte[] resultBytes = null;
int i = 0;
while (i < fb.length) {
int c = (fb[i + 0] * 256) + (fb[i + 1]);
int d = (fb[i + 2] * 256) + (fb[i + 3]);
int lenRead = 0;
byte[] a1 = convert2ByteArray(new int[] { 31, 139, 9, 0 });
byte[] a2 = Arrays.copyOfRange(fb, i, i + c);
byte[] gzchunk = copyByteArray(a1, a2);
GZIPInputStream gf = null;
byte[] readBytes;
int readSize;
try {
while (true) {
readSize = Math.min(16384, (d - lenRead));
gf = new GZIPInputStream(new ByteArrayInputStream(gzchunk), readSize);
int read = gf.read();
readBytes = ByteBuffer.allocate(4).putInt(read).array();
lenRead += readBytes.length;
resultBytes = copyByteArray(resultBytes, readBytes);
if (readBytes.length == 0 | (d - lenRead) <= 0) {
break;
}
}
} catch (Exception e) {
e.printStackTrace();
}
i += c + 4;
}
Thanks for your support
Update:
1. Different byte array values
More clarification on the failure. So, the for python
print(fb[i + 0])
print(fb[i + 1])
print(fb[i + 2])
print(fb[i + 4])
Output is:
30
208
234
120
with my Java code the output is:
30
-48
-22
96
2. IllegalArgumentException
java.lang.IllegalArgumentException: buffer size <= 0
on line
gf = new GZIPInputStream(new ByteArrayInputStream(gzchunk), readSize);
Update 2:
#Joop:
With your suggestion, I know have written the equivalent for the python code:
c = decoded[i + 0] * 256 + decoded[i + 1]
d = decoded[i + 2] * 256 + decoded[i + 3]
in java
int c= ((fb[i + 0] & 0xFF) << 8) | (fb[i + 1] & 0xFF);
int d= ((fb[i + 2] & 0xFF) << 8) + (fb[i + 3]);
But i still do receive different values for the same data:
Python:
c = 7888
d = 60000
Java:
c = 27375
d = 48829
Update 3:
Basically what i try to do is:
Base-64 decode the whole thing. Then on the decoded bytes:
Compute the next compressed length from the first two bytes in big-endian order. Call it c.
Compute the next uncompressed length from the next two bytes in big-endian order. Call it u.
Use zlib to decode the zlib stream consisting of the next c bytes. Verify that zlib decoded exactly c bytes, and that the resulting uncompressed data is exactly u bytes long.
Repeat starting at step 2 until the data is consumed. (It should all be exactly consumed.)
The error: The type byte is signed and to prevent sign extension as an int has to be masked (& 0xFF).
int c = (fb[i + 0] & 0xFF) << 8) | (fb[i + 1] & 0xFF);
The signed byte is between -128 and 127, so a1 could simply be written:
byte[] a1 = new byte[] { 31, (byte)139, 9, 0 };
Then you can use java to the fullest as:
// read file-content into byte array
Path path = Paths.get(fullFilePath);
byte[] decoded = Files.readAllBytes(path);
// Decode
byte[] fb = Base64.getDecoder().decode(decoded);
I would catch the exceptions at a larger scope, as one has to stop on a larger scope, and it's easier.
The loop I did not check; that is something to simplify as soon it works.
After more debug info in question:
The debugging code
print(fb[i + 4])
should be
print(fb[i + 3])
c is now correct; java delivering -48 instead of 208 is caused by byte being signed: 256-48 = 208 and 256-22 = 234. For d a bit of old code still messes the sign extension.
int d = ((fb[i + 2] & 0xFF) << 8) | (fb[i + 3] & 0xFF);
I tried myself on simplifying the loopings, no guarantee.
ByteArrayOutputStream out = new ByteArrayOutputStream();
int i = 0;
ByteBuffer inbuf = ByteBuffer.wrap(fb);
while (inbuf.hasRemaining()) {
int c = inbuf.getShort() & 0xFFFF;
int d = inbuf.getShort() & 0xFFFF;
assert c <= inbuf.limit();
byte[] gzchunk = new byte[4 + c];
gzchunk[0] = 31;
gzchunk[1] = (byte)139;
gzchunk[2] = 9;
gzchunk[3] = 0;
inbuf.get(gzchunk, 4, c);
byte[] readBytes = new byte[d];
GZIPInputStream gf = new GZIPInputStream(
new ByteArrayInputStream(gzchunk), d);
int nread = gf.read(readBytes, 0, d);
// No loop required as non-blocking ByteArrayInputStream.
assert nread == d;
out.write(readBytes);
gf.close();
i += 4 + c;
assert inbuf.position() == i;
}
out.close();
return out.toByteArray();
As there is no limit on 16K (python limit perhaps?), the reading becomes simpler. Instead of read one should use readAllBytes when java > 8. read can deliver a partial result on what is available. However a ByteArrayInputStream has all data available.
Using a ByteBuffer that by default has order ByteOrder.BIG_ENDIAN will allow getShort doing away with our calculation.
I'm reading proc/net/tcp file from filereader and parsing data from regex to get what need .
The example string from proc/net/tcp is :
0: 0401A8C0:D366 FFB79E23:01BB 01 00000000:00000000 00:00000000 00000000 11269 0 14392479 1 00000000 30 4 30 10 -1
So the local address : port is : 0401A8C0:D366 , i have tried convert hex to string via that method but it doesn't return valid data .... Can someone help how to read datas ?
It should give something like : 192.168.*.* .
To parse the ip address should get bytearray from this 0401A8C0 Little Endian String , but can't solve
The Hex to String method :
public String fromHex(String hex) throws UnsupportedEncodingException {
hex = hex.replaceAll("^(00)+", "");
byte[] bytes = new byte[hex.length() / 2];
for (int i = 0; i < hex.length(); i += 2) {
bytes[i / 2] = (byte) ((Character.digit(hex.charAt(i), 16) << 4) + Character.digit(hex.charAt(i + 1), 16));
}
return new String(bytes);
}
I have solved it by converting the string : 0401A8C0 to byte array :
public static byte[] hexStringToByteArray(String s) {
int len = s.length();
byte[] data = new byte[len / 2];
for (int i = 0; i < len; i += 2) {
data[i / 2] = (byte) ((Character.digit(s.charAt(i), 16) << 4)
+ Character.digit(s.charAt(i+1), 16));
}
return data;
}
And than get IP address by :
InetAddress addresses = InetAddresses.fromLittleEndianByteArray(byte[]);
I have been attempting to write a java HTTP server from scratch for my chat program that sends data over websockets and everything has been going well until I started trying to send large images with my program using google chrome. I discovered that when chrome is sending multiple frames of data to my server, it never sends the proper length of the data. The first frame is always fine, but all frames after the initial one send a number that is significantly smaller than the data that the frame contains. For example, it will send a frame saying that it contains 123 bytes of data, when it is really sending over 100 kilobytes. I have tested other browsers like firefox and safari (on my phone) and they don't seem to split up large data into frames like chrome does, so they don't have the same issue. Here are some screenshots of my results:
Initial frame:
enter image description here
Following frames (This is where it fails):
enter image description here
And here is my server side code that gets the data from an input stream and writes it to a file:
byte[] headerdata = new byte[12];
input.read(headerdata, 0, 2);
int frame = (headerdata[0] & 0b10000000) & 0xFF;
long length = (headerdata[1] & 0b01111111) & 0xFF;
System.out.format("Before: %d, Frame: %d%n", length, frame);
if(length == 127){
input.read(headerdata, 0, 12);
length = ((long)(headerdata[0] << 56) & 0xFFFFFFFFFFFFFFFFL | (long)(headerdata[1] << 48) & 0xFFFFFFFFFFFFFFL | (long)(headerdata[2] << 40) & 0xFFFFFFFFFFFFL | (long)(headerdata[3] << 32) & 0xFFFFFFFFFFL | (headerdata[4] << 24) & 0xFFFFFFFFL | (headerdata[5] << 16) & 0xFFFFFFL | (headerdata[6] << 8) & 0xFFFFL | headerdata[7] & 0xFFL)
} else if(length == 126){
input.read(headerdata, 6, 6);
length = ((headerdata[6] << 8) & 0xFFFF | headerdata[7] & 0xFF);
}else{
input.read(headerdata, 8, 4);
}
System.out.println("After: " + length);
int count;
int total = 0;
byte[] buffer = new byte[16384];
boolean done = false;
while(done == false){
count = input.read(buffer, 0, buffer.length);
//System.out.println(count);
for(int i = 0; i < count; i++){
buffer[i] = (byte)(buffer[i] ^ headerdata[(total & 3) + 8]);
total++;
}
fileoutput.write(buffer, 0, count);
System.out.format("Count: %d Left: %d%n", count, (length - total));
if(total >= length){
done = true;
}
}
if(frame == 128) break;
Any explanation as to why I might be failing to read the length of following frames would be greatly appreciated (I have a few suspicions of my own as well). Thank you for your time and efforts.
I found out that the buffer ends up overflowing, resulting in data being misread or lost, which is why I was getting the wrong length values from the frames. I fixed it by adding an if statement that checks to see if the number of bytes read are approaching the length value of that frame, and then only read what is left instead of the full buffer size to stop it from overflowing. Here is the code with the modification:
byte[] headerdata = new byte[12];
input.read(headerdata, 0, 2);
int frame = (headerdata[0] & 0b10000000) & 0xFF;
long length = (headerdata[1] & 0b01111111) & 0xFF;
System.out.format("Before: %d, Frame: %d%n", length, frame);
if(length == 127){
input.read(headerdata, 0, 12);
length = ((long)(headerdata[0] << 56) & 0xFFFFFFFFFFFFFFFFL | (long)(headerdata[1] << 48) & 0xFFFFFFFFFFFFFFL | (long)(headerdata[2] << 40) & 0xFFFFFFFFFFFFL | (long)(headerdata[3] << 32) & 0xFFFFFFFFFFL | (headerdata[4] << 24) & 0xFFFFFFFFL | (headerdata[5] << 16) & 0xFFFFFFL | (headerdata[6] << 8) & 0xFFFFL | headerdata[7] & 0xFFL);
}else if(length == 126){
input.read(headerdata, 6, 6);
length = ((headerdata[6] << 8) & 0xFFFF | headerdata[7] & 0xFF);
}else{
input.read(headerdata, 8, 4);
}
System.out.println("After: " + length);
int count;
int total = 0;
byte[] buffer = new byte[16384];
boolean done = false;
while(done == false){
if(length - total > buffer.length) count = input.read(buffer, 0, buffer.length);
else count = input.read(buffer, 0, (int) (length - total));
//System.out.println(count);
for(int i = 0; i < count; i++){
buffer[i] = (byte)(buffer[i] ^ headerdata[(total & 3) + 8]);
total++;
}
fileoutput.write(buffer, 0, count);
//System.out.format("Count: %d Left: %d%n", count, (length - total));
if(total >= length){
done = true;
}
}
if(frame == 128) break;
I also did some research on different ways to read byte streams from a socket and found that it would also be possible to use DataInputStream.readFully().
So I'm trying to understand base64 encoding better and I came across this implementation on wikipedia
private static String base64Encode(byte[] in) {
StringBuffer out = new StringBuffer((in.length * 4) / 3);
int b;
for (int i = 0; i < in.length; i += 3) {
b = (in[i] & 0xFC) >> 2;
out.append(codes.charAt(b));
b = (in[i] & 0x03) << 4;
if (i + 1 < in.length) {
b |= (in[i + 1] & 0xF0) >> 4;
out.append(codes.charAt(b));
b = (in[i + 1] & 0x0F) << 2;
if (i + 2 < in.length) {
b |= (in[i + 2] & 0xC0) >> 6;
out.append(codes.charAt(b));
b = in[i + 2] & 0x3F;
out.append(codes.charAt(b));
} else {
out.append(codes.charAt(b));
out.append('=');
}
} else {
out.append(codes.charAt(b));
out.append("==");
}
}
return out.toString();
}
And I'm following along and I get to the line:
b = (in[i] & 0xFC) >> 2;
and I don't get it...why would you bitwise and 252 to a number then shift it right 2...wouldn't it be the same if you just shifted the byte itself without doing the bitwise operation? example:
b = in[i] >> 2;
Say my in[i] was the letter e...represented as 101 or in binary 01100101. If I shift that 2 to the right I get 011001 or 25. If I bitwise & it I get
01100101
11111100
--------
01100100
but then the shift is going to chop off the last 2 anyway...so why bother doing it?
Can somebody clarify for me please. Thanks.
IN in[i] >> 2, in[i] is converted to an int first. If it was a negative byte (with the high bit set) it will be converted to a negative int (with the now-highest 24 bits set as well).
In (in[i] & 0xFC) >> 2, in[i] is converted to an int as above, and then & 0xFC makes sure the extra bits are all reset to 0.
You're partially right, in that (in[i] & 0xFF) >> 2 would give the same result. & 0xFF is a common way to convert a byte to a non-negative int in the range 0 to 255.
The only way to know for sure why the original developer used 0xFC, and not 0xFF, is to ask them - but I speculate that it's to make it more obvious which bits are being used.
I can't understand how to get RGBA values from a short[] that I get from bufferedImage.getRaster().dataBuffer.getData() if dataBuffer is an instance of DataBufferUShort.
How to convert these values (that can be even -30000) to 0..255? If dataBuffer is an instance of DataBufferByte I can simply make something like this:
result[i] = (array[i] < 0) ?
array[i] + 256 :
array[i];
But what should I do with DataBufferUShort? Some PNG images has this type instead of expecting DataBufferByte.
getType() returns TYPE_CUSTOM. Here is the image: http://i.stack.imgur.com/YwmkO.png
A DataBufferUShort can hold multiple types of samples, so you first need to determine what data you have. Here are the most common ones:
If the image data represents 16 bit gray samples, all you need to do scale each value down to get an 8 bit gray value. You can do that by shifting the values 8 bits to the right.
DataBufferUShort dataBuffer;
short[] data = dataBuffer.getData();
byte[] gray = new byte[data.length];
for (int i = 0; i < data.length; i++) {
gray[i] = (byte) ((data[i] & 0xff00) >> 8);
}
If the image data represents 16 bits per sample (A)RGB values, you can do just the same as above, you will just have 3 (or 4 if there's alpha) samples or array elements per pixel instead of one.
If the data represents 16 bits per sample ARGB (as seems to be the case with your sample), you can also convert to int packed ARGB samples, like this:
DataBufferUShort dataBuffer;
short[] data = dataBuffer.getData();
int[] argb = new byte[data.length / 4];
for (int i = 0; i < data.length; += 4) {
int a = (data[i ] & 0xff00) >> 8;
int r = (data[i + 1] & 0xff00) >> 8;
int g = (data[i + 2] & 0xff00) >> 8;
int b = (data[i + 3] & 0xff00) >> 8;
argb[i / 4] = a << 24 | r << 16 | g << 8 | b;
}
If the image data represents 15/16 bit RGB (like the TYPE_USHORT_555_RGB or TYPE_USHORT_565_RGB) you'll have to scale the RGB values up to the full 8 bit/sample range. Something like:
DataBufferUShort dataBuffer;
short[] data = dataBuffer.getData();
byte[] rgb = new byte[data.length * 3];
for (int i = 0; i < data.length; i++) {
int shortRGB = data[i] & 0xffff;
// Assuming 5 bit R, 5 bit G, 5 bit B, using the lower 15 bits
rgb[i * 3 + 0] = ((((shortRGB & 0x7C00) >> 10) + 1) * 8) - 1;
rgb[i * 3 + 1] = ((((shortRGB & 0x03E0) >> 5) + 1) * 8) - 1;
rgb[i * 3 + 2] = ((((shortRGB & 0x001F) ) + 1) * 8) - 1;
}
For 565 RGB or even 4444 ARGB (as used in some Android devices), the procedure is very similar.