I have relation where for each record there is BYTEA column (UTF-8) encoding 3 numbers in the following order:
bytes 0-1: number 1
bytes 2-3: number 2
bytes 4-6: number 3
How can I parse the binary data to readable numbers?
Currently I have this and don't know how to continue:
Class.forName(dbDriver);
Connection connection = DriverManager.getConnection(dbUrl, dbUser, dbPass);
Statement st = connection.createStatement();
String query = "SELECT ...";
ResultSet rs = st.executeQuery(query);
while (rs.next()) {
byte[] data = rs.getBytes(1);
//TODO Parse
}
Thanks,
That depends on how the numbers are stored.
Are they binary?
Are they signed?
Are they big or little endian?
Assuming yes to the first two, you can use bit-manipulation, e.g.
// Little-endian
short num1 = (short) ((data[0] & 0xFF) | (data[1] & 0xFF) << 8);
// Big-endian
short num1 = (short) ((data[0] & 0xFF) << 8 | (data[1] & 0xFF));
But it's probably easier to use ByteBuffer:
ByteBuffer buf = ByteBuffer.wrap(data).order(ByteOrder.LITTLE_ENDIAN);
short num1 = buf.getShort();
short num2 = buf.getShort();
short num3 = buf.getShort();
ByteBuffer is BIG_ENDIAN by default.
Can you try this :
For String :
String tatto = "my tatto"; //for example
byte[] array = tatto.getBytes(); // Or any bytes
String s = new String(array);
System.out.println(s);
For byte[] :
byte[] data = new byte[]{ 1, 16, 84, 2, 101, 110, 83, 111};
long val = 0;
for (int i = 0; i < data.length; i++)
{
val = (val << 8) + (data[i] & 0xff);
}
System.out.println(val);
Related
I want to create an app that can open binary file, read and edit it's content in decimal values, and then save it back to binary file.
This file that I want to open have multiple parameters written in 2 bytes. e.g. 00 06 00 50 01 AB = 6, 80, 427
So far I managed to open bin file, put all data from file into byte[].
Here is the code to open and read file:
String path = getFilesDir().getAbsolutePath();
File file = new File(path + "/file.bin");
int size = (int) file.length();
byte[] bytes = new byte[size];
FileInputStream fis = new FileInputStream(file);
try {
fis.read(bytes);
...
} finally {
fis.close();
}
Here is first attempt to convert byte[] into deciaml values:
for (int i = 0; i < size / 2; i = i + 2) {
intArray[i] = ((bytes[i] & 0xff) << 8) | (bytes[i + 1] & 0xff);
}
But in this code only every second value is correct. It's shows 6,0,80,0,1 and should be 6,80,1.
The second idea was to convert byte[] into String[] and then into e.g. int[] but It doesn't work as intended. In this fragment I didn't parse whole array, but only hexArray[4] which was 50 in hex and 80 in dec, but the result was 50.
for (int i = 0; i < size; i = i + 2) {
num1 = String.format("%02X", bytes[i] & 0xFF);
num2 = String.format("%02X", bytes[i + 1] & 0xFF);
String twoByte = num1 + num2;
hexArrayWithNull[i] = twoByte;
}
String[] hexArray = Arrays.stream(hexArrayWithNull).filter(Objects::nonNull).toArray(String[]::new);
num = String.format("%d", Integer.parseInt(hexArray[4]));
I'm quite stuck here and need some guidance. I must say also that I'm new in programming.
Thanks
You increment i in each iteration by 2 and you store in intArray[i]. So, you store in index [0] then index [2] then index [4] and so on. This explains the zeros at odd indices [1], [3], ...
Also, you must continue the loop until size, not size/2.
Here is the correct loop:
for (int i = 0; i < size; i = i + 2) {
intArray[i/2] = ((bytes[i] & 0xff) << 8) | (bytes[i + 1] & 0xff);
}
Here is the another possibility:
for (int i = 0; i < size/2; i = i + 1) {
intArray[i] = ((bytes[2*i] & 0xff) << 8) | (bytes[2*i + 1] & 0xff);
}
[EDIT]
The reverse operation would be:
for (int i = 0; i < size/2; i = i + 1) {
bytes[2*i] = (intArray[i] & 0xff00) / 256; //or >>8 instead of /256
bytes[2*i+1] = intArray[i] & 0xff;
}
Python Code
decoded = base64.b64decode(base64input)
resultBytes = b""
i = 0
while i < len(decoded):
c = decoded[i + 0] * 256 + decoded[i + 1]
d = decoded[i + 2] * 256 + decoded[i + 3]
lenRead = 0
gzchunk = (bytes((31,139,8,0)) + decoded[i:i+c])
try:
with gzip.GzipFile(fileobj=io.BytesIO(gzchunk)) as gf:
while True:
readSize = min(16384, d - lenRead)
readBytes = gf.read(size=readSize)
lenRead += len(readBytes)
resultBytes += readBytes
if len(readBytes) == 0 or (d - lenRead) <= 0:
break
except IOError as err:
pass # provide error message later
i += c + 4
i tried it with this Java Code, but it fails
// read file-content into byte array
byte[] decoded = null;
try {
decoded = IOUtils.toByteArray(new FileReader(fullFilePath), org.apache.commons.codec.Charsets.UTF_8);
} catch (Exception e) {
e.printStackTrace();
}
// Decode
byte[] fb = null;
try {
fb = StringUtils.newStringUtf8(Base64.decodeBase64(decoded)).getBytes("UTF-8");
} catch (Exception e1) {
e1.printStackTrace();
}
byte[] resultBytes = null;
int i = 0;
while (i < fb.length) {
int c = (fb[i + 0] * 256) + (fb[i + 1]);
int d = (fb[i + 2] * 256) + (fb[i + 3]);
int lenRead = 0;
byte[] a1 = convert2ByteArray(new int[] { 31, 139, 9, 0 });
byte[] a2 = Arrays.copyOfRange(fb, i, i + c);
byte[] gzchunk = copyByteArray(a1, a2);
GZIPInputStream gf = null;
byte[] readBytes;
int readSize;
try {
while (true) {
readSize = Math.min(16384, (d - lenRead));
gf = new GZIPInputStream(new ByteArrayInputStream(gzchunk), readSize);
int read = gf.read();
readBytes = ByteBuffer.allocate(4).putInt(read).array();
lenRead += readBytes.length;
resultBytes = copyByteArray(resultBytes, readBytes);
if (readBytes.length == 0 | (d - lenRead) <= 0) {
break;
}
}
} catch (Exception e) {
e.printStackTrace();
}
i += c + 4;
}
Thanks for your support
Update:
1. Different byte array values
More clarification on the failure. So, the for python
print(fb[i + 0])
print(fb[i + 1])
print(fb[i + 2])
print(fb[i + 4])
Output is:
30
208
234
120
with my Java code the output is:
30
-48
-22
96
2. IllegalArgumentException
java.lang.IllegalArgumentException: buffer size <= 0
on line
gf = new GZIPInputStream(new ByteArrayInputStream(gzchunk), readSize);
Update 2:
#Joop:
With your suggestion, I know have written the equivalent for the python code:
c = decoded[i + 0] * 256 + decoded[i + 1]
d = decoded[i + 2] * 256 + decoded[i + 3]
in java
int c= ((fb[i + 0] & 0xFF) << 8) | (fb[i + 1] & 0xFF);
int d= ((fb[i + 2] & 0xFF) << 8) + (fb[i + 3]);
But i still do receive different values for the same data:
Python:
c = 7888
d = 60000
Java:
c = 27375
d = 48829
Update 3:
Basically what i try to do is:
Base-64 decode the whole thing. Then on the decoded bytes:
Compute the next compressed length from the first two bytes in big-endian order. Call it c.
Compute the next uncompressed length from the next two bytes in big-endian order. Call it u.
Use zlib to decode the zlib stream consisting of the next c bytes. Verify that zlib decoded exactly c bytes, and that the resulting uncompressed data is exactly u bytes long.
Repeat starting at step 2 until the data is consumed. (It should all be exactly consumed.)
The error: The type byte is signed and to prevent sign extension as an int has to be masked (& 0xFF).
int c = (fb[i + 0] & 0xFF) << 8) | (fb[i + 1] & 0xFF);
The signed byte is between -128 and 127, so a1 could simply be written:
byte[] a1 = new byte[] { 31, (byte)139, 9, 0 };
Then you can use java to the fullest as:
// read file-content into byte array
Path path = Paths.get(fullFilePath);
byte[] decoded = Files.readAllBytes(path);
// Decode
byte[] fb = Base64.getDecoder().decode(decoded);
I would catch the exceptions at a larger scope, as one has to stop on a larger scope, and it's easier.
The loop I did not check; that is something to simplify as soon it works.
After more debug info in question:
The debugging code
print(fb[i + 4])
should be
print(fb[i + 3])
c is now correct; java delivering -48 instead of 208 is caused by byte being signed: 256-48 = 208 and 256-22 = 234. For d a bit of old code still messes the sign extension.
int d = ((fb[i + 2] & 0xFF) << 8) | (fb[i + 3] & 0xFF);
I tried myself on simplifying the loopings, no guarantee.
ByteArrayOutputStream out = new ByteArrayOutputStream();
int i = 0;
ByteBuffer inbuf = ByteBuffer.wrap(fb);
while (inbuf.hasRemaining()) {
int c = inbuf.getShort() & 0xFFFF;
int d = inbuf.getShort() & 0xFFFF;
assert c <= inbuf.limit();
byte[] gzchunk = new byte[4 + c];
gzchunk[0] = 31;
gzchunk[1] = (byte)139;
gzchunk[2] = 9;
gzchunk[3] = 0;
inbuf.get(gzchunk, 4, c);
byte[] readBytes = new byte[d];
GZIPInputStream gf = new GZIPInputStream(
new ByteArrayInputStream(gzchunk), d);
int nread = gf.read(readBytes, 0, d);
// No loop required as non-blocking ByteArrayInputStream.
assert nread == d;
out.write(readBytes);
gf.close();
i += 4 + c;
assert inbuf.position() == i;
}
out.close();
return out.toByteArray();
As there is no limit on 16K (python limit perhaps?), the reading becomes simpler. Instead of read one should use readAllBytes when java > 8. read can deliver a partial result on what is available. However a ByteArrayInputStream has all data available.
Using a ByteBuffer that by default has order ByteOrder.BIG_ENDIAN will allow getShort doing away with our calculation.
I'm reading proc/net/tcp file from filereader and parsing data from regex to get what need .
The example string from proc/net/tcp is :
0: 0401A8C0:D366 FFB79E23:01BB 01 00000000:00000000 00:00000000 00000000 11269 0 14392479 1 00000000 30 4 30 10 -1
So the local address : port is : 0401A8C0:D366 , i have tried convert hex to string via that method but it doesn't return valid data .... Can someone help how to read datas ?
It should give something like : 192.168.*.* .
To parse the ip address should get bytearray from this 0401A8C0 Little Endian String , but can't solve
The Hex to String method :
public String fromHex(String hex) throws UnsupportedEncodingException {
hex = hex.replaceAll("^(00)+", "");
byte[] bytes = new byte[hex.length() / 2];
for (int i = 0; i < hex.length(); i += 2) {
bytes[i / 2] = (byte) ((Character.digit(hex.charAt(i), 16) << 4) + Character.digit(hex.charAt(i + 1), 16));
}
return new String(bytes);
}
I have solved it by converting the string : 0401A8C0 to byte array :
public static byte[] hexStringToByteArray(String s) {
int len = s.length();
byte[] data = new byte[len / 2];
for (int i = 0; i < len; i += 2) {
data[i / 2] = (byte) ((Character.digit(s.charAt(i), 16) << 4)
+ Character.digit(s.charAt(i+1), 16));
}
return data;
}
And than get IP address by :
InetAddress addresses = InetAddresses.fromLittleEndianByteArray(byte[]);
I know there are many similar questions in here, but I have some weird case. What I want to do is to convert a byte[4] to an int.
Here is the conversion from int to byte:
int data24bit = 51;
for (int i = 0; i < 3; i++) {
data8bit = (byte)(data24bit & 0x0000FF);
data24bit = data24bit >> 8;
byte_file.put(data8bit);
}
So far is clear enough. After that I want to read this 4 bytes to get the 51 back. I tried to do it by different ways:
Reading 4 bytes:
byte[] bytes = new byte[4];
for (int i = 0; i < 3; i++) {
byte b = dis.readByte();
bytes[i] = b;
}
// bytes[3] = (byte)(0x000000);
Convert bytes to int:
int value = 0;
value = ((0xFF & bytes[0]) << 24) | ((0xFF & bytes[1]) << 16) |
((0xFF & bytes[2]) << 8) | (0xFF & bytes[3]);
or
value = ByteBuffer.wrap(bytes).getInt();
or
value = new BigInteger(bytes).intValue();
I always get 855638016 as result where 51 is expected.
When I debug the code and look into the byte array I can see the following content: [51, 0, 0, 0].
What am I doing wrong?
The problem is that you're writing the bytes in little-endian (least significant byte first), but read it back assuming big-endian.
After writing it out, your byte array looks like this:
[51, 0, 0, 0]
Then you're trying to convert that back into an integer, like in this example from your post:
value = ((0xFF & bytes[0]) << 24)
| ((0xFF & bytes[1]) << 16)
| ((0xFF & bytes[2]) << 8)
| (0xFF & bytes[3]);
If you fill in the actual values, that calculation is basically this:
value = 51 * 256 * 256 * 256
+ 0 * 256 * 256
+ 0 * 256
+ 0
= 855638016
While what you actually want is this:
value = 0 * 256 * 256 * 256
+ 0 * 256 * 256
+ 0 * 256
+ 51
= 51
The fixed calculation would thus be this:
value = ((0xFF & bytes[3]) << 24)
| ((0xFF & bytes[2]) << 16)
| ((0xFF & bytes[1]) << 8)
| (0xFF & bytes[0]);
Ok stupid enough but I just didn't preserve the byte order.
[51, 0, 0, 0] -> is 855638016
[0, 0, 0, 51] -> is 51
I have an 8 byte array and I want to convert it to its corresponding numeric value.
e.g.
byte[] by = new byte[8]; // the byte array is stored in 'by'
// CONVERSION OPERATION
// return the numeric value
I want a method that will perform the above conversion operation.
One could use the Buffers that are provided as part of the java.nio package to perform the conversion.
Here, the source byte[] array has a of length 8, which is the size that corresponds with a long value.
First, the byte[] array is wrapped in a ByteBuffer, and then the ByteBuffer.getLong method is called to obtain the long value:
ByteBuffer bb = ByteBuffer.wrap(new byte[] {0, 0, 0, 0, 0, 0, 0, 4});
long l = bb.getLong();
System.out.println(l);
Result
4
I'd like to thank dfa for pointing out the ByteBuffer.getLong method in the comments.
Although it may not be applicable in this situation, the beauty of the Buffers come with looking at an array with multiple values.
For example, if we had a 8 byte array, and we wanted to view it as two int values, we could wrap the byte[] array in an ByteBuffer, which is viewed as a IntBuffer and obtain the values by IntBuffer.get:
ByteBuffer bb = ByteBuffer.wrap(new byte[] {0, 0, 0, 1, 0, 0, 0, 4});
IntBuffer ib = bb.asIntBuffer();
int i0 = ib.get(0);
int i1 = ib.get(1);
System.out.println(i0);
System.out.println(i1);
Result:
1
4
Assuming the first byte is the least significant byte:
long value = 0;
for (int i = 0; i < by.length; i++)
{
value += ((long) by[i] & 0xffL) << (8 * i);
}
Is the first byte the most significant, then it is a little bit different:
long value = 0;
for (int i = 0; i < by.length; i++)
{
value = (value << 8) + (by[i] & 0xff);
}
Replace long with BigInteger, if you have more than 8 bytes.
Thanks to Aaron Digulla for the correction of my errors.
If this is an 8-bytes numeric value, you can try:
BigInteger n = new BigInteger(byteArray);
If this is an UTF-8 character buffer, then you can try:
BigInteger n = new BigInteger(new String(byteArray, "UTF-8"));
Simply, you could use or refer to guava lib provided by google, which offers utiliy methods for conversion between long and byte array. My client code:
long content = 212000607777l;
byte[] numberByte = Longs.toByteArray(content);
logger.info(Longs.fromByteArray(numberByte));
You can also use BigInteger for variable length bytes. You can convert it to Long, Integer or Short, whichever suits your needs.
new BigInteger(bytes).intValue();
or to denote polarity:
new BigInteger(1, bytes).intValue();
Complete java converter code for all primitive types to/from arrays
http://www.daniweb.com/code/snippet216874.html
Each cell in the array is treated as unsigned int:
private int unsignedIntFromByteArray(byte[] bytes) {
int res = 0;
if (bytes == null)
return res;
for (int i=0;i<bytes.length;i++){
res = res | ((bytes[i] & 0xff) << i*8);
}
return res;
}
public static long byteArrayToLong(byte[] bytes) {
return ((long) (bytes[0]) << 56)
+ (((long) bytes[1] & 0xFF) << 48)
+ ((long) (bytes[2] & 0xFF) << 40)
+ ((long) (bytes[3] & 0xFF) << 32)
+ ((long) (bytes[4] & 0xFF) << 24)
+ ((bytes[5] & 0xFF) << 16)
+ ((bytes[6] & 0xFF) << 8)
+ (bytes[7] & 0xFF);
}
convert bytes array (long is 8 bytes) to long
You can try use the code from this answer: https://stackoverflow.com/a/68393576/7918717
It parses bytes as a signed number of arbitrary length. A few examples:
bytesToSignedNumber(false, 0xF1, 0x01, 0x04) returns 15794436 (3 bytes as int)
bytesToSignedNumber(false, 0xF1, 0x01, 0x01, 0x04) returns -251592444 (4 bytes as int)
bytesToSignedNumber(false, 0xF1, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x04) returns -1080581331768770303 (8 of 9 bytes as long)