We Keep Coding

How to edit binary file in Android?

I want to create an app that can open binary file, read and edit it's content in decimal values, and then save it back to binary file.
This file that I want to open have multiple parameters written in 2 bytes. e.g. 00 06 00 50 01 AB = 6, 80, 427
So far I managed to open bin file, put all data from file into byte[].
Here is the code to open and read file:
String path = getFilesDir().getAbsolutePath();
File file = new File(path + "/file.bin");
int size = (int) file.length();
byte[] bytes = new byte[size];
FileInputStream fis = new FileInputStream(file);
try {
fis.read(bytes);
...
} finally {
fis.close();
}
Here is first attempt to convert byte[] into deciaml values:
for (int i = 0; i < size / 2; i = i + 2) {
intArray[i] = ((bytes[i] & 0xff) << 8) | (bytes[i + 1] & 0xff);
}
But in this code only every second value is correct. It's shows 6,0,80,0,1 and should be 6,80,1.
The second idea was to convert byte[] into String[] and then into e.g. int[] but It doesn't work as intended. In this fragment I didn't parse whole array, but only hexArray[4] which was 50 in hex and 80 in dec, but the result was 50.
for (int i = 0; i < size; i = i + 2) {
num1 = String.format("%02X", bytes[i] & 0xFF);
num2 = String.format("%02X", bytes[i + 1] & 0xFF);
String twoByte = num1 + num2;
hexArrayWithNull[i] = twoByte;
}
String[] hexArray = Arrays.stream(hexArrayWithNull).filter(Objects::nonNull).toArray(String[]::new);
num = String.format("%d", Integer.parseInt(hexArray[4]));
I'm quite stuck here and need some guidance. I must say also that I'm new in programming.
Thanks

You increment i in each iteration by 2 and you store in intArray[i]. So, you store in index [0] then index [2] then index [4] and so on. This explains the zeros at odd indices [1], [3], ...
Also, you must continue the loop until size, not size/2.
Here is the correct loop:
for (int i = 0; i < size; i = i + 2) {
intArray[i/2] = ((bytes[i] & 0xff) << 8) | (bytes[i + 1] & 0xff);
}
Here is the another possibility:
for (int i = 0; i < size/2; i = i + 1) {
intArray[i] = ((bytes[2*i] & 0xff) << 8) | (bytes[2*i + 1] & 0xff);
}
[EDIT]
The reverse operation would be:
for (int i = 0; i < size/2; i = i + 1) {
bytes[2*i] = (intArray[i] & 0xff00) / 256; //or >>8 instead of /256
bytes[2*i+1] = intArray[i] & 0xff;
}

Need python-code to be translated into in Java

Python Code
decoded = base64.b64decode(base64input)
resultBytes = b""
i = 0
while i < len(decoded):
c = decoded[i + 0] * 256 + decoded[i + 1]
d = decoded[i + 2] * 256 + decoded[i + 3]
lenRead = 0
gzchunk = (bytes((31,139,8,0)) + decoded[i:i+c])
try:
with gzip.GzipFile(fileobj=io.BytesIO(gzchunk)) as gf:
while True:
readSize = min(16384, d - lenRead)
readBytes = gf.read(size=readSize)
lenRead += len(readBytes)
resultBytes += readBytes
if len(readBytes) == 0 or (d - lenRead) <= 0:
break
except IOError as err:
pass # provide error message later
i += c + 4
i tried it with this Java Code, but it fails
// read file-content into byte array
byte[] decoded = null;
try {
decoded = IOUtils.toByteArray(new FileReader(fullFilePath), org.apache.commons.codec.Charsets.UTF_8);
} catch (Exception e) {
e.printStackTrace();
}
// Decode
byte[] fb = null;
try {
fb = StringUtils.newStringUtf8(Base64.decodeBase64(decoded)).getBytes("UTF-8");
} catch (Exception e1) {
e1.printStackTrace();
}
byte[] resultBytes = null;
int i = 0;
while (i < fb.length) {
int c = (fb[i + 0] * 256) + (fb[i + 1]);
int d = (fb[i + 2] * 256) + (fb[i + 3]);
int lenRead = 0;
byte[] a1 = convert2ByteArray(new int[] { 31, 139, 9, 0 });
byte[] a2 = Arrays.copyOfRange(fb, i, i + c);
byte[] gzchunk = copyByteArray(a1, a2);
GZIPInputStream gf = null;
byte[] readBytes;
int readSize;
try {
while (true) {
readSize = Math.min(16384, (d - lenRead));
gf = new GZIPInputStream(new ByteArrayInputStream(gzchunk), readSize);
int read = gf.read();
readBytes = ByteBuffer.allocate(4).putInt(read).array();
lenRead += readBytes.length;
resultBytes = copyByteArray(resultBytes, readBytes);
if (readBytes.length == 0 | (d - lenRead) <= 0) {
break;
}
}
} catch (Exception e) {
e.printStackTrace();
}
i += c + 4;
}
Thanks for your support
Update:
1. Different byte array values
More clarification on the failure. So, the for python
print(fb[i + 0])
print(fb[i + 1])
print(fb[i + 2])
print(fb[i + 4])
Output is:
30
208
234
120
with my Java code the output is:
30
-48
-22
96
2. IllegalArgumentException
java.lang.IllegalArgumentException: buffer size <= 0
on line
gf = new GZIPInputStream(new ByteArrayInputStream(gzchunk), readSize);
Update 2:
#Joop:
With your suggestion, I know have written the equivalent for the python code:
c = decoded[i + 0] * 256 + decoded[i + 1]
d = decoded[i + 2] * 256 + decoded[i + 3]
in java
int c= ((fb[i + 0] & 0xFF) << 8) | (fb[i + 1] & 0xFF);
int d= ((fb[i + 2] & 0xFF) << 8) + (fb[i + 3]);
But i still do receive different values for the same data:
Python:
c = 7888
d = 60000
Java:
c = 27375
d = 48829
Update 3:
Basically what i try to do is:
Base-64 decode the whole thing. Then on the decoded bytes:
Compute the next compressed length from the first two bytes in big-endian order. Call it c.
Compute the next uncompressed length from the next two bytes in big-endian order. Call it u.
Use zlib to decode the zlib stream consisting of the next c bytes. Verify that zlib decoded exactly c bytes, and that the resulting uncompressed data is exactly u bytes long.
Repeat starting at step 2 until the data is consumed. (It should all be exactly consumed.)

The error: The type byte is signed and to prevent sign extension as an int has to be masked (& 0xFF).
int c = (fb[i + 0] & 0xFF) << 8) | (fb[i + 1] & 0xFF);
The signed byte is between -128 and 127, so a1 could simply be written:
byte[] a1 = new byte[] { 31, (byte)139, 9, 0 };
Then you can use java to the fullest as:
// read file-content into byte array
Path path = Paths.get(fullFilePath);
byte[] decoded = Files.readAllBytes(path);
// Decode
byte[] fb = Base64.getDecoder().decode(decoded);
I would catch the exceptions at a larger scope, as one has to stop on a larger scope, and it's easier.
The loop I did not check; that is something to simplify as soon it works.
After more debug info in question:
The debugging code
print(fb[i + 4])
should be
print(fb[i + 3])
c is now correct; java delivering -48 instead of 208 is caused by byte being signed: 256-48 = 208 and 256-22 = 234. For d a bit of old code still messes the sign extension.
int d = ((fb[i + 2] & 0xFF) << 8) | (fb[i + 3] & 0xFF);
I tried myself on simplifying the loopings, no guarantee.
ByteArrayOutputStream out = new ByteArrayOutputStream();
int i = 0;
ByteBuffer inbuf = ByteBuffer.wrap(fb);
while (inbuf.hasRemaining()) {
int c = inbuf.getShort() & 0xFFFF;
int d = inbuf.getShort() & 0xFFFF;
assert c <= inbuf.limit();
byte[] gzchunk = new byte[4 + c];
gzchunk[0] = 31;
gzchunk[1] = (byte)139;
gzchunk[2] = 9;
gzchunk[3] = 0;
inbuf.get(gzchunk, 4, c);
byte[] readBytes = new byte[d];
GZIPInputStream gf = new GZIPInputStream(
new ByteArrayInputStream(gzchunk), d);
int nread = gf.read(readBytes, 0, d);
// No loop required as non-blocking ByteArrayInputStream.
assert nread == d;
out.write(readBytes);
gf.close();
i += 4 + c;
assert inbuf.position() == i;
}
out.close();
return out.toByteArray();
As there is no limit on 16K (python limit perhaps?), the reading becomes simpler. Instead of read one should use readAllBytes when java > 8. read can deliver a partial result on what is available. However a ByteArrayInputStream has all data available.
Using a ByteBuffer that by default has order ByteOrder.BIG_ENDIAN will allow getShort doing away with our calculation.

Get Bit values from Byte Array

I have a byte array from which I need to read specific bits and convert to int (see the byte array structure below). Even though the bits information I want to read is in 3 bytes, I tried reading 4 bytes (6-9) as integer and then read the bits from that integer value with bits or bitsValue method but somehow I am not able to see the right values from the bit manipulation. And with my expertise in bits I am pretty sure I am doing something wrong.
Can someone please suggest I am doing it correctly and why its not working. Thanks in Advance!!
Byte array is in Little Endian format.
0th byte - Some Value
1st Byte - Some Value
2nd - 5th Byte - Some Value
6th - 9th Byte - first 18 bits represent some value
- Next 5 bits represent some value
- Next 1 bit represent some value
- Last 8 bits represent some value
public class Test {
public static void main(String... dataProvider) {
String s = "46 00 ef 30 e9 08 cc a5 03 43";
byte[] bytes = new byte[s.length()];
bytes = hexStringToByteArray(s);
int bytePointer = 0;
int msgType = getIntFromSingleByte(bytes[bytePointer]); // 0th byte
int version = getIntFromSingleByte(bytes[++bytePointer]); // 1st byte
int tickInMS = getIntValue(bytes, ++bytePointer); // 2nd-5th bytes
bytePointer = bytePointer + 4;
int headr = getIntValue(bytes, bytePointer); // 6th-9th bytes
int utcTime = bits(headr, 0, 18); // 6th-9th bytes - 18 bits
int reserved = bits(headr, 18, 5); // 6th-9th bytes- 5 bits
int reportOrEvent = bits(headr, 23, 1); // 6th-9th bytes - 1 bits
int reportId = bitsValue(headr, 24, 32); // 6th-9th- 8 bits
}
public static int getIntFromSingleByte(byte data) {
return (data & 0xFF);
}
public static int getIntValue(byte[] bytes, int startPosition) {
byte[] dest = new byte[4];
System.arraycopy(bytes, startPosition, dest, 0, dest.length);
return toInt(dest);
}
// took from Stack overflow
static int bits(int n, int offset, int length) {
// shift the bits rightward, so that the desired chunk is at the right end
n = n >> (31 - offset - length);
// prepare a mask where only the rightmost `length` bits are 1's
int mask = ~(-1 << length);
// zero out all bits but the right chunk
return n & mask;
}
public static int bitsValue(int intNum, int startBitPos, int endBitPos) {
// parameters checking ignored for now
int tempValue = intNum << endBitPos;
return tempValue >> (startBitPos + endBitPos);
}
public static byte[] hexStringToByteArray(final String s) {
String[] splits = s.split(" ");
final byte[] data = new byte[splits.length];
for (int i = 0; i < splits.length; i++) {
char first = splits[i].length() < 2 ? '0' : splits[i].charAt(0);
char second = splits[i].length() < 2 ? splits[i].charAt(0) : splits[i].charAt(1);
data[i] = (byte) ((Character.digit(first, 16) << 4) + Character.digit(second, 16));
}
return data;
}
public static int toInt(byte[] data) {
if (data == null || data.length != 4)
return 0x0;
return (int) ((0xff & data[0]) << 24 | (0xff & data[1]) << 16 | (0xff & data[2]) << 8
| (0xff & data[3]) << 0);
}
}

Wrapping your input data in a ByteBuffer will simplify parsing and allow you to adjust endianness as necessary.
Your bits method is wrong. The constant 31 should be 32. Also, the method uses MSB 0 bit numbering, which is odd for little-endian data. You should confirm that your input is documented as using this bit numbering scheme.
Your bitsValue method is wrong too. May as well just use bits after fixing it.
This code is simpler and extracts the bit fields correctly:
public static void main(String... args) {
String s = "46 0 79 37 a8 3 9f 37 1 43 eb 7a f 3 3 fe c4 1 c5 4 c5 5e";
byte[] input = hexStringToByteArray(s);
// Wrap the input in a ByteBuffer for parsing. Adjust endianness if necessary.
ByteBuffer buffer = ByteBuffer.wrap(input).order(ByteOrder.BIG_ENDIAN);
int msgType = buffer.get() & 0xff;
int version = buffer.get() & 0xff;
int tickInMS = buffer.getInt();
int header = buffer.getInt();
int utcTime = bits(header, 0, 18); // 6th-9th bytes - 18 bits
int reserved = bits(header, 18, 5); // 6th-9th bytes - 5 bits
int reportOrEvent = bits(header, 23, 1); // 6th-9th bytes - 1 bit
int reportId = bits(header, 24, 8); // 6th-9th bytes - 8 bits
System.out.printf("utc: %d, report? %d, id: %d\n", utcTime, reportOrEvent, reportId);
}
/**
* Extract a bit field from an int. Bit numbering is MSB 0.
*/
public static int bits(int n, int offset, int length) {
return n >> (32 - offset - length) & ~(-1 << length);
}

Instead of error prone bit mangeling you should rather use the binary string representation of the numbers and do your "bit picking" as string operations:
String s = "46 00 ef 30 e9 08 cc a5 03 43";
String[] hexNumbers = s.split(" ");
for(String hexNumber : hexNumbers) {
String binaryNumber = String.format("%8s", new BigInteger(hexNumber,16).toString(2)).replace(' ','0');
System.out.print(String.format("value in hex : %s, value in binary: %s", hexNumber,binaryNumber));
}

JAVA JDBC Driver PostgreSQL: Parse numbers encoded as BYTEA object

I have relation where for each record there is BYTEA column (UTF-8) encoding 3 numbers in the following order:
bytes 0-1: number 1
bytes 2-3: number 2
bytes 4-6: number 3
How can I parse the binary data to readable numbers?
Currently I have this and don't know how to continue:
Class.forName(dbDriver);
Connection connection = DriverManager.getConnection(dbUrl, dbUser, dbPass);
Statement st = connection.createStatement();
String query = "SELECT ...";
ResultSet rs = st.executeQuery(query);
while (rs.next()) {
byte[] data = rs.getBytes(1);
//TODO Parse
}
Thanks,

That depends on how the numbers are stored.
Are they binary?
Are they signed?
Are they big or little endian?
Assuming yes to the first two, you can use bit-manipulation, e.g.
// Little-endian
short num1 = (short) ((data[0] & 0xFF) | (data[1] & 0xFF) << 8);
// Big-endian
short num1 = (short) ((data[0] & 0xFF) << 8 | (data[1] & 0xFF));
But it's probably easier to use ByteBuffer:
ByteBuffer buf = ByteBuffer.wrap(data).order(ByteOrder.LITTLE_ENDIAN);
short num1 = buf.getShort();
short num2 = buf.getShort();
short num3 = buf.getShort();
ByteBuffer is BIG_ENDIAN by default.

Can you try this :
For String :
String tatto = "my tatto"; //for example
byte[] array = tatto.getBytes(); // Or any bytes
String s = new String(array);
System.out.println(s);
For byte[] :
byte[] data = new byte[]{ 1, 16, 84, 2, 101, 110, 83, 111};
long val = 0;
for (int i = 0; i < data.length; i++)
{
val = (val << 8) + (data[i] & 0xff);
}
System.out.println(val);

byte values in hex format and writes these to the byte stream?

Takes a sequence of byte values in hex format and writes these to the byte stream.
"01 02 1a" => writes bytes 0x01 0x02 0x1a to the byte stream.
What does this mean?

Here is a possible solution:
String hex = "01 02 1a";
// Remove spaces
hex = hex.replace(" ", "");
// Array containing bytes
byte[] bytes = new byte[hex.length() / 2];
int k = 0;
for(int i=0; i < hex.length(); i = i +2 ) {
// Read and parse each byte
int b = Integer.parseInt(hex.substring(i, i + 2), 16);
bytes[k] = (byte) b;
k++;
}
// Write bytes to an outputStram
OutputStream out;
for(Byte b: bytes) {
out.write(b);
}