I am experiencing some sort of difficulties with storing structure {int, int, long} as byte array in java and reading it as binary structure in Cpp.
I have tried nearly everything. My biggest success was when I could read Long value properly, but integers were some random numbers.
I am affraid of endianness and I am not sure how can I decide which language use little or big endianness. Can anybody, please, tell me, how can I store primitive types such as int, long, double in java and read it in Cpp?
Thank you, it would be really helpful.
EDIT:
I know how do I want to read it in C++:
struct tick {
int x;
int y;
long time;
};
...
tick helpStruct;
input.open("test_file", ios_base::in | ios_base::binary);
input.read((char*) &helpStruct, sizeof(tick));
In Java, I've tried many ways, my last try was:
DataOutput stream = new DataOutputStream(new FileOutputStream(new File("test_file")));
byte[] bytes = ByteBuffer.allocate(4).order(ByteOrder.LITTLE_ENDIAN).putInt(1).array();
for (byte b : bytes) {
stream.write(b);
}
but Java code is open.
You wrote only the very first integer.. You never wrote the second one followed by the long..
Thus any values you read would be random of course. Just remember that sizeof(long) in C++ might not actually be 8 as it is in java! Also don't forget that the structure in C++ might be padded and it'd be better to read each value one at a time into the struct's fields.
This works..
On the java side:
package test;
import java.io.*;
import java.nio.*;
public class Test {
public static void main(String[] args) throws FileNotFoundException, IOException {
DataOutput stream = new DataOutputStream(new FileOutputStream(new File("C:/Users/Brandon/Desktop/test_file.dat")));
int sizeofint = 4;
int sizeoflong = 4;
ByteBuffer buffer = ByteBuffer.allocate(sizeofint + sizeofint + sizeoflong).order(ByteOrder.LITTLE_ENDIAN);
buffer.putInt(5).putInt(6).putInt(7);
byte[] bytes = buffer.array();
for (byte b : bytes) {
stream.write(b);
}
}
}
and on the C++ side:
#include <fstream>
#include <iostream>
struct tick
{
int x;
int y;
long time;
};
int main()
{
std::fstream file("C:/Users/Brandon/Desktop/test_file.dat", std::ios::in | std::ios::binary);
if (file.is_open())
{
tick t = {0};
file.read(reinterpret_cast<char*>(&t), sizeof(t));
file.close();
std::cout<<t.x<<" "<<t.y<<" "<<t.time<<"\n";
}
}
Results are: 5 6 7.
It might even be better to do:
file.read(reinterpret_cast<char*>(&t.x), sizeof(t.x));
file.read(reinterpret_cast<char*>(&t.y), sizeof(t.y));
file.read(reinterpret_cast<char*>(&t.time), sizeof(t.time));
Related
I am working on a BitBuffer that will take x bits from a ByteBuffer as an int, long, etc, but I seem to be having a whole lot of problems.
I've tried loading a long at a time and using bit shifting, but the difficulty comes from rolling from one long into the next. I am wondering if there's just a better way. Anyone have any suggestions?
public class BitBuffer
{
final private ByteBuffer bb;
public BitBuffer(byte[] bytes)
{
this.bb = ByteBuffer.wrap(bytes);
}
public int takeInt(int bits)
{
int bytes = toBytes(bits);
if (bytes > 4) throw new RuntimeException("Too many bits requested");
int i=0;
// take bits from bb and fill it into an int
return i;
}
}
More specifically, I am trying to take x bits from the buffer and return them as an int (the minimal case). I can access bytes from the buffer, but let's say I only want to take just the first 4 bits instead.
Example:
If my buffer is filled with "101100001111", if I run these in order:
takeInt(4) // should return 11 (1011)
takeInt(2) // should return 0 (00)
takeInt(2) // should return 0 (00)
takeInt(1) // should return 1 (1)
takeInt(3) // should return 7 (111)
I would like to use something like this for bit packed encoded data where an integer can be stored in just a few bits of a byte.
The BitSet and ByteBuffer ideas were a bit too difficult to control so instead, I went with a binary string approach that basically takes a whole lot of headache out of managing an intermediate buffer of bits.
public class BitBuffer
{
final private String bin;
private int start;
public BitBuffer(byte[] bytes)
{
this.bin = toBinaryString(bytes); // TODO: create this function
this.start = 0;
}
public int takeInt(int nbits)
{
// TODO: handle edge cases
String bits = bin.substring(start, start+=nbits);
return Integer.parseInt(bits, 2);
}
}
Out of everything I've tried this was the cleanest and easiest approach, but I am open to suggestions!
You can convert the ByteBuffer into BitSet and then you'll have continuous access to the bits
public class BitBuffer
{
final private BitSet bs;
public BitBuffer(byte[] bytes)
{
this.bs = BitSet.valueOf(bytes);
}
public int takeInt(int bits)
{
int bytes = toBytes(bits);
if (bytes > 4) throw new RuntimeException("Too many bits requested");
int i=0;
// take bits from bs and fill it into an int
return i;
}
}
Hello boys and girls.
I'm developing a terminal based client application which communicates over TCP/IP to server and sends and receives an arbitary number of raw bytes. Each byte represents a command which I need to parse to Java classes representing these commands, for further use.
My question how I should parse these bytes efficiently. I don't want to end up with bunch of nested ifs and switch-cases.
I have the data classes for these commands ready to go. I just need to figure out the proper way of doing the parsing.
Here's some sample specifications:
Byte stream can be for example in
integers:[1,24,2,65,26,18,3,0,239,19,0,14,0,42,65,110,110,97,32,109,121,121,106,228,42,15,20,5,149,45,87]
First byte is 0x01 which is start of header containing only one byte.
Second one is the length which is the number of bytes in certain
commands, only one byte here also.
The next can be any command where the first byte is the command, 0x02
in this case, and it follows n number of bytes which are included in
the command.
So on. In the end there are checksum related bytes.
Sample class representing the set_cursor command:
/**
* Sets the cursor position.
* Syntax: 0x0E | position
*/
public class SET_CURSOR {
private final int hexCommand = 0x0e;
private int position;
public SET_CURSOR(int position) {
}
public int getPosition() {
return position;
}
public int getHexCommnad() {
return hexCommand;
}
}
When parsing byte streams like this the best Design Pattern to use is the Command Pattern. Each of the different Commands will act as handlers to process the next several bytes in the stream.
interface Command{
//depending on your situation,
//either use InputStream if you don't know
//how many bytes each Command will use
// or the the commands will use an unknown number of bytes
//or a large number of bytes that performance
//would be affected by copying everything.
void execute(InputStream in);
//or you can use an array if the
//if the number of bytes is known and small.
void execute( byte[] data);
}
Then you can have a map containing each Command object for each of the byte "opcodes".
Map<Byte, Command> commands = ...
commands.put(Byte.parseByte("0x0e", 16), new SetCursorCommand() );
...
Then you can parse the message and act on the Commands:
InputStream in = ... //our byte array as inputstream
byte header = (byte)in.read();
int length = in.read();
byte commandKey = (byte)in.read();
byte[] data = new byte[length]
in.read(data);
Command command = commands.get(commandKey);
command.execute(data);
Can you have multiple Commands in the same byte message? If so you could then easily wrap the Command fetching and parsing in a loop until the EOF.
you can try JBBP library for that https://github.com/raydac/java-binary-block-parser
#Bin class Parsed { byte header; byte command; byte [] data; int checksum;}
Parsed parsed = JBBPParser.prepare("byte header; ubyte len; byte command; byte [len] data; int checksum;").parse(theArray).mapTo(Parsed.class);
This is a huge and complex subject.
It depends on the type of the data that you will read.
Is it a looooong stream ?
Is it a lot of small independent structures/objects ?
Do you have some references between structures/objects of your flow ?
I recently wrote a byte serialization/deserialization library for a proprietary software.
I took a visitor-like approach with type conversion, the same way JAXB works.
I define my object as a Java class. Initialize the parser on the class, and then pass it the bytes to unserialize or the Java object to serialize.
The type detection (based on the first byte of your flow) is done forward with a simple case matching mechanism (1 => ClassA, 15 => ClassF, ...).
EDIT: It may be complex or overloaded with code (embedding objects) but keep in mind that nowadays, java optimize this well, it keeps code clear and understandable.
ByteBuffer can be used for parsing byte stream - What is the use of ByteBuffer in Java?:
byte[] bytesArray = {4, 2, 6, 5, 3, 2, 1};
ByteBuffer bb = ByteBuffer.wrap(bytesArray);
int intFromBB = bb.order(ByteOrder.LITTLE_ENDIAN).getInt();
byte byteFromBB = bb.get();
short shortFromBB = bb.getShort();
as the title already says, I need to convert an int[] to a ByteBuffer in Java. Is there a recommended way to do this ?
I want to pass the ByteBuffer over JNI to C++. What do I have to look out for regarding any specific endian conversions in this case ?
Edit: Sorry, I mistakenly wrote ByteArray but meant the type ByteBuffer.
Edit: Sample code:
I stripped out the unnecessary parts. I call a Java function over JNI from c++ to load a resource and pass it back to c++ as bytebuffer. It works with various other resources. Now I have an "int []" and would like to know if there is an elegant way to convert it to a bytebuffer or if I have to go the oldfashioned way and fill it in a for loop.
ByteBuffer resource= null;
resource = ByteBuffer.allocateDirect((x*y+2)*4).order(ByteOrder.nativeOrder());
.
.
ByteBuffer GetResourcePNG(String text)
{
.
.
int [] pix;
map.getPixels(pix,0,x,0,0,x,y);
return resource;
}
You have to use ByteBuffer.allocateDirect if you want to be able to use JNI's GetDirectBufferAddress.
Use ByteBuffer.order(ByteOrder.nativeOrder()) to adjust the ByteBuffer instance's endianness to match the current platform.
After the ByteBuffer's endianness is properly configured, use ByteBuffer.asIntBuffer() to get a view of it as a java.nio.IntBuffer and fill it with your data.
Full Example:
import java.nio.ByteBuffer; import java.nio.ByteOrder; import java.nio.IntBuffer;
public class Test {
static final int bytes_per_datum = 4;
public static void main(String args[]) {
main2("Native Endian", ByteOrder.nativeOrder());
main2("Big Endian", ByteOrder.BIG_ENDIAN);
main2("Little Endian", ByteOrder.LITTLE_ENDIAN);
}
static void main2(String comment, ByteOrder endian) {
int[] data = { 1, 0xF, 0xFF, 0xFFF, 0xFFFF, 0xFFFFF, 0xFFFFFF, 0xFFFFFFF, 0xFFFFFFFF };
ByteBuffer bb = ByteBuffer.allocateDirect(data.length * bytes_per_datum);
bb.order(endian); // endian must be set before putting ints into the buffer
put_ints(bb, data);
System.out.println(comment + ": ");
print(bb);
}
static void put_ints(ByteBuffer bb, int[] data) {
IntBuffer b = bb.asIntBuffer(); // created IntBuffer starts only from the ByteBuffer's relative position
// if you plan to reuse this IntBuffer, be mindful of its position
b.put(data); // position of this IntBuffer changes by +data.length;
} // this IntBuffer goes out of scope
static void print(ByteBuffer bb) { // prints from start to limit
ByteBuffer bb_2 = bb.duplicate(); // shares backing content, but has its own capacity/limit/position/mark (equivalent to original buffer at initialization)
bb_2.rewind();
for (int x = 0, xx = bb_2.limit(); x < xx; ++x) {
System.out.print((bb_2.get() & 0xFF) + " "); // 0xFF for display, since java bytes are signed
if ((x + 1) % bytes_per_datum == 0) {
System.out.print(System.lineSeparator());
}
}
}
}
you could convert to matrix in this way:
public static final byte[] intToByteArray(int value) {
return new byte[] {
(byte)(value >>> 24),
(byte)(value >>> 16),
(byte)(value >>> 8),
(byte)value};
}
int[] arrayOfInt = {1,2,3,4,5,6};
byte[][] matrix = new byte[arrayOfInt.length][size];
for(int i=0;i<arrayOfInt.length;i++)
byte[i] = intToByteArray(arrayOfInt[i]);
Any reason for not passing the int[] array directly to the C++ or C code using JNI as per the example mentioned here?
Is there an equivalent function to this Python function in Java?
struct.unpack(fmt, string)
I'm trying to port a parser written in Python to Java, and I'm looking for a way to implement the following line of code:
handle, msgVer, source, startTime, dataFormat, sampleCount, sampleInterval, physDim, digMin, digMax, physMin, physMax, freq, = unpack(self.headerFormat,self.unprocessed[pos:pos+calcsize(self.headerFormat)])
I'm using this in the context of a project where I receive bytes from the network and need to extract a specific part of the bytes to display them.
[EDIT 2]
The conclusion I had posted as an update was wrong. I deleted it to avoid misleading others.
I am not aware of any real equivalent to Python's unpack in Java.
The conventional approach would be to read the data from a stream (originating either from a socket, or from a byte array read from the socket, via a ByteArrayInputStream) using a DataInputStream. That class has a set of methods for reading various primitives.
In your case, you would do something like:
DataInputStream in;
char[] handle = new char[6]; in.readFully(handle);
byte messageVersion = in.readByte();
byte source = in.readByte();
int startTime = in.readInt();
byte dataFormat = in.readByte();
byte sampleCount = in.readByte();
int sampleInterval = in.readInt();
short physDim = in.readShort();
int digMin = in.readInt();
int digMax = in.readInt();
float physMin = in.readFloat();
float physMax = in.readFloat();
int freq = in.readInt();
And then turn those variables into a suitable object.
Note that i've opted to pack each field into the smallest primitive which will hold it; that means putting unsigned values into signed types of the same size. You might prefer to put them in bigger types, so that they keep their sign (eg putting an unsigned short into an int); DataInputStream has a set of readUnsignedXXX() methods that you can use for that.
I have a BitSet and want to write it to a file- I came across a solution to use a ObjectOutputStream using the writeObject method.
I looked at the ObjectOutputStream in the java API and saw that you can write other things (byte, int, short etc)
I tried to check out the class so I tried to write a byte to a file using the following code but the result gives me a file with 7 bytes instead of 1 byte
my question is what are the first 6 bytes in the file? why are they there?
my question is relevant to a BitSet because i don't want to start writing lots of data to a file and realize I have random bytes inserted in the file without knowing what they are.
here is the code:
byte[] bt = new byte[]{'A'};
File outFile = new File("testOut.txt");
FileOutputStream fos = new FileOutputStream(outFile);
ObjectOutputStream oos = new ObjectOutputStream(fos);
oos.write(bt);
oos.close();
thanks for any help
Avner
The other bytes will be type information.
Basically ObjectOutputStream is a class used to write Serializable objects to some destination (usually a file). It makes more sense if you think about InputObjectStream. It has a readObject() method on it. How does Java know what Object to instantiate? Easy: there is type information in there.
You could be writing any objects out to an ObjectOutputStream, so the stream holds information about the types written as well as the data needed to reconstitute the object.
If you know that the stream will always contain a BitSet, don't use an ObjectOutputStream - and if space is a premium, then convert the BitSet to a set of bytes where each bit corresponds to a bit in the BitSet, then write that directly to the underlying stream (e.g. a FileOutputStream as in your example).
The serialisation format, like many others, includes a header with magic number and version information. When you use DataOutput/OutputStream methods on ObjectOutputStream are placed in the middle of the serialised data (with no type information). This is typically only done in writeObject implementations after a call to defaultWriteObject or use of putFields.
If you only use the saved BitSet in Java, the serialization works fine. However, it's kind of annoying if you want share the bitset across multi platforms. Besides the overhead of Java serialization, the BitSet is stored in units of 8-bytes. This can generate too much overhead if your bitset is small.
We wrote this small class so we can exract byte arrays from BitSet. Depending on your usecase, it might work better than Java serialization for you.
public class ExportableBitSet extends BitSet {
private static final long serialVersionUID = 1L;
public ExportableBitSet() {
super();
}
public ExportableBitSet(int nbits) {
super(nbits);
}
public ExportableBitSet(byte[] bytes) {
this(bytes == null? 0 : bytes.length*8);
for (int i = 0; i < size(); i++) {
if (isBitOn(i, bytes))
set(i);
}
}
public byte[] toByteArray() {
if (size() == 0)
return new byte[0];
// Find highest bit
int hiBit = -1;
for (int i = 0; i < size(); i++) {
if (get(i))
hiBit = i;
}
int n = (hiBit + 8) / 8;
byte[] bytes = new byte[n];
if (n == 0)
return bytes;
Arrays.fill(bytes, (byte)0);
for (int i=0; i<n*8; i++) {
if (get(i))
setBit(i, bytes);
}
return bytes;
}
protected static int BIT_MASK[] =
{0x80, 0x40, 0x20, 0x10, 0x08, 0x04, 0x02, 0x01};
protected static boolean isBitOn(int bit, byte[] bytes) {
int size = bytes == null ? 0 : bytes.length*8;
if (bit >= size)
return false;
return (bytes[bit/8] & BIT_MASK[bit%8]) != 0;
}
protected static void setBit(int bit, byte[] bytes) {
int size = bytes == null ? 0 : bytes.length*8;
if (bit >= size)
throw new ArrayIndexOutOfBoundsException("Byte array too small");
bytes[bit/8] |= BIT_MASK[bit%8];
}
}