We Keep Coding

How to map binary data in java class object?

Input data :hexadecimal 64 byte
String binaryData="01000076183003104000800180f5010100010100000063000000630000006300000063000000000000000000820000000200b8010307010700640005e1cbe180";
Question is to read this binary data and set in class object
Here is the model
public class Transaction_PLUSale {
public byte opcode;
public byte[] code=new byte[7];
public byte flag1;
public byte flag2;
public byte flag3;
public byte flag4;
public byte flag5;
public short deptnum;
public byte multi_sell_unit;
public byte return_type;
public byte tax_pointer;
public int qty;
public int price;
public int amount;
public int no_tax_price;
public int no_tax_amount;
public int return_surcharge_percent;
public byte product_code;
public byte flags;
public TransactionTail tail;
}
I am currently doing this way to set values in each fields.
String hexArray[]= binaryData.split("(?<=\\G..)");
public static void readPLUSalesData(String hexArray[]) {
Transaction_PLUSale pluSale=new Transaction_PLUSale();
pluSale.setOpcode(Byte.valueOf(hexArray[0]));
byte arr[]=new byte[7];
for(int i=1;i<=7;i++) {
arr[i-1]=Byte.valueOf(hexArray[i]);
}
pluSale.setCode(arr);
pluSale.setFlag1(Byte.valueOf(hexArray[8]));
pluSale.setFlag2(Byte.valueOf(hexArray[9]));
pluSale.setFlag3(Byte.valueOf(hexArray[10]));
pluSale.setFlag4(Byte.valueOf(hexArray[11]));
pluSale.setFlag5(Byte.valueOf(hexArray[12]));
pluSale.setDeptnum((short)Integer.parseInt((hexArray[14]+hexArray[13]),16));
pluSale.setMulti_sell_unit(Byte.valueOf(hexArray[15]));
pluSale.setReturn_type(Byte.valueOf(hexArray[16]));;
pluSale.setTax_pointer(Byte.valueOf(hexArray[17]));
pluSale.setQty(Integer.parseInt((hexArray[21]+hexArray[20]+hexArray[19]+hexArray[18]),16));
pluSale.setPrice(Integer.parseInt((hexArray[25]+hexArray[24]+hexArray[23]+hexArray[22]),16));
pluSale.setAmount(Integer.parseInt((hexArray[29]+hexArray[28]+hexArray[27]+hexArray[26]),16));
pluSale.setNo_tax_price(Integer.parseInt((hexArray[33]+hexArray[32]+hexArray[31]+hexArray[30]),16));
pluSale.setNo_tax_amount(Integer.parseInt((hexArray[37]+hexArray[36]+hexArray[35]+hexArray[34]),16));
pluSale.setReturn_surcharge_percent(Integer.parseInt((hexArray[41]+hexArray[40]+hexArray[39]+hexArray[38]),16));
pluSale.setProduct_code(Byte.valueOf(hexArray[42]));
pluSale.setFlags(Byte.valueOf(hexArray[43]));
}
It is working fine. But I want it to be generic. So instead of giving byte by byte value. I want to direct map it to class fields.
In .net we are doing the marshalling for same feature that I need.
Here is the example
foreach (KeyValuePair<string, byte[]> s in t)
{
//byte array consist of bytes of the above hexadecimal string.
Ticket ticket = new Ticket();
int count = Marshal.SizeOf(typeof(Transaction_Coupon));
MemoryStream ms = new MemoryStream(s.Value);
byte[] readBuffer = new byte[count];
BinaryReader br = new BinaryReader(ms);
readBuffer = br.ReadBytes(count);
GCHandle handle = GCHandle.Alloc(readBuffer, GCHandleType.Pinned);
//here we are mapping byte data to each field
Transaction_PLUSale t_plusale = (Transaction_PLUSale)Marshal.PtrToStructure(handle.AddrOfPinnedObject(), typeof(Transaction_PLUSale));
}

To convert binary data, a byte[] to a class with fields, there is no memory template to shift the data in. A good solution is using a ByteBuffer, either on a byte array or InputStream.
public static void readPLUSalesData(String[] hexArray) {
byte[] bytes = new byte[hexArray.length];
for (int i = 0; i < bytes.length; ++i) {
bytes[i] = Byte.parseByte(hexArray[i], 16);
}
ByteBuffer buf = ByteBuffer.wrap(bytes).order(ByteOrder.LITTLE_ENDIAN;
Transaction_PLUSale pluSale=new Transaction_PLUSale();
pluSale.setOpcode(buf.get());
byte[] arr[] = new byte[7];
buf.get(arr);
pluSale.setCode(arr);
pluSale.setFlag1(buf.get());
pluSale.setFlag2(buf.get());
pluSale.setFlag3(buf.get());
pluSale.setFlag4(buf.get());
pluSale.setFlag5(buf.get());
pluSale.setDeptnum(buf.getShort());
pluSale.setMulti_sell_unit(buf.get());
pluSale.setReturn_type(buf.get());
pluSale.setTax_pointer(buf.get());
pluSale.setQty(buf.getInt());
pluSale.setPrice(buf.getInt());
pluSale.setAmount(buf.getInt());
pluSale.setNo_tax_price(buf.getInt());
pluSale.setNo_tax_amount(buf.getInt());
pluSale.setReturn_surcharge_percent(buf.getInt());
pluSale.setProduct_code(buf.get());
pluSale.setFlags(buf.get());
}
There exist other solutions, like using reflection, which is inefficient.
I used little endian byte order here, default in java is big endian.
There is the ObjectOutputStream, Serializable, persistence using serialisation.
It stores class data too, so is not the language agnostic format you desire.
While developing with a ByteBuffer is makes sense to check the read position.
If you are interested in XML persistence, JAXB with annotations offers a nice reflection based way, without need of handling every field.
A remark: Type[] variable is the preferred notation; Type var[] was initially added to java to be compatible with C/C++.

Java How to convert a java.nio.Buffer into a ByteBuffer

I'm facing a little problem I have two libraries one send me the output as java.nio.Buffer and the other receives the input as a java.nio.ByteBuffer how do I make the conversion?
Thanks
the Buffer is from javaCV from this piece of code:
private BytePointer[] image_ptr;
private Buffer[] image_buf;
// Determine required buffer size and allocate buffer
int size = avpicture_get_size(fmt, width, height);
image_ptr = new BytePointer[] { new BytePointer(av_malloc(size)).capacity(size) };
image_buf = new Buffer[] { image_ptr[0].asBuffer() };
// Assign appropriate parts of buffer to image planes in picture_rgb
// Note that picture_rgb is an AVFrame, but AVFrame is a superset of AVPicture
avpicture_fill(new AVPicture(picture_rgb), image_ptr[0], fmt, width, height);
picture_rgb.format(fmt);
picture_rgb.width(width);
picture_rgb.height(height);

First of all, since Buffer is an abstract class, and ByteBuffer is one of its subclasses, it's entirely possible that the output you're getting from the first library is in fact a ByteBuffer. If possible, check to see which implementation of Buffer the library is returning, because if it's actually returning a ByteBuffer you can just cast the output to ByteBuffer and be done.
If you don't know which implementation of Buffer the library returns, you'll have to resort to instanceof tests to determine what subclass it is, and copy the data from the returned Buffer to a new ByteBuffer after downcasting it to a subclass. This is because the Buffer interface doesn't actually provide any methods to read the data from the buffer; only the subclasses (ByteBuffer, ShortBuffer, LongBuffer, etc.) do. Fortunately, there are only 7 possible subclasses of Buffer, one for each primitive type.
Once you've determined which subclass of Buffer you have, you can copy the data to a ByteBuffer using the "asXXXBuffer()" method described in this answer, as #Tunaki pointed out.
The code would look something like this:
Buffer outputBuffer = library.getBuffer();
ByteBuffer byteBuffer;
if (outputBuffer instanceof ByteBuffer) {
byteBuffer = (ByteBuffer) outputBuffer;
} else if (outputBuffer instanceof CharBuffer) {
byteBuffer = ByteBuffer.allocate(outputBuffer.capacity());
byteBuffer.asCharBuffer().put((CharBuffer) outputBuffer);
} else if (outputBuffer instanceof ShortBuffer) {
byteBuffer = ByteBuffer.allocate(outputBuffer.capacity() * 2);
byteBuffer.asShortBuffer().put((ShortBuffer) outputBuffer);
} else if (outputBuffer instanceof IntBuffer) {
byteBuffer = ByteBuffer.allocate(outputBuffer.capacity() * 4);
byteBuffer.asIntBuffer().put((IntBuffer) outputBuffer);
} else if (outputBuffer instanceof LongBuffer) {
byteBuffer = ByteBuffer.allocate(outputBuffer.capacity() * 8);
byteBuffer.asLongBuffer().put((LongBuffer) outputBuffer);
} else if (outputBuffer instanceof FloatBuffer) {
byteBuffer = ByteBuffer.allocate(outputBuffer.capacity() * 4);
byteBuffer.asFloatBuffer().put((FloatBuffer) outputBuffer);
} else if (outputBuffer instanceof DoubleBuffer) {
byteBuffer = ByteBuffer.allocate(outputBuffer.capacity() * 8);
byteBuffer.asDoubleBuffer().put((DoubleBuffer) outputBuffer);
}
Note that the size of the ByteBuffer you allocate depends on which subclass of Buffer you're copying from, since different primitive types are stored using different numbers of bytes. For example, since an int is 4 bytes, if your library gives you an IntBuffer, you need to allocate a ByteBuffer with 4 times the capacity.

Error while passing array of structures conataining char array to native function using JNA

Here is my updated code after implementing suggestions. But still problems persist.
typedef struct S1{
char temp1[100];
char temp2[100];
}S1
...
int manipulateTemp(S1 s1Arr[] );
JNA interface looks like this
public interface Add extends Library
{
Add INSTANCE = (Add) Native.loadLibrary("add", Add.class);
public static class S1 extends Structure {
public byte[] temp1 = new byte[100];
public byte[] temp2 = new byte[100];
public static class ByReference extends S1 implements Structure.ByReference {
};
};
int manipulateTemp( S1[]);
}
//
public static byte[] toByteArray(char[] a ,Charset c){
CharBuffer cBuffer = CharBuffer.wrap(a);
ByteBuffer bBuffer = c.encode(cBuffer);
return bBuffer.array;
}
//in main method
Add lib = Add.INSTANCE;
Add.S1.ByReference s1Ref = new Add.S1.ByReference();
Add.S1[] s1Arr = (Add.S1[])s1Ref.toArray(10);
s1Ref.clear();
//initialize array
for(int i =0;i<s1Arr.lenth ;i++){
byte[] data = toByteArray("myString1".toCharArray,Charset.defaultCharSet
System.arrarycopy(data,0, s1Arr[i].temp1,0,data.length);
data = toByteArray("myString2".toCharArray,Charset.defaultCharSet
System.arrarycopy(data,0, s1Arr[i].temp2,0,data.length);
}
// calling native function
lib.manipulateTemp(s1Arr[]);
After execution
Exception in thread "main" java.lang.Error: Invalid memory access
at com.sun.jna.Function.invokeInt(Native Method)
at com.sun.jna.Function.invoke(Function.java:344)
at com.sun.jna.Function.invoke(Function.java:276)
at com.sun.jna.Library$Handler.invoke(Library.java:216)
at com.sun.proxy.$Proxy0.manipulateTemp((Unknown Source)
at LoanTest.newTestCalc.main(newTestCalc.java:288)
I even checked memory dump, structures are seems to be allocated stored correctly.Structure size is also correct = 200 bytes
Any clues about this error?

You need to copy values into the existing temp field, not overwrite it. When you overwrite it, you're actually changing its size, which JNA uses to determine the structure size. Following is how you should initialize your structure data:
class S1 extends Structure {
public byte[] temp = new byte[100];
...
}
S1 s = new S1();
S1[] array = (S1[])s.toArray(ARRAY_SIZE);
System.setProperty("jna.encoding", "utf-8"); // You probably want utf-8; utf-16 has 16-bit code units, so unless your native code is actually expecting a utf-16 encoding broken down into byte units, use utf-8
byte[] data = Native.toByteArray("myString"); // includes NUL terminator
System.arraycopy(data, 0, array[0].temp, 0, data.length);
// Repeat as needed for other members of the array
lib.manipulateTemp(array);
Note that the declarations manipulateTemp(S1 s) or manipulateTemp(S1[] array) will both work, although the latter is more accurate and conveys your intent explicitly.

Java Automatic Field Value Serialization

Lets say we have a struct-llike java class
public class Person {
private int height;
private byte nChildren;
private int salary;
public byte[] serializeField() {
ByteBuffer buf = ByteBuffer.allocate(4 + 1 + 4);
buf.order(ByteOrder.BIG_ENDIAN);
buf.putInt(height);
buf.put(nChildren);
buf.putInt(salary);
return buf.array();
}
/*
* setters and getters
*/
}
Is there a library that can perform the serializeField() function automatically for any given class? It should be able to maintain the exact order of the fields as defined in the class and perhaps have the ability to ignore certain fields (like the serialVersionUID).

You can write one using reflection.
If performance matters, you can
pass in a ByteBuffer to append to to avoid creating lots of buffers you discard.
public void serializeField(ByteBuffer buf) {
buf.putInt(height);
buf.put(nChildren);
buf.putInt(salary);
}
and generate your code (using reflection etc) to avoid using reflection at runtime. i.e. use reflection to generate the code above or in the next example.
map the fields getters and setters into a ByteBuffer so there is no serialization/deserialization as such.
Say you have an object with a width and height. As you can see the values are updated into a ByteBuffer (e.g. a direct or memory mapped file) This way there is no serialization overhead as such.
class Sized {
private ByteBuffer buffer;
private static final int HEIGHT_OFFSET = 0;
private static final int WIDTH_OFFSET = 4;
public void setBuffer(ByteBuffer buffer) {this.buffer = buffer; }
public int getHeight() { return buffer.getInt(HEIGHT_OFFSET); }
public void setHeight(int height) { buffer.putInt(HEIGHT_OFFSET, height); }
public int getWidth() { return buffer.getInt(WIDTH_OFFSET); }
public void setWidth(int width) { buffer.putInt(WIDTH_OFFSET, width); }
}
To handle the memory mapped data you can use a library like this.
https://github.com/peter-lawrey/Java-Chronicle
This library allow you to create any number of events/records (billions) which can be updated (but not grown in size once written) very quickly. e.g. millions of records per second with sub-microsecond latency. It can also be shared between processes. Once you exceed your main memory, the main limitation is the speed of your hard drive if you exceed your memory usage. i.e. an SSD will really help. It has a mode where you use the Unsafe class directly (more low level than ByteBuffer) by just enabling it.

Deep copy duplicate() of Java's ByteBuffer

java.nio.ByteBuffer#duplicate() returns a new byte buffer that shares the old buffer's content. Changes to the old buffer's content will be visible in the new buffer, and vice versa. What if I want a deep copy of the byte buffer?

I think the deep copy need not involve byte[]. Try the following:
public static ByteBuffer clone(ByteBuffer original) {
ByteBuffer clone = ByteBuffer.allocate(original.capacity());
original.rewind();//copy from the beginning
clone.put(original);
original.rewind();
clone.flip();
return clone;
}

As this question still comes up as one of the first hits to copying a ByteBuffer, I will offer my solution. This solution does not touch the original buffer, including any mark set, and will return a deep copy with the same capacity as the original.
public static ByteBuffer cloneByteBuffer(final ByteBuffer original) {
// Create clone with same capacity as original.
final ByteBuffer clone = (original.isDirect()) ?
ByteBuffer.allocateDirect(original.capacity()) :
ByteBuffer.allocate(original.capacity());
// Create a read-only copy of the original.
// This allows reading from the original without modifying it.
final ByteBuffer readOnlyCopy = original.asReadOnlyBuffer();
// Flip and read from the original.
readOnlyCopy.flip();
clone.put(readOnlyCopy);
return clone;
}
If one cares for the position, limit, or order to be set the same as the original, then that's an easy addition to the above:
clone.position(original.position());
clone.limit(original.limit());
clone.order(original.order());
return clone;

Based off of mingfai's solution:
This will give you an almost true deep copy. The only thing lost will be the mark. If orig is a HeapBuffer and the offset is not zero or the capacity is less than the backing array than the outlying data is not copied.
public static ByteBuffer deepCopy( ByteBuffer orig )
{
int pos = orig.position(), lim = orig.limit();
try
{
orig.position(0).limit(orig.capacity()); // set range to entire buffer
ByteBuffer toReturn = deepCopyVisible(orig); // deep copy range
toReturn.position(pos).limit(lim); // set range to original
return toReturn;
}
finally // do in finally in case something goes wrong we don't bork the orig
{
orig.position(pos).limit(lim); // restore original
}
}
public static ByteBuffer deepCopyVisible( ByteBuffer orig )
{
int pos = orig.position();
try
{
ByteBuffer toReturn;
// try to maintain implementation to keep performance
if( orig.isDirect() )
toReturn = ByteBuffer.allocateDirect(orig.remaining());
else
toReturn = ByteBuffer.allocate(orig.remaining());
toReturn.put(orig);
toReturn.order(orig.order());
return (ByteBuffer) toReturn.position(0);
}
finally
{
orig.position(pos);
}
}

One more simple solution
public ByteBuffer deepCopy(ByteBuffer source, ByteBuffer target) {
int sourceP = source.position();
int sourceL = source.limit();
if (null == target) {
target = ByteBuffer.allocate(source.remaining());
}
target.put(source);
target.flip();
source.position(sourceP);
source.limit(sourceL);
return target;
}

You'll need to iterate the entire buffer and copy by value into the new buffer.

I believe this should supply a full deep copy, including the mark, "out-of-bounds" data, etc...just in case you need the most complete sandbox-safe carbon copy of a ByteBuffer.
The only thing it doesn't copy is the read-only trait, which you can easily get by just calling this method and tagging on a ".asReadOnlyBuffer()"
public static ByteBuffer cloneByteBuffer(ByteBuffer original)
{
//Get position, limit, and mark
int pos = original.position();
int limit = original.limit();
int mark = -1;
try
{
original.reset();
mark = original.position();
}
catch (InvalidMarkException e)
{
//This happens when the original's mark is -1, so leave mark at default value of -1
}
//Create clone with matching capacity and byte order
ByteBuffer clone = (original.isDirect()) ? ByteBuffer.allocateDirect(original.capacity()) : ByteBuffer.allocate(original.capacity());
clone.order(original.order());
//Copy FULL buffer contents, including the "out-of-bounds" part
original.limit(original.capacity());
original.position(0);
clone.put(original);
//Set mark of both buffers to what it was originally
if (mark != -1)
{
original.position(mark);
original.mark();
clone.position(mark);
clone.mark();
}
//Set position and limit of both buffers to what they were originally
original.position(pos);
original.limit(limit);
clone.position(pos);
clone.limit(limit);
return clone;
}