We Keep Coding

How to map binary data in java class object?

Input data :hexadecimal 64 byte
String binaryData="01000076183003104000800180f5010100010100000063000000630000006300000063000000000000000000820000000200b8010307010700640005e1cbe180";
Question is to read this binary data and set in class object
Here is the model
public class Transaction_PLUSale {
public byte opcode;
public byte[] code=new byte[7];
public byte flag1;
public byte flag2;
public byte flag3;
public byte flag4;
public byte flag5;
public short deptnum;
public byte multi_sell_unit;
public byte return_type;
public byte tax_pointer;
public int qty;
public int price;
public int amount;
public int no_tax_price;
public int no_tax_amount;
public int return_surcharge_percent;
public byte product_code;
public byte flags;
public TransactionTail tail;
}
I am currently doing this way to set values in each fields.
String hexArray[]= binaryData.split("(?<=\\G..)");
public static void readPLUSalesData(String hexArray[]) {
Transaction_PLUSale pluSale=new Transaction_PLUSale();
pluSale.setOpcode(Byte.valueOf(hexArray[0]));
byte arr[]=new byte[7];
for(int i=1;i<=7;i++) {
arr[i-1]=Byte.valueOf(hexArray[i]);
}
pluSale.setCode(arr);
pluSale.setFlag1(Byte.valueOf(hexArray[8]));
pluSale.setFlag2(Byte.valueOf(hexArray[9]));
pluSale.setFlag3(Byte.valueOf(hexArray[10]));
pluSale.setFlag4(Byte.valueOf(hexArray[11]));
pluSale.setFlag5(Byte.valueOf(hexArray[12]));
pluSale.setDeptnum((short)Integer.parseInt((hexArray[14]+hexArray[13]),16));
pluSale.setMulti_sell_unit(Byte.valueOf(hexArray[15]));
pluSale.setReturn_type(Byte.valueOf(hexArray[16]));;
pluSale.setTax_pointer(Byte.valueOf(hexArray[17]));
pluSale.setQty(Integer.parseInt((hexArray[21]+hexArray[20]+hexArray[19]+hexArray[18]),16));
pluSale.setPrice(Integer.parseInt((hexArray[25]+hexArray[24]+hexArray[23]+hexArray[22]),16));
pluSale.setAmount(Integer.parseInt((hexArray[29]+hexArray[28]+hexArray[27]+hexArray[26]),16));
pluSale.setNo_tax_price(Integer.parseInt((hexArray[33]+hexArray[32]+hexArray[31]+hexArray[30]),16));
pluSale.setNo_tax_amount(Integer.parseInt((hexArray[37]+hexArray[36]+hexArray[35]+hexArray[34]),16));
pluSale.setReturn_surcharge_percent(Integer.parseInt((hexArray[41]+hexArray[40]+hexArray[39]+hexArray[38]),16));
pluSale.setProduct_code(Byte.valueOf(hexArray[42]));
pluSale.setFlags(Byte.valueOf(hexArray[43]));
}
It is working fine. But I want it to be generic. So instead of giving byte by byte value. I want to direct map it to class fields.
In .net we are doing the marshalling for same feature that I need.
Here is the example
foreach (KeyValuePair<string, byte[]> s in t)
{
//byte array consist of bytes of the above hexadecimal string.
Ticket ticket = new Ticket();
int count = Marshal.SizeOf(typeof(Transaction_Coupon));
MemoryStream ms = new MemoryStream(s.Value);
byte[] readBuffer = new byte[count];
BinaryReader br = new BinaryReader(ms);
readBuffer = br.ReadBytes(count);
GCHandle handle = GCHandle.Alloc(readBuffer, GCHandleType.Pinned);
//here we are mapping byte data to each field
Transaction_PLUSale t_plusale = (Transaction_PLUSale)Marshal.PtrToStructure(handle.AddrOfPinnedObject(), typeof(Transaction_PLUSale));
}

To convert binary data, a byte[] to a class with fields, there is no memory template to shift the data in. A good solution is using a ByteBuffer, either on a byte array or InputStream.
public static void readPLUSalesData(String[] hexArray) {
byte[] bytes = new byte[hexArray.length];
for (int i = 0; i < bytes.length; ++i) {
bytes[i] = Byte.parseByte(hexArray[i], 16);
}
ByteBuffer buf = ByteBuffer.wrap(bytes).order(ByteOrder.LITTLE_ENDIAN;
Transaction_PLUSale pluSale=new Transaction_PLUSale();
pluSale.setOpcode(buf.get());
byte[] arr[] = new byte[7];
buf.get(arr);
pluSale.setCode(arr);
pluSale.setFlag1(buf.get());
pluSale.setFlag2(buf.get());
pluSale.setFlag3(buf.get());
pluSale.setFlag4(buf.get());
pluSale.setFlag5(buf.get());
pluSale.setDeptnum(buf.getShort());
pluSale.setMulti_sell_unit(buf.get());
pluSale.setReturn_type(buf.get());
pluSale.setTax_pointer(buf.get());
pluSale.setQty(buf.getInt());
pluSale.setPrice(buf.getInt());
pluSale.setAmount(buf.getInt());
pluSale.setNo_tax_price(buf.getInt());
pluSale.setNo_tax_amount(buf.getInt());
pluSale.setReturn_surcharge_percent(buf.getInt());
pluSale.setProduct_code(buf.get());
pluSale.setFlags(buf.get());
}
There exist other solutions, like using reflection, which is inefficient.
I used little endian byte order here, default in java is big endian.
There is the ObjectOutputStream, Serializable, persistence using serialisation.
It stores class data too, so is not the language agnostic format you desire.
While developing with a ByteBuffer is makes sense to check the read position.
If you are interested in XML persistence, JAXB with annotations offers a nice reflection based way, without need of handling every field.
A remark: Type[] variable is the preferred notation; Type var[] was initially added to java to be compatible with C/C++.

Library for serializing java objects to fixed-width byte arrays

I would like to store a very simple pojo object in binary format:
public class SampleDataClass {
private long field1;
private long field2;
private long field3;
}
To do this, I have written a simple serialize/deserialize pair of methods:
public class SampleDataClass {
// ... Fields as above
public static void deserialize(ByteBuffer buffer, SampleDataClass into) {
into.field1 = buffer.getLong();
into.field2 = buffer.getLong();
into.field3 = buffer.getLong();
}
public static void serialize(ByteBuffer buffer, SampleDataClass from) {
buffer.putLong(from.field1);
buffer.putLong(from.field2);
buffer.putLong(from.field3);
}
}
Simple and efficient, and most importantly the size of the objects in binary format is fixed. I know the size of each record serialized will be 3 x long, i.e. 3 x 8bytes = 24 bytes.
This is crucial, as I will be recording these sequentially and I need to be able to find them by index later on, i.e. "Find me the 127th record".
This is working fine for me, but I hate the boilerplate - and the fact that at some point I'm going to make a mistake and end up write a load of data that can't be read-back because there's an inconsistency between my serialize / deserialize method.
Is there a library that generate something like this for me?
Ideally I'm looking for something like protobuf, with a fixed-length encoding scheme. Later-on, I'd like to encode strings too. These will also have a fixed length. If a string exceeds the length it's truncated to n bytes. If a string is too short, I'll null-terminate it (or similar).
Finally, protobuf supports different versions of the protocol. It is inevitable I'll need to do that eventually.
I was hoping someone had a suggestion, before I start rolling-my-own

Make your class inherit the java.io.Serializable interface. Then you can use java.io.ObjectOutputStream and java.io.ObjectInputStream to serialize / deserialize objects to / from streams. The write and read methods take byte arrays as arguments.
To make it fixed length, standardize the size of the byte[] arrays used.

The most difficult part here is capping your strings or collections. You can do this with Kryo for Strings by overriding default serializers. Placing strings into a custom buffer class (i.e. FixedSerializableBuffer) which stores or is annotated with a length to cut also makes sense.
public class KryoDemo {
static class Foo{
String s;
long v;
Foo() {
}
Foo(String s, long v) {
this.s = s;
this.v = v;
}
#Override
public String toString() {
final StringBuilder sb = new StringBuilder("Foo{");
sb.append("s='").append(s).append('\'');
sb.append(", v=").append(v);
sb.append('}');
return sb.toString();
}
}
public static void main(String[] args) {
Kryo kryo = new Kryo();
Foo foo = new Foo("test string", 1);
kryo.register(String.class, new Serializer<String>() {
{
setImmutable(true);
setAcceptsNull(true);
}
public void write(Kryo kryo, Output output, String s) {
if (s.length() > 4) {
s = s.substring(0, 4);
}
output.writeString(s);
}
public String read(Kryo kryo, Input input, Class<String> type) {
return input.readString();
}
});
// serialization part, data is binary inside this output
ByteBufferOutput output = new ByteBufferOutput(100);
kryo.writeObject(output, foo);
System.out.println("before: " + foo);
System.out.println("after: " + kryo.readObject(new Input(output.toBytes()), Foo.class));
}
}
This prints:
before: Foo{s='test string', v=1}
after: Foo{s='test', v=1}

If the only additional requirement over standard serialization is efficient random access to the n-th entry, there are alternatives to fixed-size entries, and that you will be storing variable length entries (such as strings) makes me think that these alternatives deserve consideration.
One such alternative is to have a "directory" with fixed length entries, each of which points to the variable length content. Random access to an entry is then implemented by reading the corresponding pointer from the directory (which can be done with random access, as the directory entries are fixed size), and then reading the block it points to. This approach has the disadvantage that an additional I/O access is required to access the data, but permits a more compact representation of the data, as you don't have to pad variable length content, which in turn speeds up sequential reading. Of course, neither the problem nor the above solution is novel - file systems have been around for a long time ...

Error while passing array of structures conataining char array to native function using JNA

Here is my updated code after implementing suggestions. But still problems persist.
typedef struct S1{
char temp1[100];
char temp2[100];
}S1
...
int manipulateTemp(S1 s1Arr[] );
JNA interface looks like this
public interface Add extends Library
{
Add INSTANCE = (Add) Native.loadLibrary("add", Add.class);
public static class S1 extends Structure {
public byte[] temp1 = new byte[100];
public byte[] temp2 = new byte[100];
public static class ByReference extends S1 implements Structure.ByReference {
};
};
int manipulateTemp( S1[]);
}
//
public static byte[] toByteArray(char[] a ,Charset c){
CharBuffer cBuffer = CharBuffer.wrap(a);
ByteBuffer bBuffer = c.encode(cBuffer);
return bBuffer.array;
}
//in main method
Add lib = Add.INSTANCE;
Add.S1.ByReference s1Ref = new Add.S1.ByReference();
Add.S1[] s1Arr = (Add.S1[])s1Ref.toArray(10);
s1Ref.clear();
//initialize array
for(int i =0;i<s1Arr.lenth ;i++){
byte[] data = toByteArray("myString1".toCharArray,Charset.defaultCharSet
System.arrarycopy(data,0, s1Arr[i].temp1,0,data.length);
data = toByteArray("myString2".toCharArray,Charset.defaultCharSet
System.arrarycopy(data,0, s1Arr[i].temp2,0,data.length);
}
// calling native function
lib.manipulateTemp(s1Arr[]);
After execution
Exception in thread "main" java.lang.Error: Invalid memory access
at com.sun.jna.Function.invokeInt(Native Method)
at com.sun.jna.Function.invoke(Function.java:344)
at com.sun.jna.Function.invoke(Function.java:276)
at com.sun.jna.Library$Handler.invoke(Library.java:216)
at com.sun.proxy.$Proxy0.manipulateTemp((Unknown Source)
at LoanTest.newTestCalc.main(newTestCalc.java:288)
I even checked memory dump, structures are seems to be allocated stored correctly.Structure size is also correct = 200 bytes
Any clues about this error?

You need to copy values into the existing temp field, not overwrite it. When you overwrite it, you're actually changing its size, which JNA uses to determine the structure size. Following is how you should initialize your structure data:
class S1 extends Structure {
public byte[] temp = new byte[100];
...
}
S1 s = new S1();
S1[] array = (S1[])s.toArray(ARRAY_SIZE);
System.setProperty("jna.encoding", "utf-8"); // You probably want utf-8; utf-16 has 16-bit code units, so unless your native code is actually expecting a utf-16 encoding broken down into byte units, use utf-8
byte[] data = Native.toByteArray("myString"); // includes NUL terminator
System.arraycopy(data, 0, array[0].temp, 0, data.length);
// Repeat as needed for other members of the array
lib.manipulateTemp(array);
Note that the declarations manipulateTemp(S1 s) or manipulateTemp(S1[] array) will both work, although the latter is more accurate and conveys your intent explicitly.

Java how common is extending/wrapping built-in classes

I'm new to the Java language and I've tried to write my first relatively complex program. After I wrote a few classes I've realized that I barely use built-in classes (like BigInteger, MessageDigest, ByteBuffer) directly because they don't totally fit my needs. Instead I write my own class and inside the class I use the built-in class as an attribute.
Example:
public class SHA1 {
public static final int SHA_DIGEST_LENGTH = 20;
private MessageDigest md;
public SHA1() {
try {
md = MessageDigest.getInstance("SHA-1");
} catch (NoSuchAlgorithmException e) {
e.printStackTrace();
}
}
public void update(byte[] data) {
md.update(data);
}
public void update(BigNumber bn) {
md.update(bn.asByteArray());
}
public void update(String data) {
md.update(data.getBytes());
}
public byte[] digest() {
return md.digest();
}
}
With the following simple class I don't have to use try catch when using SHA1, I can put my custom BigNumber class as parameter and I can also put String as parameter to update function.
The following BigNumber class contains all of the functions what I need and exactly how I need them.
public class BigNumber {
private BigInteger m_bn;
public BigNumber() {
m_bn = new BigInteger("0");
}
public BigNumber(BigInteger bn) {
m_bn = bn;
}
public BigNumber(String hex) {
setHexStr(hex);
}
//reversed no minsize
public byte[] asByteArray() {
return asByteArray(0, true);
}
//reversed with minsize
public byte[] asByteArray(int minSize) {
return asByteArray(minSize, true);
}
public byte[] asByteArray(int minSize, boolean rev) {
byte[] mag = m_bn.toByteArray();
//delete sign bit
//there is always a sign bit! so if bitNum % 8 is zero then
//the sign bit created a new byte (0th)
if(getNumBits() % 8 == 0) {
byte[] tmp = new byte[mag.length-1];
System.arraycopy(mag, 1, tmp, 0, mag.length-1);
mag = tmp;
}
//extend the byte array if needed
int byteSize = (minSize >= getNumBytes()) ? minSize : getNumBytes();
byte[] tmp = new byte[byteSize];
//if tmp's length smaller then byteSize then we keep 0x00-s from left
System.arraycopy(mag, 0, tmp, byteSize-mag.length, mag.length);
if(rev) ByteManip.reverse(tmp);
return tmp;
}
public String asHexStr() {
return ByteManip.byteArrayToHexStr(asByteArray(0, false));
}
public void setHexStr(String hex) {
m_bn = new BigInteger(hex, 16);
}
public void setBinary(byte[] data) {
//reverse = true
ByteManip.reverse(data);
//set as hex (binary set has some bug with the sign bit...)
m_bn = new BigInteger(ByteManip.byteArrayToHexStr(data), 16);
}
public void setRand(int byteSize) {
byte[] tmp = new byte[byteSize];
new Random().nextBytes(tmp);
//reversing byte order, but it doesn't really matter since it is a random
//number
setBinary(tmp);
}
public int getNumBytes() {
return (m_bn.bitLength() % 8 == 0) ? (m_bn.bitLength() / 8) : (m_bn.bitLength() / 8 + 1);
}
public int getNumBits() {
return m_bn.bitLength();
}
public boolean isZero() {
return m_bn.equals(BigInteger.ZERO);
}
//operations
public BigNumber modExp(BigNumber exp, BigNumber mod) {
return new BigNumber(m_bn.modPow(exp.m_bn, mod.m_bn));
}
public BigNumber mod(BigNumber m) {
return new BigNumber(m_bn.mod(m.m_bn));
}
public BigNumber add(BigNumber bn) {
return new BigNumber(m_bn.add(bn.m_bn));
}
public BigNumber subtract(BigNumber bn) {
return new BigNumber(m_bn.subtract(bn.m_bn));
}
public BigNumber multiply(BigNumber bn) {
return new BigNumber(m_bn.multiply(bn.m_bn));
}
}
My question is that how common in Java language to use these kind of classes instead of the built-in classes? Does it make my code unreadable for other programmers (compared to implementing everything with built-in classes)?
I've read that new C++ programmers desperately trying to write codes they used to write in C therefore the benefits of C++ remains hidden for them.
I'm afraid I do something like that in Java: trying to implement everything on my own instead of using the build-in classes directly.
Is this happening (for example in the BigNumber class)?
Thank you for your opinions!

I normally write a utility class which will support me to handle logics. Such as
public class CommonUtil{
public byte[] asByteArray(int minSize)
{
return "something".getBytes();
}
// add more utility methods
}

Wrapping a class makes sense when you add some value by doing so. If you are adding small functionality it can be worth using a Utility class instead of wrapping an existing one.

I think that if you do not have a very good reason for implementing the same functionality again you should not probably do it. Here are several reasons:
Built-in classes are used by a lot of people around the world and therefore there are less bugs than in your code
Users that are experienced in Java will be better in using standard classes than your classes and they will need less time to understand your code and write something new in your project
Built-in classes have good documentations and therefore it is much easier to use them
You are wasting your time by implementing something that was implemented and tested by Java professionals. It is better to concentrate on your own project
If you are writing a long-term project you will need to support all your classes. Oracle is already supporting built-in classes. Let them do their job!
The last but not the least. Are you sure that you know more about the problem than an author of a built-in class? Only if the answer is yes, consider writing your own implementation. Even implementation of daily used classes, such as collections or time-related classes can be tricky.

You're not gaining anything by making a class that does this stuff for you. If you're going to be doing certain operations a lot, then you might want to create a new class with static methods that do these important things for you.
Let's assume that you want a sorted array at all times. You could make a new class, let's call it SortedArray. You could sort it whenever you add something in, but why would you do that when you can just add in everything and then call the (utility) method Arrays.sort?
For common operations, take a look at Java's Arrays class - if you are doing something often, that's something you make a method for, like searching and sorting. In your case, you might make a utility method that turns the BigInteger into a byte array for you. You shouldn't be just making your own, 'better' version that does what you want it. When other people look at your code, when you use standard objects it's much better, instead of having custom objects that don't really do anything.

As #Shark commented, there's no point in creating your own solutions, because:
They take time to create
They become not as flexible
However, you can extend classes (it's recommended) or use 3rd party frameworks that might suit you better.

Are there any tricks to reduce memory usage when storing String data type in hashmap?

I need to store value pair (word and number) in the Map.
I am trying to use TObjectIntHashMap from Trove library with char[] as the key, because I need to minimize the memory usage. But with this method, I can not get the value when I use get() method.
I guess I can not use primitive char array to store in a Map because hashcode issues.
I tried to use TCharArrayList but that takes much memory also.
I read in another stackoverflow question that similar with my purpose and have suggestion to use TLongIntHashMap , store encode values of String word in long data type. In this case my words may contains of latin characters or various other characters that appears in wikipedia collections, I do not know whether the Long is enough for encode or not.
I have tried using Trie data structure to store it, but I need to consider my performance also and choose the best for both memory usage and performance.
Do you have any idea or suggestion for this issue?

It sounds like the most compact way to store the data is to use a byte[] encoded in UTF-8 or similar. You can wrap this in your own class or write you own HashMap which allows byte[] as a key.
I would reconsider how much time it is worth spending to save some memory. If you are talking about a PC or Server, at minimum wage you need to save 1 GB for an hours work so if you are only looking to save 100 MB that's about 6 minutes including testing.

Write your own class that implements CharSequence, and write your own implementation of equals() and hashcode(). The implementation would also pre-allocate large shared char[] storage, and use bits of it at a time. (You can definitely incorporate #Peter Lawrey's excellent suggestion into this, too, and use byte[] storage.)
There's also an opportunity to do a 'soft intern()' using an LRU cache. I've noted where the cache would go.
Here's a simple demonstration of what I mean. Note that if you need heavily concurrent writes, you can try to improve the locking scheme below...
public final class CompactString implements CharSequence {
private final char[] _data;
private final int _offset;
private final int _length;
private final int _hashCode;
private static final Object _lock = new Object();
private static char[] _storage;
private static int _nextIndex;
private static final int LENGTH_THRESHOLD = 128;
private CompactString(char[] data, int offset, int length, int hashCode) {
_data = data; _offset = offset; _length = length; _hashCode = hashCode;
}
private static final CompactString EMPTY = new CompactString(new char[0], 0, 0, "".hashCode());
private static allocateStorage() {
synchronized (_lock) {
_storage = new char[1024];
_nextIndex = 0;
}
}
private static CompactString storeInShared(String value) {
synchronized (_lock) {
if (_nextIndex + value.length() > _storage.length) {
allocateStorage();
}
int start = _nextIndex;
// You would need to change this loop and length to do UTF encoding.
for (int i = 0; i < value.length(); ++i) {
_storage[_nextIndex++] = value.charAt(i);
}
return new CompactString(_storage, start, value.length(), value.hashCode());
}
}
static {
allocateStorage();
}
public static CompactString valueOf(String value) {
// You can implement a soft .intern-like solution here.
if (value == null) {
return null;
} else if (value.length() == 0) {
return EMPTY;
} else if (value.length() > LENGTH_THRESHOLD) {
// You would need to change .toCharArray() and length to do UTF encoding.
return new CompactString(value.toCharArray(), 0, value.length(), value.hashCode());
} else {
return storeInShared(value);
}
}
// left to reader: implement equals etc.
}