We Keep Coding

Vertx: pass initial data to verticle on it deployment

How can I pass copy of int[][] array to verticle on it deployment?
I have a ServerVerticle from which deploy 5-10 ServiceVerticles.
Each of ServiceVerticle must use the same shared data structure - Map<Integer, Short[]> which can be 100-2000 Mb.
Problem - I can't create Local map with array as a value.
The only in-memory solution I see - pass copy of int[][] to each ServiceVerticle on it deployment and keep 5-10 copies of data.
P.S. This data structure must have as fast as possible lookup, so I dislike cluster-wide solutions like Hazelcast IMap.

While there isn't much freedom in the types you can use in a LocalMap you can use Buffers. A buffer is an optimized byte array and you can quickly adapt it to your use case. Using a Buffer also means you will have a compact in memory representation so you can save memory and any operations will be fast.
You only need to write a transformation from a 2D plane to a 1D line. For example say that you have the following array (2 x 3):
int[][] data = new int[] {
new int[] {1, 2, 3},
new int[] {4, 5, 6},
};
If you transform it to a buffer:
Buffer.buffer()
.appendInt(1).appendInt(2).appendInt(3)
.appendInt(4).appendInt(5).appendInt(6);
(You can later just use the byte representation, this is just to illustrate how it works).
Now you can refer to any x, y by doing:
int getInt(int x, int y) {
// transform from 2D to 1D
int pos = y * LENGTH + x;
// Where LENGTH is the the example: 3
// For safety assert on length
if (pos > buffer.length()) throw new ArrayIndexOutOfBoundsException();
// TODO: assert on min, max (x, y)
return buffer.getInt(pos);
}

Convert matrix back to array in EJML

I have started using the EJML library for representing matrices. I will use the SimpleMatrix. I did not find two important things which I need. Perhaps somebody can help me identify if the following operations are possible and if yes, how this can be done:
Is it possible to convert a matrix back to a 1D double array (double[]) or 2D double array (double[][]) without just looping through all elements which would be very inefficient? I did not find a method for that. For example, Jeigen library provides a conversion to a 1D array (but I don't know how this is internally done).
Is it possible to delete a row or column?
By the way, does somebody know how EJML compares to Jeigen for large matrices in terms of runtime? EJML provides much more functionality and is much better documented but I'm a bit afraid in terms of runtime.

The underlying array of a SimpleMatrix (it's always 1-dimensional to keep all elements in the same area in RAM) can be retrieved by first getting the underlying DenseMatrix64F and then getting the public data field of D1Matrix64F base class
// where matrix is a SimpleMatrix
double[] data = matrix.getMatrix().data;
I don't see a straightforward way to delete arbitrary rows, columns. One workaround is to use extractMatrix (it copies the underlying double[]) to get 2 parts of the original matrix and then combine them to a new matrix. E.g. to delete the middle column of this 2x3 matrix :
SimpleMatrix fullMatrix = new SimpleMatrix(new double[][]{{2, 3, 4}, {7, 8, 9}});
SimpleMatrix a = fullMatrix.extractMatrix(0, 2, 0, 1);
SimpleMatrix b = fullMatrix.extractMatrix(0, 2, 2, 3);
SimpleMatrix matrix = a.combine(0, 1, b);
Or to delete specifically the first column you can simply do:
SimpleMatrix matrix = fullMatrix.extractMatrix(0, 2, 1, 3);
Or to delete specifically the last column you can simply do (doesn't delete, copy underlying data[]):
matrix.getMatrix().setNumCols(matrix.numCols() - 1);
I will refer to this answer for benchmarks / performance of various java matrix libraries. The performance of ejml is excellent for small matrices and for say size 100 or more doesn't compete well with libraries backed by native C/C++ libraries (like Jeigen). As always, your mileage may vary.

Manos' answer to the first question did not work for me. This is what I did instead:
public double[][] matrix2Array(SimpleMatrix matrix) {
double[][] array = new double[matrix.numRows()][matrix.numCols()];
for (int r = 0; r < matrix.numRows(); r++) {
for (int c = 0; c < matrix.numCols(); c++) {
array[r][c] = matrix.get(r, c);
}
}
return array;
}
I don't know how it compares in performance to other methods, but it works fast enough for what I needed it for.

How do I convert mixed Java data types into a Java byte array?

I need to construct a Java byte array out of mixed data types, but I don't know how to do this. These are my types:
byte version = 1; // at offset 0
short message_length = // the size of the byte[] message I am constructing here, at offset 1
short sub_version = 15346; // at offset 3
byte message_id = 2; // at offset 5
int flag1 = 10; // at offset 6
int flag2 = 0; // at offset 10
int flag3 = 0; // at offset 14
int flag4 = 0; // at offset 18
String message = "the quick brown fox jumps over the lazy dog"; // at offset 22
I know for the String, I can use
message.getBytes("US_ASCII");
I know for the int values, I can use
Integer.byteValue();
I know for the short values, I can use
Short.byteValue();
And the byte values are already bytes, I am just not sure of how to combine all of these into a single byte array. I have read about
System.arraycopy();
Is this the correct process, I just convert all the data to bytes, and start "concatenating" the byte array with arraycopy?
I am communicating with some distant server I have no control over, and this is the message process they require.

Wrap a DataOutputStream around a ByteArrayOutputStream. This way you can write all the primitive types like int and short directly to the DataOutputStream, which converts them to bytes and forwards them to the ByteArrayOutputStream, from which you can then retrieve the whole thing as one byte array:
ByteArrayOutputStream bOut = new ByteArrayOutputStream();
DataOutputStream dOut = new DataOutputStream(bOut);
dOut.writeByte(version);
dOut.writeShort(message_length);
dOut.writeShort(sub_version);
dOut.writeByte(message_id);
dOut.writeInt(flag1);
dOut.writeInt(flag2);
dOut.writeInt(flag3);
dOut.writeInt(flag4);
dOut.write(message.getBytes(), 0, message.length());
dOut.flush();
byte[] result = bOut.toByteArray();
The best thing about this is that you can do the exact opposite (extracting values from a byte array) with DataInputStream and ByteArrayInputStream completely analoguously to the above code.

If by a 'mixed type' you mean a class with different member field types, then one approach is to make your class serializable, and use ApacheUtils
byte[] data = SerializationUtils.serialize(yourObject);

All, I wanted to post my own solution to my problem here. I did a quick Google search on how to insert a short into java byte array. One of the results talked about a Java ByteBuffer. After some reading, I determined this was the best and quickest way for me to get the results I needed. One section in the Java API that really made me interested in the ByteBuffer was this:
Methods in this class that do not otherwise have a value to return are specified to return the buffer upon which they are invoked. This allows method invocations to be chained. The sequence of statements
bb.putInt(0xCAFEBABE);
bb.putShort(3);
bb.putShort(45);
can, for example, be replaced by the single statement
bb.putInt(0xCAFEBABE).putShort(3).putShort(45);
So, that is what I did:
byte version = 1;
short message_length = 72;
short sub_version = 15346;
byte message_id = 2;
int flag1 = 10;
int flag2 = 0;
int flag3 = 0;
int flag4 = 0;
String message = "the quick brown fox jumps over the lazy dog";
ByteBuffer messageBuffer = ByteBuffer.allocate(message_length);
messageBuffer.put(version).putShort(message_length).putShort(sub_version).put(message_id).putInt(flag1).putInt(flag2).putInt(flag3).putInt(flag4).put(message.getBytes());
byte[] myArray = messageBuffer.array();
That was fast and easy, and just what I needed. Thank you all who took the time to read and reply.

Certainly you can concatenate these values with arrayCopy, as you've suggested.
You can also append your bytes onto a ByteArrayOutputStream.
The key is to understand exactly what the receiving system is expecting. How does it know where one field ends and the next begins? How does it know what type it's reading at a given position in the stream? There are lots of ways they could have chosen to do that - with length headers in the protocol; with type headers; with null-termination of strings; with a set order of fields and their lengths; and so on.
Whatever method you choose, write unit tests that check for edge cases like negative numbers, very large numbers, non-ASCII text and so on. It's easy to get stung when everything has been working fine, then suddenly the server chokes on a Unicode character or a negative number that it interprets as a very large number.
One other option -- perhaps slight overkill for your needs, but flexible and with high performance -- is Google's protocol buffers library.

why can't byte array be stored in integer array in java

This code is valid
int h;
byte r;
h=r;
but these are not
int[] h;
byte[] r;
h=r;
or say
int[] h =new byte[4];
I would like to know why?

There's an implicit conversion from byte to int, but not from byte[] to int[]. This makes a lot of sense - the JIT compiler knows that to get to a value in an int[], it just needs to multiply the index by 4 and add that to the start of the data (after validation, and assuming no extra padding, of course). That wouldn't work if you could assign a byte[] reference to an int[] variable - the representations are different.
The language could have been designed to allow that conversion but make it create a new int[] which contained a copy of all the bytes, but that would have been pretty surprising in terms of the design of the rest of Java, where the assignment operator just copies a value from the right hand side of the operator to the variable on the left.
Alternatively, we could have imposed a restriction on the VM that every array access would have to look at the actual type of the array object in question, and work out how to get to the element appropriately... but that would have been horrible (even worse than the current nastiness of reference-type array covariance).

That's the design. When you assign byte to wider int, that's okay. But when you declare new byte[4], that's a ["continuous"] part of memory which is, roughly speaking, equal to 4 * 8 bits (or 4 bytes). And one int is 32 bits, so, technically, all your byte array's size is equal to size of one int. In C, where you have a direct memory access, you could do some pointer magic and get your byte pointer casted to int pointer. In Java, you cant and that's safe.
Anyway, why do you want that?
Disclaimer: the code below is considered to be extremely unlikely seen anywhere except for the most critical sections in some performance-sensitive libraries/apps. Ideone: http://ideone.com/e14Omr
Comments are explanatory enough, I hope.
import sun.misc.Unsafe;
import java.lang.reflect.Field;
public class Main {
public static void main(String[] args) throws NoSuchFieldException, IllegalAccessException, InstantiationException {
/* too lazy to run with VM args, use Reflection */
Field f = Unsafe.class.getDeclaredField("theUnsafe");
f.setAccessible(true);
/* get array address */
Unsafe unsafe = (Unsafe)f.get(null);
byte four_bytes[] = {25, 25, 25, 25};
Object trash[] = new Object[] { four_bytes };
long base_offset_bytes = unsafe.arrayBaseOffset(Object[].class);
long four_bytes_address = unsafe.getLong(trash, base_offset_bytes); // <- this is it
long ints_addr = unsafe.allocateMemory(16); // allocate 4 * 4 bytes, i.e. 4 ints
unsafe.copyMemory(four_bytes_address + base_offset_bytes, ints_addr, 4); // copy all four bytes
for(int i = 0; i < 4; i++) {
System.out.println(unsafe.getInt(ints_addr + i)); //run through entire allocated int[],
// get some intestines
}
System.out.println("*****************************");
for(int i = 0; i < 16; i++) {
System.out.println(unsafe.getByte(ints_addr + i)); //run through entire allocated int[],
// get some intestines
}
}
}

The difference is firstly due to the difference in behavior between primitive types and reference types.
In case you're not familiar with it, primitive types have "value semantics". This means that when you do a = b; when a and b are a primitive type (byte, short, int, long, float, double, boolean, or char) the numeric/boolean value is copied. For example:
int a = 3;
int b = a; // int value of a is copied to b
a = 5;
System.out.println(b); // outputs: 3
But arrays are objects, and objects have "reference semantics". That means that when you do a = b; where a and b are both declared as an array type, the array object that is referred to becomes shared. In a sense the value is still copied, but here the "value" is just the pointer to the object located elsewhere in memory. For example:
int[] a = new int[] { 3 };
int[] b = a; // pointer value of a is copied to b, so a and b now point at the same array object
a[0] = 5;
System.out.println(b[0]); // outputs: 5
a = null; // note: 'a' now points at no array, although this has no effect on b
System.out.println(b[0]); // outputs: 5
So it is okay to do int = byte because the numeric value is going to be copied (as they are both primitive types) and also because any possible value of type byte can be safely stored in an int (it is a "widening" primitive conversion).
But int[] and byte[] are both object types, so when you do int[] = byte[] you are asking for the object (the array) to be shared (not copied).
Now you have to ask, why can't an int array and a byte array share their array memory? And what would if mean if they did?
Ints are 4 times the size of bytes, so if the int and byte arrays were to have the same number of elements, then this causes all sorts of nonsense. If you tried to implement it in a memory efficient way, then complex (and very slow) run-time logic would be needed when accessing elements of int arrays to see if they were actually byte arrays. Int reads from byte array memory would have to read and widen the byte value, and int stores would have to either lose the upper 3 bytes, or throw an exception saying that there isn't enough space. Or, you could do it in a fast but memory-wasting way, by padding all byte arrays so that there are 3 wasted bytes per element, just in case somebody wants to use the byte array as an int array.
On the other hand, perhaps you want to pack 4 bytes per int (in this case, the shared array won't have the same number of elements depending on the type of the variable you use to access it). Unfortunately this also causes nonsense. The biggest problem is that it is not portable across CPU architectures. On a little-endian PC, b[0] would refer to the low byte of i[0], but on an ARM device b[0] might point at the high byte of i[0] (and it could even change while the program is running as ARM has a switchable endianness). The overhead of accessing the array's length property would also be made more complicated, and just what should happen if the byte array's length is not divisible by 4?!
You can do this in C, but that's because C arrays don't have a well-defined length property and because C doesn't try to protect you from the other issues. C doesn't care if you go outside the array bounds or muddle up endianness. But Java does care, so it is not feasible to share the array memory in Java. (Java doesn't have unions.)
That's why int[].class and byte[].class both separately extend class Object, but neither class extends the other. You can't store a reference to a byte array in a variable that is declared to point at int arrays, in the same way you can't store a reference to a List in a variable of type String; they're just incompatible classes.

When you say
int[] arr = new byte[5];
you copy references. On the right hand side is a reference to a byte array. Essentially, this looks like:
|__|__|__|__|__|
0 1 2 3 4 offset of elements, in bytes
^
|
reference to byte array
On the left hand side is a reference to an int array. This, however, is expected to look thus:
|________|________|________|________|________|
0 4 8 12 16
^
|
reference to int array
Hence, simply copying the reference is not possible. For, to get arr[1], the code would look at the starting address+4 (rather than starting adress+1).
The only way to achieve what you want is to create an int[] that has the same number of elements and copy the bytes there.
The rationale behind not doing that automatically:
interpreting a single byte as an int comes at essentially no cost, especially no memory must be allocated.
copying a byte array is completly different. The new int array must be allocated, which is at least 4 times at big as the byte array. The copy process itself could take some time.
Conclusion: In Java, you can always say "I want to treat this special byte as if it were an int." But you can not say: "I want to treat some data structure (like an array, or a class instance) that contains bytes as if it contained ints."

Simply, Type byte[] does not extend int[]

you cant because its like big element is going to be stored in smaller one.Integer cant be stored in byte.Its our memory design who decides these type of allocation

How to pass a pointer to a struct array to a function in java using JNA?

I need to pass pointer of an array of IplImage
(IplImage extends CvArray extends Structure implements cloneable) to a function The native code in C is as follows:
cvCalcEigenObjects(
nTrainFaces,
(void*)faceImgArr,
(void*)eigenVectArr,
CV_EIGOBJ_NO_CALLBACK,
0,
0,
&calcLimit,
pAvgTrainImg,
eigenValMat->data.fl);
I tried this:
cvCalcEigenObjects(
nTrainFaces,
faceImgArr[0].getPointer(),
eigenVectArr[0].getPointer(),
CV_EIGOBJ_NO_CALLBACK,
0,
null,
calcLimit,
pAvgTrainImg,
eigenValMat.data.getFloatArray(0, Pointer.SIZE));
but it didn't work. The declaration of this function in Java is like this:
public static void cvCalcEigenObjects(int i,
Pointer pntr,
Pointer pntr1,
int i1,
int 2,
Pointer pntr2,
cxcore.CvTermCriteria ctc,
cxcore.IplImage ii,
FloatBuffer fb)

Your C prototype is quite unclear but I'll give you something that's not obvious at first glance in JNA but that might be the cause of your troubles.
When dealing with array of structures you need to do something like these :
// Syntax to get a new empty structure array (4 cells) to pass to a function
// which will populate it
MyStructureClass[] incomingStructArray = new MyStructureClass().toArray(4);
// Syntax to transform a standard java array to an array suitable
// to be passed to a C function
MyStructureClass[] standardJavaStructArray = ....
MyStructureClass[] outgoingStructArray = new MyStructureClass().toArray(standardJavaStructArray);
Now if you wonder why one would need to do so (which is completly crazy from the java point of view) you need to remember you're not coding Java, you're coding C
A standard java array is in fact a void* but a standard C array is a MyStructure*
If MyStructure uses 12 Bytes in memory :
a 4 cell Java array of MyStructureClass uses 16 Bytes (= 4 cell x 4 Bytes per pointer) in memory (not entirely true but let say so ; if all cells != null then an additional 48 Bytes will be used for the MyStructureClass themselves )
a 4 cell C array of MyStructure uses 48 Bytes (= 4 cell x 12 Bytes per MyStructure)
That's why when using JNA and array of structures you need to be very carefull with what you do, beacause an array of structure is very different from an array of pointers to structure