How can I pass copy of int[][] array to verticle on it deployment?
I have a ServerVerticle from which deploy 5-10 ServiceVerticles.
Each of ServiceVerticle must use the same shared data structure - Map<Integer, Short[]> which can be 100-2000 Mb.
Problem - I can't create Local map with array as a value.
The only in-memory solution I see - pass copy of int[][] to each ServiceVerticle on it deployment and keep 5-10 copies of data.
P.S. This data structure must have as fast as possible lookup, so I dislike cluster-wide solutions like Hazelcast IMap.
While there isn't much freedom in the types you can use in a LocalMap you can use Buffers. A buffer is an optimized byte array and you can quickly adapt it to your use case. Using a Buffer also means you will have a compact in memory representation so you can save memory and any operations will be fast.
You only need to write a transformation from a 2D plane to a 1D line. For example say that you have the following array (2 x 3):
int[][] data = new int[] {
new int[] {1, 2, 3},
new int[] {4, 5, 6},
};
If you transform it to a buffer:
Buffer.buffer()
.appendInt(1).appendInt(2).appendInt(3)
.appendInt(4).appendInt(5).appendInt(6);
(You can later just use the byte representation, this is just to illustrate how it works).
Now you can refer to any x, y by doing:
int getInt(int x, int y) {
// transform from 2D to 1D
int pos = y * LENGTH + x;
// Where LENGTH is the the example: 3
// For safety assert on length
if (pos > buffer.length()) throw new ArrayIndexOutOfBoundsException();
// TODO: assert on min, max (x, y)
return buffer.getInt(pos);
}
Related
in java if we have:
int[][] x = new int[3][3];
the memory address of x is different from the memory address of x[0]. As x[0] gives the memory address of the first column. So the memory address of x[0][0] is different than the memory address of x[0].
are there any computer languages that store a 2d array as a matrix and not as an array of arrays?
would the address of x always be different from x[0] and the address of x[0] equal x[0][0]?
are there any computer languages that store a 2d array as a matrix and
not as an array of arrays?
Yes. Or, at least, there used to be.
There is the possibility of using an assembler language, where the programmer has extreme control over how arrays might be handled. But, let's assume the question is about high-level languages (>=3GL).
I don't know about modern version of Fortran, but the early versions of FORTRAN stored any array, including multi-dimensional arrays, in consecutive storage locations. So, for example, if you declared an array as INTEGER FOO (3,4,5), then FOO and FOO (1,1,1) would have the same memory address. FOO would occupy a block of 60 INTEGER sized locations. The compiler generates code to find, from the subscript values, the location of an element in a manner similar to what #Jesse described in a comment on the question. It's slightly different to allow for the fact that FORTRAN subscripts started at one instead of zero.
By the way, FORTRAN subscript are in opposite order of most other languages. In Java, C, C++, and COBOL, the major subscripts are to the left. In FORTRAN, they were to the right.
FORTRAN syntax didn't allow missing subscripts. So, continuing the example, something like FOO (2,3) would generate a compiler error.
Now, suppose there was the following method:
REAL FUNCTION MEAN (ARR, N)
INTEGER N, ARR (N)
REAL SUM
DO 400 I = 1,N,1
SUM = SUM + ARR (I)
400 CONTINUE
RETURN SUM / N
END
A programmer could it use to calculate the mean of the entire FOO array, or any part of it:
REAL ALLMEAN, LEVEL3MEAN, ROWMEAN
ALLMEAN = MEAN (FOO(1,1,1), 60)
LEVEL3MEAN = MEAN (FOO(1,1,3), 12)
ROWMEAN = MEAN (FOO(1,2,3), 4)
Suppose, for some strange reason, there was this:
AVGPART = MEAN (FOO (2,3,2), 20)
This would use 20 consecutive elements of FOO, even if those elements were in different rows or levels.
When I took a C++ course, someone didn't like having to type separate [x] subscripts for multidimensional arrays. Instead of foo [2][1][0], he would rather type something like foo.get (2,1,0), so wrote a convenience wrapper class for an array. Such code might still have foo [t][r][c] inside the wrapper class. Or, it could allocate a 1D array. Once the class was created, it allowed him to specify subscripts as arguments in a call to a method.
Code in Java to do that, using the 1D array option, might look like this:
public class Block {
// class for regular 3D Array
private int [] array;
private int rows, int columns, int levels;
public Block (int t, int r, int c) {
rows = r;
columns = c;
levels = t;
array = new array [ t * r * c];
}
public int get (int t, int r, int c) {
return array [ t * rows * columns + r * columns + c ];
}
public void set (int value, int t, int r, int c) {
array [ t * rows * columns + r * columns + c ] = value;
}
...
}
PdfPTable table = new PdfPTable(new float[]{4,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,2});
This is the initializer I currently use.
It's for a scorecard in Disc Golf, using integers to tell how wide each cell should be, but some courses have different numbers of holes (9, 18 ,24, etc). The array MUST start with a 4, for the players name, and MUST end with a 2, for totals. All cell values for individual holes are set at 1. I want to save code by using a variable in the initializer. Any help would be awesome!!
You do know that you can create arrays of variable size by using a variable as the array length?
public float[] newFloatArray(int size) {
float[] array = new float[size];
return array;
}
Filling the array can be done with a loop or using the JRE supplied Arrays class helper methods. You will need to handle the first and last index in the array separately:
public float[] newGolfArray(int size) {
float[] array = new float[size];
Arrays.fill(array, 1F);
array[0] = 4F;
array[size - 1] = 2F;
return array;
}
I have started using the EJML library for representing matrices. I will use the SimpleMatrix. I did not find two important things which I need. Perhaps somebody can help me identify if the following operations are possible and if yes, how this can be done:
Is it possible to convert a matrix back to a 1D double array (double[]) or 2D double array (double[][]) without just looping through all elements which would be very inefficient? I did not find a method for that. For example, Jeigen library provides a conversion to a 1D array (but I don't know how this is internally done).
Is it possible to delete a row or column?
By the way, does somebody know how EJML compares to Jeigen for large matrices in terms of runtime? EJML provides much more functionality and is much better documented but I'm a bit afraid in terms of runtime.
The underlying array of a SimpleMatrix (it's always 1-dimensional to keep all elements in the same area in RAM) can be retrieved by first getting the underlying DenseMatrix64F and then getting the public data field of D1Matrix64F base class
// where matrix is a SimpleMatrix
double[] data = matrix.getMatrix().data;
I don't see a straightforward way to delete arbitrary rows, columns. One workaround is to use extractMatrix (it copies the underlying double[]) to get 2 parts of the original matrix and then combine them to a new matrix. E.g. to delete the middle column of this 2x3 matrix :
SimpleMatrix fullMatrix = new SimpleMatrix(new double[][]{{2, 3, 4}, {7, 8, 9}});
SimpleMatrix a = fullMatrix.extractMatrix(0, 2, 0, 1);
SimpleMatrix b = fullMatrix.extractMatrix(0, 2, 2, 3);
SimpleMatrix matrix = a.combine(0, 1, b);
Or to delete specifically the first column you can simply do:
SimpleMatrix matrix = fullMatrix.extractMatrix(0, 2, 1, 3);
Or to delete specifically the last column you can simply do (doesn't delete, copy underlying data[]):
matrix.getMatrix().setNumCols(matrix.numCols() - 1);
I will refer to this answer for benchmarks / performance of various java matrix libraries. The performance of ejml is excellent for small matrices and for say size 100 or more doesn't compete well with libraries backed by native C/C++ libraries (like Jeigen). As always, your mileage may vary.
Manos' answer to the first question did not work for me. This is what I did instead:
public double[][] matrix2Array(SimpleMatrix matrix) {
double[][] array = new double[matrix.numRows()][matrix.numCols()];
for (int r = 0; r < matrix.numRows(); r++) {
for (int c = 0; c < matrix.numCols(); c++) {
array[r][c] = matrix.get(r, c);
}
}
return array;
}
I don't know how it compares in performance to other methods, but it works fast enough for what I needed it for.
This code is valid
int h;
byte r;
h=r;
but these are not
int[] h;
byte[] r;
h=r;
or say
int[] h =new byte[4];
I would like to know why?
There's an implicit conversion from byte to int, but not from byte[] to int[]. This makes a lot of sense - the JIT compiler knows that to get to a value in an int[], it just needs to multiply the index by 4 and add that to the start of the data (after validation, and assuming no extra padding, of course). That wouldn't work if you could assign a byte[] reference to an int[] variable - the representations are different.
The language could have been designed to allow that conversion but make it create a new int[] which contained a copy of all the bytes, but that would have been pretty surprising in terms of the design of the rest of Java, where the assignment operator just copies a value from the right hand side of the operator to the variable on the left.
Alternatively, we could have imposed a restriction on the VM that every array access would have to look at the actual type of the array object in question, and work out how to get to the element appropriately... but that would have been horrible (even worse than the current nastiness of reference-type array covariance).
That's the design. When you assign byte to wider int, that's okay. But when you declare new byte[4], that's a ["continuous"] part of memory which is, roughly speaking, equal to 4 * 8 bits (or 4 bytes). And one int is 32 bits, so, technically, all your byte array's size is equal to size of one int. In C, where you have a direct memory access, you could do some pointer magic and get your byte pointer casted to int pointer. In Java, you cant and that's safe.
Anyway, why do you want that?
Disclaimer: the code below is considered to be extremely unlikely seen anywhere except for the most critical sections in some performance-sensitive libraries/apps. Ideone: http://ideone.com/e14Omr
Comments are explanatory enough, I hope.
import sun.misc.Unsafe;
import java.lang.reflect.Field;
public class Main {
public static void main(String[] args) throws NoSuchFieldException, IllegalAccessException, InstantiationException {
/* too lazy to run with VM args, use Reflection */
Field f = Unsafe.class.getDeclaredField("theUnsafe");
f.setAccessible(true);
/* get array address */
Unsafe unsafe = (Unsafe)f.get(null);
byte four_bytes[] = {25, 25, 25, 25};
Object trash[] = new Object[] { four_bytes };
long base_offset_bytes = unsafe.arrayBaseOffset(Object[].class);
long four_bytes_address = unsafe.getLong(trash, base_offset_bytes); // <- this is it
long ints_addr = unsafe.allocateMemory(16); // allocate 4 * 4 bytes, i.e. 4 ints
unsafe.copyMemory(four_bytes_address + base_offset_bytes, ints_addr, 4); // copy all four bytes
for(int i = 0; i < 4; i++) {
System.out.println(unsafe.getInt(ints_addr + i)); //run through entire allocated int[],
// get some intestines
}
System.out.println("*****************************");
for(int i = 0; i < 16; i++) {
System.out.println(unsafe.getByte(ints_addr + i)); //run through entire allocated int[],
// get some intestines
}
}
}
The difference is firstly due to the difference in behavior between primitive types and reference types.
In case you're not familiar with it, primitive types have "value semantics". This means that when you do a = b; when a and b are a primitive type (byte, short, int, long, float, double, boolean, or char) the numeric/boolean value is copied. For example:
int a = 3;
int b = a; // int value of a is copied to b
a = 5;
System.out.println(b); // outputs: 3
But arrays are objects, and objects have "reference semantics". That means that when you do a = b; where a and b are both declared as an array type, the array object that is referred to becomes shared. In a sense the value is still copied, but here the "value" is just the pointer to the object located elsewhere in memory. For example:
int[] a = new int[] { 3 };
int[] b = a; // pointer value of a is copied to b, so a and b now point at the same array object
a[0] = 5;
System.out.println(b[0]); // outputs: 5
a = null; // note: 'a' now points at no array, although this has no effect on b
System.out.println(b[0]); // outputs: 5
So it is okay to do int = byte because the numeric value is going to be copied (as they are both primitive types) and also because any possible value of type byte can be safely stored in an int (it is a "widening" primitive conversion).
But int[] and byte[] are both object types, so when you do int[] = byte[] you are asking for the object (the array) to be shared (not copied).
Now you have to ask, why can't an int array and a byte array share their array memory? And what would if mean if they did?
Ints are 4 times the size of bytes, so if the int and byte arrays were to have the same number of elements, then this causes all sorts of nonsense. If you tried to implement it in a memory efficient way, then complex (and very slow) run-time logic would be needed when accessing elements of int arrays to see if they were actually byte arrays. Int reads from byte array memory would have to read and widen the byte value, and int stores would have to either lose the upper 3 bytes, or throw an exception saying that there isn't enough space. Or, you could do it in a fast but memory-wasting way, by padding all byte arrays so that there are 3 wasted bytes per element, just in case somebody wants to use the byte array as an int array.
On the other hand, perhaps you want to pack 4 bytes per int (in this case, the shared array won't have the same number of elements depending on the type of the variable you use to access it). Unfortunately this also causes nonsense. The biggest problem is that it is not portable across CPU architectures. On a little-endian PC, b[0] would refer to the low byte of i[0], but on an ARM device b[0] might point at the high byte of i[0] (and it could even change while the program is running as ARM has a switchable endianness). The overhead of accessing the array's length property would also be made more complicated, and just what should happen if the byte array's length is not divisible by 4?!
You can do this in C, but that's because C arrays don't have a well-defined length property and because C doesn't try to protect you from the other issues. C doesn't care if you go outside the array bounds or muddle up endianness. But Java does care, so it is not feasible to share the array memory in Java. (Java doesn't have unions.)
That's why int[].class and byte[].class both separately extend class Object, but neither class extends the other. You can't store a reference to a byte array in a variable that is declared to point at int arrays, in the same way you can't store a reference to a List in a variable of type String; they're just incompatible classes.
When you say
int[] arr = new byte[5];
you copy references. On the right hand side is a reference to a byte array. Essentially, this looks like:
|__|__|__|__|__|
0 1 2 3 4 offset of elements, in bytes
^
|
reference to byte array
On the left hand side is a reference to an int array. This, however, is expected to look thus:
|________|________|________|________|________|
0 4 8 12 16
^
|
reference to int array
Hence, simply copying the reference is not possible. For, to get arr[1], the code would look at the starting address+4 (rather than starting adress+1).
The only way to achieve what you want is to create an int[] that has the same number of elements and copy the bytes there.
The rationale behind not doing that automatically:
interpreting a single byte as an int comes at essentially no cost, especially no memory must be allocated.
copying a byte array is completly different. The new int array must be allocated, which is at least 4 times at big as the byte array. The copy process itself could take some time.
Conclusion: In Java, you can always say "I want to treat this special byte as if it were an int." But you can not say: "I want to treat some data structure (like an array, or a class instance) that contains bytes as if it contained ints."
Simply, Type byte[] does not extend int[]
you cant because its like big element is going to be stored in smaller one.Integer cant be stored in byte.Its our memory design who decides these type of allocation
I'm currently trying to migrate a bit of legacy code from iPhone to Android. This code uses the OpenCV library to do some image processing. Overall it goes well, but I'm stuck on one line of code I have no idea how can be converted into Java code:
Scalar dMean;
Scalar scalar;
std::vector<Mat> channels;
split(mat, channels);
for(int i=0; i<3; ++i) {
channels[i] += dMean[i];
}
The question is - what should be used instead of the += operator in Java code to add a Scalar object to a Mat?
Note: take this answer with a grain of salt, I haven't fully tested this ;)
OPTION1:
The most direct way, and if you are only going to process a few pixels, is by using your_mat.put(row, col, data) and your_mat.get(row, col).
Because the put() method does not accept Scalar objects as the data parameter, you have to convert the Scalar to something that put() accepts.
So if your Scalar is (1,2,3) maybe an int array int[] scalar = {1,2,3}; should do the trick.
int[] scalar = ... // convert from Scalar object
// assuming the result of get() is an int[], sum both arrays:
int[] data = your_mat.get(row, col) + scalar // <- pseudo-code alert :D
your_mat.put(row, col, data);
OPTION2:
But the recommended way, for a lot of pixel processing, is to first convert a Mat to a Java primitive, process the primitive and then convert it back to Mat. This is to avoid too many JNI calls, this method does 2 JNI calls while the former makes one per put/get.
The corresponding Java primitive array type depends on the Mat type:
CV_8U and CV_8S -> byte[];
CV_16U and CV_16S -> short[];
CV_32S -> int[];
CV_32F -> float[];
CV_64F-> double[];
So the code will be something like this:
// assuming Mat it's of CV_32S type
int buff[] = new int[your_mat.total() * your_mat.channels()];
your_mat.get(0, 0, buff);
// buff is now Mat converted to int[]
your_mat.put(0, 0, buff); // after processing buff, convert it back to Mat type
OPTION 3:
Ok so those solutions are pretty ugly, this one is not the most effective but it's a little less ugly, in a Java way:
List<Integer> scalarList = ... // your conversion from a Scalar to a List
List<Integer> channelsList = new ArrayList<Integer>();
Converters.Mat_to_vector_int(channels, channelsList); // this is an existing OpenCV4Android converter
// process channelsList and scalarList, store in channelsList
channels = Converters.vector_int_to_Mat(channelsList); // after processing, convert it back to Mat type
Now that I think about it option 3 is quite similar to option 2, if OpenCV's Converters work similar internally as the option 2 conversion.