We Keep Coding

Is java array elements always stored in contiguous memory incase of reference types?

I've read that The elements of an array are stored in a contiguous memory location . Is it always the case ? I understand this is the case with array of primitives. I'm a bit confused about array of objects. An array of objects basically contains reference of the real objects right? references must be stored in contiguous locations but what about the real objects are they really stored in contiguous locations?

Is it always the case ?
Yes, but keep reading.
I'm a bit confused about array of objects. An array of objects basically contains reference of the real objects right? references must be stored in contiguous locations but what about the real objects are they really stored in contiguous locations?
Exactly, what the array contains is object references, not the objects themselves. Think of object references as being similar to long values that tell the JVM where the object is elsewhere in memory (it's more complicated than that, but that's a useful mental model). The object references are stored in the array's contiguous memory block. The objects are elsewhere, and may or may not be stored in contiguous memory (probably not).
So for instance, say we have:
int[] a = new int[] { 1, 2, 3 };
In memory we'll get something like:
+−−−−−−−+
a: Ref51234−−−>| int[] |
+−−−−−−−+
| 1 |
| 2 |
| 3 |
+−−−−−−−+
a contains a reference to the array, which is elsewhere in memory and has a contiguous data block containing 1, 2, and 3.
Now let's look at an object array (where the Example class stores the given constructor parameter as a private x field):
Example[] a = new Example[] { new Example(1), new Example(2), new Example(3) };
That might give us something like this:
+−−−−−−−−−−−+ +−−−−−−−−−+
a: Ref51234−−−>| Example[] | +−−−>| Example |
+−−−−−−−−−−−+ | +−−−−−−−−−+
| Ref81372 |−−−−−+ | x: 1 |
| Ref96315 |−−−−+ +−−−−−−−−−+
| Ref12975 |−−+ | +−−−−−−−−−+
+−−−−−−−−−−−+ | +−−−−−−−−−−−−−−−−−−>| Example |
| +−−−−−−−−−+
| | x: 2 |
| +−−−−−−−−−+
| +−−−−−−−−−+
+−−>| Example |
+−−−−−−−−−+
| x: 3 |
+−−−−−−−−−+
Some environments have packed arrays as an adjunct. For instance, IBM documents packed arrays for its environment here.

Shenandoah 2.0 elimination of forwarding pointer

In Shenandoah 1.0 every single Object had an additional header - called forwarding pointer. Why was that needed and what is the reason that lead to its elimination in Shenandoah 2.0?

First of all, every single java Object has two headers: klass and mark. They have been there in each instance since forever (they can slightly change how a JVM handles their flags internally with recent JVMs, for example) and are used for various reasons (will go into detail about only one of them a bit further in the answer).
The need for a forwarding pointer is literally in the second part of this answer. The forwarding pointer is needed in both read barrier and write barrier in Shenandoah 1.0 (though the read could skip the barrier for some field types - will not go into detail). In very simple words it simplifies concurrent copy very much. As said in that answer, it allows to atomically switch the forwarding pointer to the new copy of the Object and then concurrently update all references to point to that new Object.
Things have changed a bit in Shenandoah 2.0 where the "to-space invariant" is in place : meaning all the writes and reads are done via the to-space.This means one interesting thing : once the to-space copy is established, the from-copy is never used. Imagine a situation like this:
refA refB
| |
fwdPointer1 ---- fwdPointer2
|
--------- ---------
| i = 0 | | i = 0 |
| j = 0 | | j = 0 |
--------- ---------
In Shenandoah 1.0 there were cases when reading via the refA could bypass the barrier (not use it at all) and still read via the from-copy. This was allowed for final fields, for example (via a special flag). This means that even if to-space copy already existed and there were already references to it, there could still be reads (via refA) that would go to the from-space copy. In Shenandoah 2.0 this is prohibited.
This information was used in a rather interesting way. Every object in Java is aligned to 64 bits - meaning the last 3 bits are always zero. So, they dropped the forwarding pointer and said that : if the last two bits of the mark word are 11 (this is allowed since no else uses it in this manner) -> this is a forwarding pointer, otherwise the to-space copy does yet exists and this is a plain header. You can see it in action right here and you can trace the masking here and here.
It used to look like this:
| -------------------|
| forwarding Pointer |
| -------------------|
| -------------------|
| mark |
| -------------------|
| -------------------|
| class |
| -------------------|
And has transformed to:
| -------------------|
| mark or forwarding | // depending on the last two bits
| -------------------|
| -------------------|
| class |
| -------------------|
So here is a possible scenario (I'll skip class header for simplicity):
refA, refB
|
mark (last two bits are 00)
|
---------
| i = 0 |
| j = 0 |
---------
GC kicks in. The object referenced by refA/refB is alive, thus must be evacuated (it is said to be in the "collection set"). First a copy is created and atomically mark is made to reference that copy (also the last two bits are marked as 11 to now make it a forwardee and not a mark word):
refA, refB
|
mark (11) ------ mark (00)
|
--------- ---------
| i = 0 | | i = 0 |
| j = 0 | | j = 0 |
--------- ---------
Now one of the mark words has a bit pattern (ends in 11) that indicates that it is a forwardee and not a mark word anymore.
refA refB
| |
mark (11) ------ mark (00)
|
--------- ---------
| i = 0 | | i = 0 |
| j = 0 | | j = 0 |
--------- ---------
refB can move concurrently, so then refA, ultimately there are not references to the from-space object and it is garbage. This is how mark word acts as a forwarding pointer, if needed.

Implement minhash LSH using Spark (Java)

this is quite long, and I am sorry about this.
I have been trying to implement the Minhash LSH algorithm discussed in chapter 3 by using Spark (Java). I am using a toy problem like this:
+--------+------+------+------+------+
|element | doc0 | doc1 | doc2 | doc3 |
+--------+------+------+------+------+
| d | 1 | 0 | 1 | 1 |
| c | 0 | 1 | 0 | 1 |
| a | 1 | 0 | 0 | 1 |
| b | 0 | 0 | 1 | 0 |
| e | 0 | 0 | 1 | 0 |
+--------+------+------+------+------+
the goal is to identify, among these four documents (doc0,doc1,doc2 and doc3), which documents are similar to each other. And obviously, the only possible candidate pair would be doc0 and doc3.
Using Spark's support, generating the following "characteristic matrix" is as far as I can reach at this point:
+----+---------+-------------------------+
|key |value |vector |
+----+---------+-------------------------+
|key0|[a, d] |(5,[0,2],[1.0,1.0]) |
|key1|[c] |(5,[1],[1.0]) |
|key2|[b, d, e]|(5,[0,3,4],[1.0,1.0,1.0])|
|key3|[a, c, d]|(5,[0,1,2],[1.0,1.0,1.0])|
+----+---------+-------------------------+
and here is the code snippets:
CountVectorizer vectorizer = new CountVectorizer().setInputCol("value").setOutputCol("vector").setBinary(false);
Dataset<Row> matrixDoc = vectorizer.fit(df).transform(df);
MinHashLSH mh = new MinHashLSH()
.setNumHashTables(5)
.setInputCol("vector")
.setOutputCol("hashes");
MinHashLSHModel model = mh.fit(matrixDoc);
Now, there seems to be two main calls on the MinHashLSHModel model that one can use: model.approxSimilarityJoin(...) and model.approxNearestNeighbors(...). Examples about using these two calls are here: https://spark.apache.org/docs/latest/ml-features.html#lsh-algorithms
On the other hand, model.approxSimilarityJoin(...) requires us to join two datasets, and I have only one dataset which has 4 documents and I would like to figure out which ones in these four are similar to each other, so I don't have a second dataset to join... Just to try it out, I actually joined my only dataset with itself. Based on the result, seems like model.approxSimilarityJoin(...) just did a pair-wise Jaccard calculation, and I don't see any impact by changing the number of Hash functions etc, left me wondering about where exactly the minhash signature was calculated and where the band/row partition has happened...
The other call, model.approxNearestNeighbors(...), actually asks a comparison point, and then the model will identify the nearest neighbor(s) to this given point... Obviously, this is not what I wanted either, since I have four toy documents, and I don't have an extra reference point.
I am running out of ideas, so I went ahead implemented my own version of the algorithm, using Spark APIs, but not much support from MinHashLSHModel model, which really made me feel bad. I am thinking I must have missed something... ??
I would love to hear any thoughts, really wish to solve the mystery.
Thank you guys in advance!

The minHash signatures calculation happens in
model.approxSimilarityJoin(...) itself where model.transform(...)
function is called on each of the input datasets and hash signatures
are calculated before joining them and doing a pair-wise jaccard
distance calculation. So, the impact of changing the number of hash
functions can be seen here.
In model.approxNearestNeighbors(...),
the impact of the same can be seen while creating the model using
minHash.fit(...) function in which transform(...) is called on
the input dataset.

Why set insert the item in ordered?

Actually Set is not an ordered one. I just create the set and insert the numbers 5,2,10.
Wen it is printed in the console, it prints as 2,5,10.
Why since set is not ordered?

This is because this speeds up queries for whether a certain element is part of the set.
The difference is that this behavior is not guaranteed. It may be beneficial to keep small sets ordered for fast lookup, but switch to a hash based implementation once a certain number of elements has been reached, at which point elements would suddenly be sorted by hash value.

Set is an interface while it has several implementations. HashSet is not guaranteed your insertion order(not ordered). LinkedHashSet preserve insertion order and TreeSet give you a sorted set(sorted set).
Then you insert 5,2,10 to HashSet you can't guaranteed the same order.

Set is just an interface, assuming that you are talking about HashSet (because it's where this happens), it doesn't keep them sorted. For example:
HashSet<Integer> set = new HashSet<Integer>();
set.add(1);
set.add(16);
System.out.println(set);
Output
[16, 1]
This is because an HashSet uses the hashcode function to compute the index where the item will be stored in an array-like structure. This way, since the hashcode never changes, it can extract the element from the correct index computing again the hashcode and checking the cell at that index.
The hashcode function converts most classes to an integer:
System.out.println(new Integer(1).hashCode()); # 1
System.out.println(new Integer(1000).hashCode()); # 1000
System.out.println("Hello".hashCode()); # 69609650
Each class can define its own way to compute the hashcode and Integer returns itself.
As you can see numbers get big soon, and we don't want to have an array with 1000 cells just to save the two integers.
To avoid the problem we can create an array with n elements and then use the remainder of the hashcode divided by n as the index.
For example if we want to find the index for 1000 in an array of 16 elements:
System.out.println(new Integer(1000).hashCode() % 16); # 8
So our dictionary will know that the integer 1000 is at index 8. That's how HashSet is implemented.
So, why [16, 1] is not ordered? That's because HashSet are created with 16 elements as capacity at the beginning (when not differently specified), and grow as needed (more on this here).
Let's compute the index to store the data having key = 2 and key = 9 in a dictionary with n = 8:
System.out.println(new Integer(1).hashCode() % 16); # 1
System.out.println(new Integer(16).hashCode() % 16); # 0
This means that the array that contains the dictionary data will be:
| index | value |
|-------|-------|
| 0 | 16 |
| 1 | 1 |
| 2 | |
| 3 | |
| 4 | |
| 5 | |
| 6 | |
| 7 | |
| 8 | |
| 9 | |
| 10 | |
| 11 | |
| 12 | |
| 13 | |
| 14 | |
| 15 | |
Iterating over it, the order will be the one presented in this representation, so 16 will be before 1.

Set is the Interface. It only indicates avoid duplicate entity of the collection.
HashSet internally uses Hashmap. Normally hashmap uses hashcode. So It wont return in ordered way. If you want insertion order you will use LinkedHashMap.

Set is just an interface. Ordering will depend on implementation. For example TreeSet is an ordered implementation of Set.

How is an object stored in heap?

How exactly an object is stored in heap. For example, a bicycle class can be defined like this:
public class Bicycle {
public int gear;
public int speed;
public Bicycle(int startSpeed, int startGear) {
gear = startGear;
speed = startSpeed;
}
public void setGear(int newValue) {
gear = newValue;
}
public void applyBrake(int decrement) {
speed -= decrement;
}
public void speedUp(int increment) {
speed += increment;
}
}
then I can create an bicycle object:
Bicycle bicycle = new Bicycle(20,10)
Then this bicycle object should be stored in heap. But I don't understand how heap exactly store these instance variables and methods such as speed and gear.I understand that heap should be implemented as tree. So how the object is stored in a tree? Also when you use bicycle.speed to find the value of speed, what will be time complexity?

Want someone else to do your CS homework eh? ;)
Depending on the language the exact implementation will be different (how it's stored in memory), but the general concept is the same.
You have your stack memory and your heap memory, the local variables and parameters go on the stack, and whenever you new something it goes into the heap. It's called a stack because values are pushed onto it when you declare it or call a function, and popped off and they go out of scope.
+--------------+
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
+--------------+
Stack Heap
Each instance variable takes up however much memory its type does (depending on the language), the compiler adds it all up and that's the sizeof the type (à la C++). Methods go into code space and does not get newed into the object (I think for now you'll be better off to not consider this in learning about how memory is organized, just think of it as magic).
So in your example:
Bicycle bicycle = new Bicycle(20,10)
bicycle is a reference to the heap memory address, today in most languages/systems it will cost you either 32 and 64 bits on the stack.
The new allocates memory in the heap. The compiler figures out the size of Bicycle and creates assembly/machine code that allocates the amount of memory it requires.
This is how the memory looks after this line:
+--------------+
| | | Bicycle obj |
| | |--------------|
| | | |
| | | |
|--------------| | |
| bicycle ref | | |
+--------------+
Stack Heap
More specifically, since the Bicycle class has two instance variables (or fields as they are called in Java) and both are ints, and an int in Java is 32 bits or 4 bytes, the size of your Bicycle object is 4 bytes * 2 fields = 8 bytes.
+-------------+
| | 0| gear |
| | 4| speed |
| | |-------------|
| | 8| |
|-------------| 12| |
| bicycle=0x4 | | |
+--------------+
Stack Heap
Time complexity to access memory is O(1). The compiler is able to figure out the exact memory address of speed, since as the second int field in the object, is at bicycle+0x4.

Bicycle bicycle = new Bicycle(20,10)
The reference bicycle will be store in stack whereas the object and instance variables will be store in heap and the address of the heap is assigned in the stack so mean to say stack will link to heap.

First of all, you should understand the meaning of Object in terms of Java.
Object is nothing but just a buffer(memory area) in Heap. That buffer or memory area is called Object.
Object contains all the non-static data member of the class.
All the-
Object stores in Heap.
Static data member stores in Class Area.
Reference variable stores in Stack.
Method (static or non-static) stores in Method Area.
Read more on Java Memory Model