So my question is basically if there is even the possibility to implement a custom made JNI so that it can give the possibility to remove immediately the object from the heap instead of waiting for the Garbage Collector to act over it.
My question goal is focusing mostly on the memory that consumption when using temporary functions that create a lot of objects to calculate somthing.
for example: I have a function like this:
public Integer[] countDuplicates(int[] values) {
Map<Integer, Integer> l = new HashMap<Integer, Integer>();
Integer c;
for (int v : values) {
c = l.get(v);
if (c == null) {
c = 0;
}
l.put(v, c + 1);
}
Integer[] result = l.values().toArray(new int[l.size()]);
// < A Way to free the Map from the heap>
return result;
}
as you can see, there is no need for the Map after the end of the method, so, my question is, is there any way, either using JNI, or Implementing any command on the GC, or even a tagword that will force the removal from the heap after the method run?
Related
I am working on a Processing program for Brownian motion tracking.
I have an ArrayList blobs and ArrayList tomerge. The first one is a list of particles which I track and the second one is a list of particles which I want to merge.
Every particle is a Blob class object. Blob object countains ArrayList of Vectors called lespoints and int id in its data.
Since I need to merge a few particles in one, I need to destroy some Blob objects, but Java doesn't have any destructors and I don't want to use finalise(). Will this work like merge + destruction?
public void delete(Blob a)
{
a = null;
}
void merge(ArrayList<Blob> tomerge)
{
int i = 0;
int j = 0;
while (i <= tomerge.size())
{
Blob k = new Blob();
k = tomerge.get(i);
while (j <= tomerge.get(i).siz()) {
Vector g = k.lespoints.get(j);
lespoints.add(g);
j++;
}
if (i > 0)
{
delete(tomerge.get(i));
}
i++;
}
}
You don't need to manually do anything. Just make sure you don't have any references to the variables you want to go away, and Java will handle the rest.
For example:
String x = "test";
x = null;
At this point, the value "test" can be garbage collected because nothing points to it.
Compare that to this:
String x = "test";
ArrayList<String> list = new ArrayList<>();
list.add(x);
x = null;
At this point, the value "test" cannot be garabage collected, because the ArrayList still points to it.
But if you did this:
list.remove("test");
Then it could be garbage collected.
So basically, all you need to do is remove the element from your ArrayList and Java will take care of the rest. Note that you probably don't want to do this in your current loop, as removing elements while you iterate over a list can cause you to skip over elements.
Instead, you probably want to use an iterator or just loop backwards over your ArrayList.
Shameless self-promotion: here is a tutorial on ArrayLists, including removing elements from them.
There is an exact reason why your code example won't work.
public void delete(Blob a)
{
a = null;
}
Blob b = new Blob();
delete(b);
In this code example, the reference which is set to null is a, not b.
You are not deleting the Blob object, you are setting the reference to null.
When the delete() method is called, there exists 2 references to the Blob.
One reference is b, which is in the calling code.
The other reference is a, which is in the called code.
a is set to null, and then the method exits. But the b reference continues to exist throughout. Therefore the Blob will never be garbage-collected.
To achieve garbage-collection, you must remove all references to an object; then it gets destructed at the JVM's convenience.
The Java Collections API for removing an object during iteration works like this:
Iterator<Blob> itr = list.iterator();
while( itr.hasNext() ) {
Blob b = itr.next();
if( /* test condition */ ) {
itr.remove(); // Safely removes object from List during iteration
}
} // Object `b` goes out of scope, and so this Blob is "lost" to the the code and is going to be destroyed
I'm currently looking at the heap dump of this silly little test class (taken at the very end of the main method):
public class WeakRefTest {
static final class RefObj1 { int i; }
static final class RefObj2 { int j; }
public static void main(String[] args) {
Set<WeakReference<?>> objects = new HashSet<>();
RefObj1 obj1 = new RefObj1();
RefObj2 obj2 = new RefObj2();
for (int i = 0; i < 1000; i++) {
objects.add(new WeakReference<RefObj1>(obj1));
objects.add(new WeakReference<RefObj2>(obj2));
}
}
}
Now I'm trying to figure out how to count the number of references to a specific class in objects. If this were a SQL database, it'd be easy:
select objects.className as referent, count(*) as cnt
from java.lang.ref.WeakReference ref
inner join heapObjects objects on ref.referent = objects.objectId
group by objects.className;
Result:
referent | cnt
===================
WeakRefTest$RefObj1 | 1000
WeakRefTest$RefObj2 | 1000
After some research, I figured I can construct a Eclipse MAT OQL query that gives me the classes involved:
select DISTINCT OBJECTS classof(ref.referent) from java.lang.ref.WeakReference ref
Alas, this doesn't include their count and OQL doesn't seem to support a GROUP BY clause. Any ideas how to get this information?
Edited to add: In reality, none of the objects added to the Set (nor the Set implementation itself, obviously) are under my control. So sorry, modifying RefObj1 and RefObj2 isn't allowed.
Edit2: I found this related question about using OQL in jvisualvm but it turns out that OQL is actually Javascript unleashed at a heap dump. I'd be fine with something like that, too. But playing around with it hasn't produced results for me, yet. I'll update the question if that changes.
Open the histogram view (there is a toolbar button for this, which looks like a bar graph).
In the first row of the histogram view where it says "Regex", type WeakReference to filter the view.
Find the java.lang.ref.WeakReference line, right-click, and choose "Show Objects By Class" -> "By Outgoing References".
The resulting view should be summarize the objects being referred to, grouped by class as you require. The Objects column should indicate the number of instances for each class.
You could just write a method in the object that returns the information and call that from Eclipse...
Since you cannot modify the object then the next best thing will be to write a utility function in some method that you can modify and call that from the eclipse debugger. I don't know Eclipse well enough to help you do it without inserting something to the source code, sorry.
I would use a Weak HashSet. You can just use set.size() to get the number of references still alive.
static final class RefObj1 { int i; }
static final class RefObj2 { int j; }
public static void main(String[] args) {
Set objects = Collections.newSetFroMap(new WeakHashMap());
RefObj1 obj1 = new RefObj1();
RefObj2 obj2 = new RefObj2();
for (int i = 0; i < 1000; i++) {
objects.add(obj1);
objects.add(obj2);
}
obj1 = null;
System.gc();
System.out.println("Objects left is " + objects.size());
}
I would expect this to print 0, 1 or 2 depending on how the objects are cleaned up.
I wrote a custom index to a custom table which uses 500MB of heap for 500k strings. Only 10% of the strings are unique; the rest are repeats. Every string is of length 4.
How i can optimize my code? Should I use another collection? I tried to implement a custom string pool to save memory:
public class StringPool {
private static WeakHashMap<String, String> map = new WeakHashMap<>();
public static String getString(String str) {
if (map.containsKey(str)) {
return map.get(str);
} else {
map.put(str, str);
return map.get(str);
}
}
}
private void buildIndex() {
if (monitorModel.getMessageIndex() == null) {
// the index, every columns create an index
ArrayList<HashMap<String, TreeSet<Integer>>> messageIndex = new ArrayList<>(filterableColumn.length);
for (int i = filterableColumn.length; i >= 0; i--) {
// key -> string, value -> treeset, the row wich contains the key
HashMap<String, TreeSet<Integer>> hash = new HashMap<>();
messageIndex.add(hash);
}
// create index for every column
for (int i = monitorModel.getParser().getMyMessages().getMessages().size() - 1; i >= 0; --i) {
TreeSet<Integer> tempList;
for (int j = 0; j < filterableColumn.length; j++) {
String value = StringPool.getString(getValueAt(i, j).toString());
if (!messageIndex.get(j).containsKey(value)) {
tempList = new TreeSet<>();
messageIndex.get(j).put(value, tempList);
} else {
tempList = messageIndex.get(j).get(value);
}
tempList.add(i);
}
}
monitorModel.setMessageIndex(messageIndex);
}
}
No need to come up with a custom pool. Just use String.intern().
You might want to examine your memory heap in a profiler. My guess is that the memory consumption isn't primarily in the String storage, but in the many TreeSet<Integer> instances. If so, you could optimize considerably by using primitive arrays (int[], short[], or byte[], depending on the actual size of the integer values you're storing). Or you could look into a primitive collection type, such as those provided by FastUtil or Trove.
If you do find that the String storage is problematic, I'll assume that you want to scale your application beyond 500k Strings, or that especially tight memory constraints require you to deduplicate even short Strings.
As Dev said, String.intern() will deduplicate Strings for you. One caveat, however - in the Oracle and OpenJDK virtual machines, String.intern() will store those Strings in the VM permanent-generation, such that they will not be garbage-collected in the future. That's appropriate (and helpful) if:
The Strings you're storing do not change throughout the life of the VM (e.g., if you read in a static list at startup and use it throughout the life of your application).
The Strings you need to store fit comfortably in the VM permanent generation (with adequate room for classloading and other consumers of PermGen). Update: see below.
If either of those conditions is false, you are probably correct to build a custom pool. But my recommendation is that you consider a simple HashMap in place of the WeakHashMap you're currently using. You probably don't want these values to be garbage-collected while they're in your cache, and WeakHashMap adds another level of indirection (and the associated object pointers), increasing memory consumption further.
Update: I'm told that JDK 7 stores interned Strings (String.intern()) in the main heap, not in perm-gen, as earlier JDKs did. That makes String.intern() less risky if you're using JDK 7.
This question already has answers here:
list.clear() vs list = new ArrayList<Integer>(); [duplicate]
(8 answers)
Closed 8 years ago.
Say we try to implement a merge sort algorithm, given an Array of Arrays to merge what is a better approach, this:
public void merge(ArrayList<ArrayList<E>> a) {
ArrayList<ArrayList<E>> tmp = new ArrayList<ArrayList<E>>() ;
while (a.size()>1) {
for (int i=1; i<a.size();i+=2) {
tmp.add(merge(a.get(i-1),a.get(i)));
}
if (a.size()%2==1) tmp.add(a.get(a.size()-1));
a = tmp;
tmp = new ArrayList<ArrayList<E>>() ;
}
}
or this :
public void merge(ArrayList<ArrayList<E>> a) {
ArrayList<ArrayList<E>> tmp = new ArrayList<ArrayList<E>>(),tmp2 ;
while (a.size()>1) {
for (int i=1; i<a.size();i+=2) {
tmp.add(merge(a.get(i-1),a.get(i)));
}
if (a.size()%2==1) tmp.add(a.get(a.size()-1));
tmp2 = a;
a = tmp;
tmp = tmp2;
tmp.clear();
}
}
to make it clearer, what i was doing is to merge each couple of neighbors in a and put the resulting merged arrays in an external Array of Arrays tmp, after merging all couples, one approach is to clear a and then move tmp to a, and then move the cleared a to tmp.
second approach is to "throw" old tmp and get a new tmp instead of reusing the old one.
As a general rule, don't spend energy trying to reuse old collections; it just makes your code harder to read (and frequently doesn't give you any actual benefit). Only try optimizations like these if you already have your code working, and you have hard numbers that say the speed of your algorithm is improved.
Always allocating a new ArrayList and filling it, will result in more garbage collections which generally slows down everything (minor GCs are cheap but not free).
Reusing the ArrayList will result in less Arrays.copyOf() which is used when the array inside the ArrayList needs to be resized (resizing is cheap but not free).
On the other hand: clear() will also nullify the array content to allow the GC to collect unused object which is of course also not free.
Still, if execution speed is concerned, I would reuse the ArrayList.
Is there any way to implement a type of reference whose value can be exchanged with another atomically?
In Java we have AtomicReference which can be swapped with a local variable but not with another AtomicReference.
You can do:
AtomicReference r1 = new AtomicReference("hello");
AtomicReference r2 = new AtomicReference("world");
and swap them with a combination of two operations:
r1.set(r2.getAndSet(r1.get()));
But this leaves them in an inconsistent state in between, where both contain "hello". Also even if you could swap them atomically, you still could not read them (as a pair) atomically.
What I would like to be able to do is:
PairableAtomicReference r1 = new PairableAtomicReference("hello");
PairableAtomicReference r2 = new PairableAtomicReference("world");
AtomicRefPair rp = new AtomicRefPair(r1, r2);
then
Object[] oldVal, newVal;
do {
oldVal = rp.get();
newVal = new Object[] {oldVal[1], oldVal[0]};
} while (! rp.compareAndSet(oldVal, newVal));
to swap the values, and in another thread:
AtomicRefPair otherRP = new AtomicRefPair(r1, r2);
System.out.println(Arrays.toString(otherRP.get()));
and be certain that the output will be either [hello, world] or [world, hello].
Notes:
r1 and r2 are paired for this operation, but it's possible that another thread will independently pair, say r1 and another r3 (unfortunately that means I cannot use this solution.)
There will be hundreds of thousands of these references, so a global ReentrantLock would be a major bottleneck.
rp and otherRP are not necessarily shared between threads, so simply locking them will not work. They could be interned, but the intern pool would need its own synchronization which would be another bottleneck.
I have only made groups of 2 references here, but the ability to group 3 or more would be a bonus.
Is it possible to implement a lock-free version of AtomicRefPair? I have a hunch that it isn't, but if not then maybe there is an article somewhere that explains why?
Related: How do I atomically swap 2 ints in C#?
Have an immutable class holding the pair. That is your atom. Swapping the pair means replacing the atom.
update: your question isn't very clear. but in general, for a concurrent system consisting of multiple variables, one might want
take a snapshot of system state. the snapshot doesn't change once taken.
atomically update system state by changing multiple variables at once. it may be required that there is no other update between my update an a previous snapshot (which my calculation was based on)
you can model your system directly in snapshots, if it doesn't consume too much resources.
I don't know if there's a nice solution, but the following ugly one could work:
public final class MyReference<T> extends ReentrantLock implements Comparable<MyReference<T>> {
public MyReference() {
id = counter.incrementAndGet();
}
public void swap(MyReference<T> other) {
if (id < other.id) {
lock();
other.lock();
} else {
other.lock();
lock();
}
final T tmp = value;
value = other.value;
other.value = tmp;
unlock();
other.unlock();
}
public static <T> List<T> consistentGet(List<MyReference<T>> references) {
final ArrayList<MyReference<T>> sortedReferences = Lists.newArrayList(references);
Collections.sort(sortedReferences);
for (val r : sortedReferences) r.lock();
final List<T> result = Lists.newArrayListWithExpectedSize(sortedReferences.size());
for (val r : references) result.add(r.value);
for (val r : sortedReferences) r.unlock();
return result;
}
#Override
public int compareTo(MyReference<T> o) {
return id < o.id ? -1 : id > o.id ? 1 : 0;
}
private final static AtomicInteger counter = new AtomicInteger();
private T value;
private final int id;
}
Use MyReference instead of AtomicReference.
It uses a lot of locks, but none of them is global.
It acquires locks in a fixed order, so it's deadlock-free.
It compiles using lombok and guava (take it as pseudocode without them).