I was studying about Weak references. And I understood that all weak references WILL be garbage collected before OutOfMemoryError occurs. I had a simple test something like this (I know catching OOME is not good but just a test) :
Integer weakInt = new Integer(10);
WeakReference<Integer> weakReference = new WeakReference<Integer>(weakInt);
try {
while (weakReference != null) {
String[] generateOutOfMemoryStr = new String[999999999];
}
}
catch (OutOfMemoryError oome) {
System.out.println(weakReference.get());
}
I expected null to be printed because the weak reference SHOULD have been collected but I always get an output of 10.
Please let me know where I am going wrong. May be I understood the concept of weak references wrong?
weakReference itself won't become null... how could it? However, its target can become null.
I suspect you mean:
while (weakReference.get() != null) {
Moreover, I suspect that unless you set weakInt itself to null, that local variable will prevent the Integer object from being garbage collected.
Moreover, I suspect you'll find your loop still won't end - because you're quite possibly asking for more memory than can be allocated even if the Integer is garbage collected.
Here's a program which demonstrates it working, at least on my box:
import java.lang.ref.*;
public class Test {
public static void main(String[] args) {
Integer weakInt = new Integer(10);
WeakReference<Integer> weakReference = new WeakReference<Integer>(weakInt);
weakInt = null;
while (weakReference.get() != null) {
System.out.println("Looping...");
String[] generateOutOfMemoryStr = new String[999999];
}
System.out.println("Weak reference collected");
}
}
Related
is it possible to check at runtime if an object has a direct or indirect reference to another object?
(I know I can use VisualVm or similar to analyze the HeapDump, but i'd like to automate it at runtime)
I'm working with WeakHashMaps, doing something like this:
public class MyClass {
// the Runnable will be eventually removed if the key is collected by the GC
private static final WeakHashMap<Object, Runnable> map = new WeakHashMap<>();
public static void main(String[] args) {
MyClass a = new MyClass(2);
MyClass b = new MyClass(20);
a = null;// no more Strong references to a
b = null;// no more Strong references to b
System.gc();
for (Runnable r : map.values()) {
r.run();
}
// will print (20), becouse using format() in the lambda cause a Strong
// reference to MyClass (b) and thus the WeakHashMap will never remove b
}
public MyClass(int i) {
if (i < 10) {
map.put(this, () -> {
System.out.println(i);
});
} else {
map.put(this, () -> {
// this is subtle, but calling format() create a strong reference
// between the Runnable and MyClass
System.out.println(format(i));
});
}
}
private String format(Integer i) {
return "(" + i + ")";
}
}
in the code, the two instance of MyClass will add themselves (as key) and a runnable (as value) to the WeakHashMap.
In the first instance (a), the Runnable simply call System.out.println() and when the instance a is no more referenced (a = null) the entry will be removed from the map.
In the second instance (b), the Runnable also call format() an instance function of MyClass. This create a strong reference to b and adding the Runnable to the map will result in a lock condition, where the value is an indirect strong reference to the key preventing the collection by the garbage collector.
Now I know to avoid these conditions (for instance, using a Weakreference inside the lambda), but this is really easy to miss in a real scenario, and will cause a memory leak.
So, prior to adding the pair to the map I'd like to check if the value is somehow referencing the key, and throw an exception if so.
This would be a "debug" task, and will be disabled in production, so I don't care if it is slow or an hack.
--- update ---
I'm trying to deal with WeakListeners, and to avoid them to be immediately collected if not referenced.
So i register them as notifier.addWeakListener(holder, e -> { ... });
and this will add the listener to a WeakHashMap preventing the listener to be collected until holder live.
But any reference to the holder in the listener will create a lock :(
Is there a better way?
The Reflection API gives you access to all fields of a run-time object (and its runtime type, and possibly the Class object). In theory, you could traverse through the tree of you instance's fields (and static fields on the class), the fields' fields etc.
While this is possible, I doubt it would be feasible. You write you don't care about performance, but it may even be too slow for development runs. Which brings us to the Rule 1 of implementing your own cache: Don't do it.
There is already a builtin feature for associations which are automatically cleaned up, ordinary instance fields. I.e
public class MyClass {
public static void main(String[] args) {
MyClass a = new MyClass(2);
MyClass b = new MyClass(20);
WeakReference<MyClass> aRef = new WeakReference<>(a), bRef = new WeakReference<>(b);
a = null;// no more Strong references to a
b = null;// no more Strong references to b
System.gc();
if(aRef.get() == null) System.out.println("a collected");
if(bRef.get() == null) System.out.println("b collected");
}
Runnable r;
public MyClass(int i) {
if (i < 10) {
r = () -> System.out.println(i);
} else {
r = () -> {
// reference from Runnable to MyClass is no problem
System.out.println(format(i));
};
}
}
private String format(Integer i) {
return "(" + i + ")";
}
}
You can put these associated objects into a weak hashmap as keys, to allow them to get garbage collected, which, of course, will only happen when the particular MyClass instance, which still holds a strong reference to it, gets garbage collected:
public class MyClass {
public static void main(String[] args) {
MyClass a = new MyClass(2);
MyClass b = new MyClass(20);
for(Runnable r: REGISTERED) r.run();
System.out.println("cleaning up");
a = null;// no more Strong references to a
b = null;// no more Strong references to b
System.gc();
// empty on common JRE implementations
for(Runnable r: REGISTERED) r.run();
}
static Set<Runnable> REGISTERED = Collections.newSetFromMap(new WeakHashMap<>());
Runnable r;
public MyClass(int i) {
r = i < 10?
() -> System.out.println(i):
() -> {
// reference from Runnable to MyClass is no problem
System.out.println(format(i));
};
REGISTERED.add(r);
}
private String format(Integer i) {
return "(" + i + ")";
}
}
But note that what works smoothly in this simple test setup is nothing you should rely on, especially as you mentioned weak listeners.
In production environments, the garbage collector runs when there are memory needs, which is not connected to application logic, i.e. whether particular actions implemented as listeners should be executed or not. One possible scenario would be that there is always enough memory, so the garbage collector never runs and obsolete listeners keep being executed forever.
But you may encounter problems into the other direction too. Your question suggests that it might be possible to write your listeners (Runnable in the example) in a way that they don’t contain references to the instance whose life time ought to determine the listener’s life time (the MyClass instance). This raises the question, in which way the life times of these objects are connected at all. You have to keep strong references to these key objects, for the sake of keeping these listeners alive, which is error prone too.
I am a fresh new learner of Java. I'm now learning the concept of WeakReference. I came across a problem which probably looks stupid but I just wanna figure out the reason. The problem is: according to Java doc, "Weak reference objects, which do not prevent their referents from being made finalizable, finalized, and then reclaimed."
So I did this small test:
import java.lang.ref.WeakReference;
public class A {
public static void main(String[] args) {
A a = new A();
WeakReference<A> wr = new WeakReference<>(a);
a = null;
A a1 = wr.get();
System.out.println(a);
System.out.println(a1);
try {
System.gc();
Thread.sleep(10000);
} catch (Exception e) {
e.printStackTrace();
}
System.out.println(a1);
}
#Override
protected void finalize( ) {
System.out.println(Thread.currentThread().getName() + ": See ya, nerds!");
}
}
However, I noticed that after GC running, wr.get() could still return object which I expected null, and the method finalize() was not invoked. So what went wrong? Thanks for your kind help in advance! :)
The premise of your test is flawed. System.gc() is only a hint to run the garbage collector. It is frequently ignored.
From the documentation:
Calling the gc method suggests that the Java Virtual Machine
expend effort toward recycling unused objects in order to make the
memory they currently occupy available for quick reuse. When control
returns from the method call, the Java Virtual Machine has made a best
effort to reclaim space from all discarded objects.
(Emphasis mine)
In future, you may use the VM options -verbose:gc and -XX:+PrintGCDetails to see what the garbage collector is doing.
More importantly, you are also very quickly taking the reference out of the weak reference and putting it back into a strong reference:
A a = new A();
WeakReference<A> wr = new WeakReference<>(a);
a = null; // no strong references remain
A a1 = wr.get(); // the instance now has a strong reference again
Unless garbage collection occurs between these exact two instructions, the object will not be garbage collected.
If you remove a1, your code behaved as you would expect when I ran it (though, because of the first part of my answer, your mileage may vary):
class A
{
public static void main(String[] args)
{
A a = new A();
WeakReference<A> wr = new WeakReference<>(a);
a = null;
System.out.println(a);
try {
System.gc(); // instance of A is garbage collected
Thread.sleep(10000);
} catch (Exception e) {
e.printStackTrace();
}
System.out.println(wr.get());
}
#Override
protected void finalize( )
{
System.out.println(Thread.currentThread().getName() + ": See ya, nerds!");
}
}
Firstly, System.gc() does not ensure a garbage collection. Instead, it's just a hint that "It's a good time to run garbage collection".
Secondly, in your code when you put A a1 = wr.get(); before calling System.gc(), it creates a new strong reference to the same object referenced by a, thus even if garbage collection runs, your object will not be garbage collected.
As we have two tasks in hand
Ensure garbage collection
Don't keep any strong reference to the object you want to be garbage collected
Let's do little modification to your code
public class A {
public static void main(String[] args) {
A a = new A();
WeakReference<A> wr = new WeakReference<>(a);
a = null;
// A a1 = wr.get(); Removing this, this does our 2nd task
System.out.println(a);
// System.out.println(a1); Removing this as a1 does not exists anymore
try {
while (null != wr.get()) { // 1st task done, the loop ensures sending the hint until your object collected
System.gc();
// Thread.sleep(10000); it does not have impact
}
} catch (Exception e) {
e.printStackTrace();
}
System.out.println(wr.get()); // Obviously prints null
}
#Override
protected void finalize() {
System.out.println(Thread.currentThread().getName() + ": See ya, nerds!");
}
}
This fails
public void testWeak() throws Exception {
waitGC();
{
Sequence a = Sequence.valueOf("123456789");
assert Sequence.used() == 1;
a.toString();
}
waitGC();
}
private void waitGC() throws InterruptedException {
Runtime.getRuntime().gc();
short count = 0;
while (count < 100 && Sequence.used() > 0) {
Thread.sleep(10);
count++;
}
assert Sequence.used() == 0: "Not removed!";
}
The test fails. Telling Not removed!.
This works:
public void testAWeak() throws Exception {
waitGC();
extracted();
waitGC();
}
private void extracted() throws ChecksumException {
Sequence a = Sequence.valueOf("123456789");
assert Sequence.used() == 1;
a.toString();
}
private void waitGC() throws InterruptedException {
Runtime.getRuntime().gc();
short count = 0;
while (count < 100 && Sequence.used() > 0) {
Thread.sleep(10);
count++;
}
assert Sequence.used() == 0: "Not removed!";
}
It seems like the curly brackets does not affect the weakness.
Some official resources?
Scope is a compile-time thing. It is not determining the reachability of objects at runtime, only has an indirect influence due to implementation details.
Consider the following variation of your test:
static boolean WARMUP;
public void testWeak1() throws Exception {
variant1();
WARMUP = true;
for(int i=0; i<10000; i++) variant1();
WARMUP = false;
variant1();
}
private void variant1() throws Exception {
AtomicBoolean track = new AtomicBoolean();
{
Trackable a = new Trackable(track);
a.toString();
}
if(!WARMUP) System.out.println("variant1: "
+(waitGC(track)? "collected": "not collected"));
}
public void testWeak2() throws Exception {
variant2();
WARMUP = true;
for(int i=0; i<10000; i++) variant2();
WARMUP = false;
variant2();
}
private void variant2() throws Exception {
AtomicBoolean track = new AtomicBoolean();
{
Trackable a = new Trackable(track);
a.toString();
if(!WARMUP) System.out.println("variant2: "
+(waitGC(track)? "collected": "not collected"));
}
}
static class Trackable {
final AtomicBoolean backRef;
public Trackable(AtomicBoolean backRef) {
this.backRef = backRef;
}
#Override
protected void finalize() throws Throwable {
backRef.set(true);
}
}
private boolean waitGC(AtomicBoolean b) throws InterruptedException {
for(int count = 0; count < 10 && !b.get(); count++) {
Runtime.getRuntime().gc();
Thread.sleep(1);
}
return b.get();
}
on my machine, it prints:
variant1: not collected
variant1: collected
variant2: not collected
variant2: collected
If you can’t reproduce it, you may have to raise the number of warmup iterations.
What it demonstrates: whether a is in scope (variant 2) or not (variant 1) doesn’t matter, in either case, the object has not been collected in cold execution, but got collected after a number of warmup iterations, in other words, after the optimizer kicked in.
Formally, a is always eligible for garbage collection at the point we’re invoking waitGC(), as it is unused from this point. This is how reachability is defined:
A reachable object is any object that can be accessed in any potential continuing computation from any live thread.
In this example, the object can not be accessed by potential continuing computation, as no such subsequent computation that would access the object exists. However, there is no guaranty that a particular JVM’s garbage collector is always capable of identifying all of those objects at each time. In fact, even a JVM not having a garbage collector at all would still comply to the specification, though perhaps not the intent.
The possibility of code optimizations having an effect on the reachability analysis has also explicitly mentioned in the specification:
Optimizing transformations of a program can be designed that reduce the number of objects that are reachable to be less than those which would naively be considered reachable. For example, a Java compiler or code generator may choose to set a variable or parameter that will no longer be used to null to cause the storage for such an object to be potentially reclaimable sooner.
So what happens technically?
As said, scope is a compile-time thing. At the bytecode level, leaving the scope defined by the curly braces has no effect. The variable a is out of scope, but its storage within the stack frame still exists holding the reference until overwritten by another variable or until the method completes. The compiler is free to reuse the storage for another variable, but in this example, no such variable exists. So the two variants of the example above actually generate identical bytecode.
In an unoptimized execution, the still existing reference within the stack frame is treated like a reference preventing the object’s collection. In an optimized execution, the reference is only held until its last actual use. Inlining of its fields can allow its collection even earlier, up to the point that it is collected right after construction (or not getting constructed at all, if it hadn’t a finalize() method). The extreme end is finalize() called on strongly reachable object in Java 8…
Things change, when you insert another variable, e.g.
private void variant1() throws Exception {
AtomicBoolean track = new AtomicBoolean();
{
Trackable a = new Trackable(track);
a.toString();
}
String message = "variant1: ";
if(!WARMUP) System.out.println(message
+(waitGC(track)? "collected": "not collected"));
}
Then, the storage of a is reused by message after a’s scope ended (that’s of course, compiler specific) and the object gets collected, even in the unoptimized execution.
Note that the crucial aspect is the actual overwriting of the storage. If you use
private void variant1() throws Exception {
AtomicBoolean track = new AtomicBoolean();
{
Trackable a = new Trackable(track);
a.toString();
}
if(!WARMUP)
{
String message = "variant1: "
+(waitGC(track)? "collected": "not collected");
System.out.println(message);
}
}
The message variable uses the same storage as a, but its assignment only happens after the invocation of waitGC(track), so you get the same unoptimized execution behavior as in the original variant.
By the way, don’t use short for local loop variables. Java always uses int for byte, short, char, and int calculations (as you know, e.g. when trying to write shortVariable = shortVariable + 1;) and requiring it to cut the result value to short (which still happens implicitly when you use shortVariable++), adds an additional operation, so if you thought, using short improved the efficiency, notice that it actually is the opposite.
I have come across this example from wikipedia regarding weak reference:
import java.lang.ref.WeakReference;
public class ReferenceTest {
public static void main(String[] args) throws InterruptedException {
WeakReference r = new WeakReference(new String("I'm here"));
WeakReference sr = new WeakReference("I'm here");
System.out.println("before gc: r=" + r.get() + ", static=" + sr.get());
System.gc();
Thread.sleep(100);
// only r.get() becomes null
System.out.println("after gc: r=" + r.get() + ", static=" + sr.get());
}
}
I don't understand in this scenario why only r.get() returns null but not the sr.get(). Can someone let me know the reason?
Many thanks.
the literal "I'm here" is a compile time constant string and as such gets placed in the constant string pool, which (up until java 7) was never garbage collected. that means sr points to an object that will never be garbage collected.
r, on the other hand, points to a copy of that string, which is not in any const pool and so is eligible for collection.
see the documentation for String.intern() for some more details on this string pool
If this reference object has been cleared, either by the program or by the garbage collector, then this method returns null.
very well explained here
I'm going to use a SoftReference-based cache (a pretty simple thing by itself). However, I've came across a problem when writing a test for it.
The objective of the test is to check if the cache does request the previously cached object from the server again after the memory cleanup occurs.
Here I find the problem how to make system to release soft referenced objects. Calling System.gc() is not enough because soft references will not be released until the memory is low. I'm running this unit test on the PC so the memory budget for the VM could be pretty large.
================== Added later ==============================
Thank you all who took care to answer!
After considering all pro's and contra's I've decided to go the brute force way as advised by nanda and jarnbjo. It appeared, however, that JVM is not that dumb - it won't even attempt garbage collecting if you ask for a block which alone is bigger than VM's memory budget. So I've modified the code like this:
/* Force releasing SoftReferences */
try {
final List<long[]> memhog = new LinkedList<long[]>();
while(true) {
memhog.add(new long[102400]);
}
}
catch(final OutOfMemoryError e) {
/* At this point all SoftReferences have been released - GUARANTEED. */
}
/* continue the test here */
This piece of code forces the JVM to flush all SoftReferences. And it's very fast to do.
It's working better than the Integer.MAX_VALUE approach, since here the JVM really tries to allocate that much memory.
try {
Object[] ignored = new Object[(int) Runtime.getRuntime().maxMemory()];
} catch (OutOfMemoryError e) {
// Ignore
}
I now use this bit of code everywhere I need to unit test code using SoftReferences.
Update: This approach will indeed work only with less than 2G of max memory.
Also, one need to be very careful with SoftReferences. It's so easy to keep a hard reference by mistake that will negate the effect of SoftReferences.
Here is a simple test that shows it working every time on OSX. Would be interested in knowing if JVM's behavior is the same on Linux and Windows.
for (int i = 0; i < 1000; i++) {
SoftReference<Object> softReference = new SoftReferencelt<Object>(new Object());
if (null == softReference.get()) {
throw new IllegalStateException("Reference should NOT be null");
}
try {
Object[] ignored = new Object[(int) Runtime.getRuntime().maxMemory()];
} catch (OutOfMemoryError e) {
// Ignore
}
if (null != softReference.get()) {
throw new IllegalStateException("Reference should be null");
}
System.out.println("It worked!");
}
An improvement that will work for more than 2G max memory. It loops until an OutOfMemory error occurs.
#Test
public void shouldNotHoldReferencesToObject() {
final SoftReference<T> reference = new SoftReference<T>( ... );
// Sanity check
assertThat(reference.get(), not(equalTo(null)));
// Force an OoM
try {
final ArrayList<Object[]> allocations = new ArrayList<Object[]>();
int size;
while( (size = Math.min(Math.abs((int)Runtime.getRuntime().freeMemory()),Integer.MAX_VALUE))>0 )
allocations.add( new Object[size] );
} catch( OutOfMemoryError e ) {
// great!
}
// Verify object has been garbage collected
assertThat(reference.get(), equalTo(null));
}
Set the parameter -Xmx to a very
small value.
Prepare your soft
reference
Create as many object as
possible. Ask for the object everytime until it asked the object from server again.
This is my small test. Modify as your need.
#Test
public void testSoftReference() throws Exception {
Set<Object[]> s = new HashSet<Object[]>();
SoftReference<Object> sr = new SoftReference<Object>(new Object());
int i = 0;
while (true) {
try {
s.add(new Object[1000]);
} catch (OutOfMemoryError e) {
// ignore
}
if (sr.get() == null) {
System.out.println("Soft reference is cleared. Success!");
break;
}
i++;
System.out.println("Soft reference is not yet cleared. Iteration " + i);
}
}
You could explicitly set the soft reference to null in your test, and as such simulate that the soft reference has been released.
This avoids any complicated test setup that is memory and garbage collection dependend.
Instead of a long running loop (as suggested by nanda), it's probably faster and easier to simply create a huge primitive array to allocate more memory than available to the VM, then catch and ignore the OutOfMemoryError:
try {
long[] foo = new long[Integer.MAX_VALUE];
}
catch(OutOfMemoryError e) {
// ignore
}
This will clear all weak and soft references, unless your VM has more than 16GB heap available.