I wanted to test Java WeakHashMap Class functionality and for that matter I wrote the following test:
public class WeakHashMapTest {
public static void main(String args[]) {
Map<String, Object> weakMap = new WeakHashMap<>();
String x = new String("x");
String x1 = new String("x1");
String x2 = new String("x2");
weakMap.put(x, x);
weakMap.put(x1, x1);
weakMap.put(x2, x2);
System.out.println("Map size :" + weakMap.size());
// force all keys be eligible
x=x1=x2=null;
// call garbage collector
System.gc();
try {
Thread.sleep(1000);
} catch (InterruptedException e) {
e.printStackTrace();
}
System.out.println("Map size :" + weakMap.size());
System.out.println("Map :" + weakMap);
}
}
After running the class WeakMapTest I was unpleasantly surprised to get the following output:
map before gc: {x=x, x1=x1, x2=x2}
map after gc: {x=x, x1=x1, x2=x2}
when I expected that the map will be empty.
That is, the garbage collector didn't do its work. But why?
The WeakHashMap will have its keys reclaimed by the garbage collector when they are no longer strongly reachable.
Implementation note: The value objects in a WeakHashMap are held by ordinary strong references. Thus care should be taken to ensure that value objects do not strongly refer to their own keys, either directly or indirectly, since that will prevent the keys from being discarded.
However, because you used the key itself as the value, the value is still strongly reachable, which means that the garbage collector can't reclaim the keys.
However, if you use other objects as values, then the only references to the keys will be the keys themselves.
weakMap.put(x, new Object());
weakMap.put(x1, new Object());
weakMap.put(x2, new Object());
Then, after clearing the variables and calling the garbage collector as you've done already, I get the output:
Map size :3
Map size :0
Map :{}
Even if the call to System.gc() doesn't guarantee the garbage collector runs, it looks like it does run here.
System.gc() is effectively a suggestion to run the garbage collector. There is no guaranteed way to force the garbage collector to run.
Related
is it possible to check at runtime if an object has a direct or indirect reference to another object?
(I know I can use VisualVm or similar to analyze the HeapDump, but i'd like to automate it at runtime)
I'm working with WeakHashMaps, doing something like this:
public class MyClass {
// the Runnable will be eventually removed if the key is collected by the GC
private static final WeakHashMap<Object, Runnable> map = new WeakHashMap<>();
public static void main(String[] args) {
MyClass a = new MyClass(2);
MyClass b = new MyClass(20);
a = null;// no more Strong references to a
b = null;// no more Strong references to b
System.gc();
for (Runnable r : map.values()) {
r.run();
}
// will print (20), becouse using format() in the lambda cause a Strong
// reference to MyClass (b) and thus the WeakHashMap will never remove b
}
public MyClass(int i) {
if (i < 10) {
map.put(this, () -> {
System.out.println(i);
});
} else {
map.put(this, () -> {
// this is subtle, but calling format() create a strong reference
// between the Runnable and MyClass
System.out.println(format(i));
});
}
}
private String format(Integer i) {
return "(" + i + ")";
}
}
in the code, the two instance of MyClass will add themselves (as key) and a runnable (as value) to the WeakHashMap.
In the first instance (a), the Runnable simply call System.out.println() and when the instance a is no more referenced (a = null) the entry will be removed from the map.
In the second instance (b), the Runnable also call format() an instance function of MyClass. This create a strong reference to b and adding the Runnable to the map will result in a lock condition, where the value is an indirect strong reference to the key preventing the collection by the garbage collector.
Now I know to avoid these conditions (for instance, using a Weakreference inside the lambda), but this is really easy to miss in a real scenario, and will cause a memory leak.
So, prior to adding the pair to the map I'd like to check if the value is somehow referencing the key, and throw an exception if so.
This would be a "debug" task, and will be disabled in production, so I don't care if it is slow or an hack.
--- update ---
I'm trying to deal with WeakListeners, and to avoid them to be immediately collected if not referenced.
So i register them as notifier.addWeakListener(holder, e -> { ... });
and this will add the listener to a WeakHashMap preventing the listener to be collected until holder live.
But any reference to the holder in the listener will create a lock :(
Is there a better way?
The Reflection API gives you access to all fields of a run-time object (and its runtime type, and possibly the Class object). In theory, you could traverse through the tree of you instance's fields (and static fields on the class), the fields' fields etc.
While this is possible, I doubt it would be feasible. You write you don't care about performance, but it may even be too slow for development runs. Which brings us to the Rule 1 of implementing your own cache: Don't do it.
There is already a builtin feature for associations which are automatically cleaned up, ordinary instance fields. I.e
public class MyClass {
public static void main(String[] args) {
MyClass a = new MyClass(2);
MyClass b = new MyClass(20);
WeakReference<MyClass> aRef = new WeakReference<>(a), bRef = new WeakReference<>(b);
a = null;// no more Strong references to a
b = null;// no more Strong references to b
System.gc();
if(aRef.get() == null) System.out.println("a collected");
if(bRef.get() == null) System.out.println("b collected");
}
Runnable r;
public MyClass(int i) {
if (i < 10) {
r = () -> System.out.println(i);
} else {
r = () -> {
// reference from Runnable to MyClass is no problem
System.out.println(format(i));
};
}
}
private String format(Integer i) {
return "(" + i + ")";
}
}
You can put these associated objects into a weak hashmap as keys, to allow them to get garbage collected, which, of course, will only happen when the particular MyClass instance, which still holds a strong reference to it, gets garbage collected:
public class MyClass {
public static void main(String[] args) {
MyClass a = new MyClass(2);
MyClass b = new MyClass(20);
for(Runnable r: REGISTERED) r.run();
System.out.println("cleaning up");
a = null;// no more Strong references to a
b = null;// no more Strong references to b
System.gc();
// empty on common JRE implementations
for(Runnable r: REGISTERED) r.run();
}
static Set<Runnable> REGISTERED = Collections.newSetFromMap(new WeakHashMap<>());
Runnable r;
public MyClass(int i) {
r = i < 10?
() -> System.out.println(i):
() -> {
// reference from Runnable to MyClass is no problem
System.out.println(format(i));
};
REGISTERED.add(r);
}
private String format(Integer i) {
return "(" + i + ")";
}
}
But note that what works smoothly in this simple test setup is nothing you should rely on, especially as you mentioned weak listeners.
In production environments, the garbage collector runs when there are memory needs, which is not connected to application logic, i.e. whether particular actions implemented as listeners should be executed or not. One possible scenario would be that there is always enough memory, so the garbage collector never runs and obsolete listeners keep being executed forever.
But you may encounter problems into the other direction too. Your question suggests that it might be possible to write your listeners (Runnable in the example) in a way that they don’t contain references to the instance whose life time ought to determine the listener’s life time (the MyClass instance). This raises the question, in which way the life times of these objects are connected at all. You have to keep strong references to these key objects, for the sake of keeping these listeners alive, which is error prone too.
I am a fresh new learner of Java. I'm now learning the concept of WeakReference. I came across a problem which probably looks stupid but I just wanna figure out the reason. The problem is: according to Java doc, "Weak reference objects, which do not prevent their referents from being made finalizable, finalized, and then reclaimed."
So I did this small test:
import java.lang.ref.WeakReference;
public class A {
public static void main(String[] args) {
A a = new A();
WeakReference<A> wr = new WeakReference<>(a);
a = null;
A a1 = wr.get();
System.out.println(a);
System.out.println(a1);
try {
System.gc();
Thread.sleep(10000);
} catch (Exception e) {
e.printStackTrace();
}
System.out.println(a1);
}
#Override
protected void finalize( ) {
System.out.println(Thread.currentThread().getName() + ": See ya, nerds!");
}
}
However, I noticed that after GC running, wr.get() could still return object which I expected null, and the method finalize() was not invoked. So what went wrong? Thanks for your kind help in advance! :)
The premise of your test is flawed. System.gc() is only a hint to run the garbage collector. It is frequently ignored.
From the documentation:
Calling the gc method suggests that the Java Virtual Machine
expend effort toward recycling unused objects in order to make the
memory they currently occupy available for quick reuse. When control
returns from the method call, the Java Virtual Machine has made a best
effort to reclaim space from all discarded objects.
(Emphasis mine)
In future, you may use the VM options -verbose:gc and -XX:+PrintGCDetails to see what the garbage collector is doing.
More importantly, you are also very quickly taking the reference out of the weak reference and putting it back into a strong reference:
A a = new A();
WeakReference<A> wr = new WeakReference<>(a);
a = null; // no strong references remain
A a1 = wr.get(); // the instance now has a strong reference again
Unless garbage collection occurs between these exact two instructions, the object will not be garbage collected.
If you remove a1, your code behaved as you would expect when I ran it (though, because of the first part of my answer, your mileage may vary):
class A
{
public static void main(String[] args)
{
A a = new A();
WeakReference<A> wr = new WeakReference<>(a);
a = null;
System.out.println(a);
try {
System.gc(); // instance of A is garbage collected
Thread.sleep(10000);
} catch (Exception e) {
e.printStackTrace();
}
System.out.println(wr.get());
}
#Override
protected void finalize( )
{
System.out.println(Thread.currentThread().getName() + ": See ya, nerds!");
}
}
Firstly, System.gc() does not ensure a garbage collection. Instead, it's just a hint that "It's a good time to run garbage collection".
Secondly, in your code when you put A a1 = wr.get(); before calling System.gc(), it creates a new strong reference to the same object referenced by a, thus even if garbage collection runs, your object will not be garbage collected.
As we have two tasks in hand
Ensure garbage collection
Don't keep any strong reference to the object you want to be garbage collected
Let's do little modification to your code
public class A {
public static void main(String[] args) {
A a = new A();
WeakReference<A> wr = new WeakReference<>(a);
a = null;
// A a1 = wr.get(); Removing this, this does our 2nd task
System.out.println(a);
// System.out.println(a1); Removing this as a1 does not exists anymore
try {
while (null != wr.get()) { // 1st task done, the loop ensures sending the hint until your object collected
System.gc();
// Thread.sleep(10000); it does not have impact
}
} catch (Exception e) {
e.printStackTrace();
}
System.out.println(wr.get()); // Obviously prints null
}
#Override
protected void finalize() {
System.out.println(Thread.currentThread().getName() + ": See ya, nerds!");
}
}
I have class Collector and ThreadLocalScope like this:
Collector {
Collector() {
events = new LinkedList<>();
}
add(Event e) {
events.add(e);
}
flush() {
LinkedList<Event> copy = events;
new Thread(() -> {
for (Event e : copy) {
sendToServer(e);
}
copy.clear();
).start();
events = new LinkedList<>();
}
}
ThreadLocalScope {
public static ThreadLocal<Collector> local = new ThreadLocal<>() {
protected Collector initialValue() {
return new Collector();
}
}
}
Collector simply adds events and when flush is called sends those events to an API in a new thread. The Collector is initialized in a ThreadLocal.
I also have a Job class which is executed several times (using Quartz). When defined like this everything works great:
Job {
execute() {
for (int i = 0; i < 100,000; i++) {
ThreadLocalScope.get().add(new Event());
}
ThreadLocalScope.get().flush();
}
}
However if instead I hold onto Collector like this:
Job {
Collector collector;
Job() {
collector = ThreadLocalScope.get();
}
execute() {
for (int i = 0; i < 100,000; i++) {
collector.add(new Event());
}
collector.flush();
}
}
I see my Old Generation Memory usage increasing rapidly and Stop-the-world Garbage Collection cycles happening all the time. The only difference is I have added Collector as a member variable rather than calling ThreadLocalScope.get() every time.
The increase could only mean that the Events are being moved into Old Generation. But why would that happen? Collector immediately clears all its references to the Events, so even if it is not GCed, the events should be.
I said:
I think you might have a thread-safety issue here.
Incorrect. I think it is simpler than that.
In the first version you are calling ThreadLocalScope.get() in the context of the thread that is executing the job.
In the second version you are calling ThreadLocalScope.get() in the contrext of the thread that is creating the Job() object. It is then squirreled away in a variable and used later in the executor thread. Assuming that the Job() objects are all created on the same thread, that means that your execute() methods share the same Collector object. And they are potentially running on different threads. Since Collector is not thread-safe, that is a hazard.
There is another thing that you might not be aware of. It is likely that Quartz is using a thread pool. That means that when an execute() call terminates, the thread goes back to the pool. Next time around, if Quartz uses the same thread, it will reuse the Collector object from last time.
I have come across this example from wikipedia regarding weak reference:
import java.lang.ref.WeakReference;
public class ReferenceTest {
public static void main(String[] args) throws InterruptedException {
WeakReference r = new WeakReference(new String("I'm here"));
WeakReference sr = new WeakReference("I'm here");
System.out.println("before gc: r=" + r.get() + ", static=" + sr.get());
System.gc();
Thread.sleep(100);
// only r.get() becomes null
System.out.println("after gc: r=" + r.get() + ", static=" + sr.get());
}
}
I don't understand in this scenario why only r.get() returns null but not the sr.get(). Can someone let me know the reason?
Many thanks.
the literal "I'm here" is a compile time constant string and as such gets placed in the constant string pool, which (up until java 7) was never garbage collected. that means sr points to an object that will never be garbage collected.
r, on the other hand, points to a copy of that string, which is not in any const pool and so is eligible for collection.
see the documentation for String.intern() for some more details on this string pool
If this reference object has been cleared, either by the program or by the garbage collector, then this method returns null.
very well explained here
I'm going to use a SoftReference-based cache (a pretty simple thing by itself). However, I've came across a problem when writing a test for it.
The objective of the test is to check if the cache does request the previously cached object from the server again after the memory cleanup occurs.
Here I find the problem how to make system to release soft referenced objects. Calling System.gc() is not enough because soft references will not be released until the memory is low. I'm running this unit test on the PC so the memory budget for the VM could be pretty large.
================== Added later ==============================
Thank you all who took care to answer!
After considering all pro's and contra's I've decided to go the brute force way as advised by nanda and jarnbjo. It appeared, however, that JVM is not that dumb - it won't even attempt garbage collecting if you ask for a block which alone is bigger than VM's memory budget. So I've modified the code like this:
/* Force releasing SoftReferences */
try {
final List<long[]> memhog = new LinkedList<long[]>();
while(true) {
memhog.add(new long[102400]);
}
}
catch(final OutOfMemoryError e) {
/* At this point all SoftReferences have been released - GUARANTEED. */
}
/* continue the test here */
This piece of code forces the JVM to flush all SoftReferences. And it's very fast to do.
It's working better than the Integer.MAX_VALUE approach, since here the JVM really tries to allocate that much memory.
try {
Object[] ignored = new Object[(int) Runtime.getRuntime().maxMemory()];
} catch (OutOfMemoryError e) {
// Ignore
}
I now use this bit of code everywhere I need to unit test code using SoftReferences.
Update: This approach will indeed work only with less than 2G of max memory.
Also, one need to be very careful with SoftReferences. It's so easy to keep a hard reference by mistake that will negate the effect of SoftReferences.
Here is a simple test that shows it working every time on OSX. Would be interested in knowing if JVM's behavior is the same on Linux and Windows.
for (int i = 0; i < 1000; i++) {
SoftReference<Object> softReference = new SoftReferencelt<Object>(new Object());
if (null == softReference.get()) {
throw new IllegalStateException("Reference should NOT be null");
}
try {
Object[] ignored = new Object[(int) Runtime.getRuntime().maxMemory()];
} catch (OutOfMemoryError e) {
// Ignore
}
if (null != softReference.get()) {
throw new IllegalStateException("Reference should be null");
}
System.out.println("It worked!");
}
An improvement that will work for more than 2G max memory. It loops until an OutOfMemory error occurs.
#Test
public void shouldNotHoldReferencesToObject() {
final SoftReference<T> reference = new SoftReference<T>( ... );
// Sanity check
assertThat(reference.get(), not(equalTo(null)));
// Force an OoM
try {
final ArrayList<Object[]> allocations = new ArrayList<Object[]>();
int size;
while( (size = Math.min(Math.abs((int)Runtime.getRuntime().freeMemory()),Integer.MAX_VALUE))>0 )
allocations.add( new Object[size] );
} catch( OutOfMemoryError e ) {
// great!
}
// Verify object has been garbage collected
assertThat(reference.get(), equalTo(null));
}
Set the parameter -Xmx to a very
small value.
Prepare your soft
reference
Create as many object as
possible. Ask for the object everytime until it asked the object from server again.
This is my small test. Modify as your need.
#Test
public void testSoftReference() throws Exception {
Set<Object[]> s = new HashSet<Object[]>();
SoftReference<Object> sr = new SoftReference<Object>(new Object());
int i = 0;
while (true) {
try {
s.add(new Object[1000]);
} catch (OutOfMemoryError e) {
// ignore
}
if (sr.get() == null) {
System.out.println("Soft reference is cleared. Success!");
break;
}
i++;
System.out.println("Soft reference is not yet cleared. Iteration " + i);
}
}
You could explicitly set the soft reference to null in your test, and as such simulate that the soft reference has been released.
This avoids any complicated test setup that is memory and garbage collection dependend.
Instead of a long running loop (as suggested by nanda), it's probably faster and easier to simply create a huge primitive array to allocate more memory than available to the VM, then catch and ignore the OutOfMemoryError:
try {
long[] foo = new long[Integer.MAX_VALUE];
}
catch(OutOfMemoryError e) {
// ignore
}
This will clear all weak and soft references, unless your VM has more than 16GB heap available.