While investigating Why ThreadPoolExecutor behaves differently when running Java program in Eclipse and from command line? I wrote a test that throws a very strange OutOfMemoryError (max mem = 256 Mb)
class A {
byte[] buf = new byte[150_000_000];
protected void finalize() {
int i = 1;
}
}
A a1 = new A();
a1 = null;
A a2 = new A();
comment out int i = 1 and the test works. As far as I understand when finalize is empty HotSpot simply ignores it. But how can just one practically empty finalize invocation break GC / JVM?
But how can just one empty finalize invocation break GC / JVM?
When there's a finalizer, objects survive one more round of garbage collection than they would otherwise (as the object itself has to be kept alive until it's finalized). Therefore if you have a large object with a finalizer, that will naturally lead to an OutOfMemoryError occurring in situations when it wouldn't without a finalizer.
In this code:
A a1 = new A();
a1 = null;
A a2 = new A();
... the GC will trigger on the last line in order to try to find enough memory to allocate the second A. Unfortunately, it can't garbage collect the first A (and the array it refers to) because the finalizer hasn't run yet. It doesn't wait until the finalizer completes, then try to garbage collect again - it just throws OutOfMemoryError.
Related
I am new to handling the memory leak situations, but one thing that I have noticed is that all the examples showing memory leaks have the activity contexts on a different thread.
So I need to know if a memory leak can happen if there is an object reference on the same thread as well, because the activity reference is stored somewhere in other classes.
Thanks in advance!
A Memory Leak is a situation when there are objects present in the heap that are no longer used, but the garbage collector is unable to remove them from memory and, thus they are unnecessarily maintained.
Memory leaks can happend in the same thread as well. For example if a method stored data in a static variable which it does need to refer in the subsequent call.
E.g: In the code below we are storing numbers generates in a static list even though we do not require those generated numbers in subsequent calls.
public class MemoryLeak{
public static List<Double> list = new ArrayList<>();
public void doSomething() {
for (int i = 0; i < 10000000; i++) {
list.add(Math.random());
}
Log.info("Debug Point 2");
}
public static void main(String[] args) {
Log.info("Debug Point 1");
new MemoryLeak().doSomething();
Log.info("Debug Point 3");
}
}
I have found a strange behaviour in my java program here is my code looks like this
public class JavaTest {
private final int dataSize = (int) (Runtime.getRuntime().maxMemory() * 0.6);
public void test() {
{
System.out.println(dataSize);
byte[] data = new byte[dataSize];
}
// for (int i = 0; i < 10; i++) {
// System.out.println("Please be so kind and release memory");
// }
System.out.println(dataSize);
byte[] data2 = new byte[dataSize];
}
public static void main(String[] args) {
JavaTest jmp = new JavaTest();
jmp.test();
}
}
Here when I am commenting the for loop I am getting Exception in thread "main" java.lang.OutOfMemoryError: Java heap space that i can understand the jvm heap sapce is full.
But with that for loop in my code, it executes properly. How comes?
I think it's because you declare byte[] data inside { } block, which means data's scope ends when the code block ends. With loop uncommented, you are probably giving time to garbage collector to free the memory taken by data. And when you comment out the loop, GC doesn't have time to free up that memory yet.
If you remove { } around data declaration, it will also throw OutOfMemoryException even with loop uncommented.
UPDATE
This blog post stated in the comments by #SubOptimal proves this theory wrong, looks like it doesn't have anything to do with time needed by GC to free the memory. I'll quote relevant parts from the blog
The majority of responses were incorrect and suggested that the for() loop either gave the GC time to do its work during the System.out.println()...
Some of my readers realised that it had nothing to do with the System.out.println and that a simple int i = 0; would suffice. If you declare any local variable immediately after the code block, you break the strong reference to the byte[] held in the stack frame 1 before you invoke the new byte[] the second time.
what will the Finalizer thread do if there is a infinite loop or deadlock in the Java finalize method.
The spec writes:
Before the storage for an object is reclaimed by the garbage collector, the Java Virtual Machine will invoke the finalizer of that object.
The Java programming language does not specify how soon a finalizer will be invoked, except to say that it will happen before the storage for the object is reused.
I read this to mean that the finalizer must have completed before the storage may be reused.
The Java programming language does not specify which thread will invoke the finalizer for any given object.
It is important to note that many finalizer threads may be active (this is sometimes needed on large shared memory multiprocessors), and that if a large connected data structure becomes garbage, all of the finalize methods for every object in that data structure could be invoked at the same time, each finalizer invocation running in a different thread.
That is, finalization may occur in the garbage collector thread, in a separate thead, or even a separate thread pool.
A JVM is not permitted to simply abort executing a finalizer, and can only use a finite number of threads (threads are operating system resources, and operating systems don't support arbitrarily many threads). Non-terminating finalizers will therefore of necessity starve that thread pool, thereby inhibit collection of any finalizable objects, and cause a memory leak.
The following test program confirms this behavior:
public class Test {
byte[] memoryHog = new byte[1024 * 1024];
#Override
protected void finalize() throws Throwable {
System.out.println("Finalizing " + this + " in thread " + Thread.currentThread());
for (;;);
}
public static void main(String[] args) {
for (int i = 0; i < 1000; i++) {
new Test();
}
}
}
On Oracle JDK 7, this prints:
Finalizing tools.Test#1f1fba0 in thread Thread[Finalizer,8,system]
Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
at tools.Test.<init>(Test.java:5)
at tools.Test.main(Test.java:15)
I would say that since the Java Specification doesn't tell how the finalize method must be invoked (just that it must be invoked, before the object is garbage collected), the behaviour is implementation specific.
The spec doesn't rule out having multiple threads running the process, but doesn't require it:
It is important to note that many finalizer threads may be active
(this is sometimes needed on large shared memory multiprocessors), and
that if a large connected data structure becomes garbage, all of the
finalize methods for every object in that data structure could be
invoked at the same time, each finalizer invocation running in a
different thread.
Looking at the sources of the JDK7, the FinalizerThread keeps the queue of objects scheduled for finalization (actually objects are added to the queue by the GC, when proven to be unreachable - check ReferenceQueue doc):
private static class FinalizerThread extends Thread {
private volatile boolean running;
FinalizerThread(ThreadGroup g) {
super(g, "Finalizer");
}
public void run() {
if (running)
return;
running = true;
for (;;) {
try {
Finalizer f = (Finalizer)queue.remove();
f.runFinalizer();
} catch (InterruptedException x) {
continue;
}
}
}
}
Each object is removed from the queue, and runFinalizer method is run on it. Check is done if the finalization had run on the object, and if not it is being invoked, as a call to a native method invokeFinalizeMethod. The method simply is calling the finalize method on the object:
JNIEXPORT void JNICALL
Java_java_lang_ref_Finalizer_invokeFinalizeMethod(JNIEnv *env, jclass clazz,
jobject ob)
{
jclass cls;
jmethodID mid;
cls = (*env)->GetObjectClass(env, ob);
if (cls == NULL) return;
mid = (*env)->GetMethodID(env, cls, "finalize", "()V");
if (mid == NULL) return;
(*env)->CallVoidMethod(env, ob, mid);
}
This should lead to a situation, where the objects get queued in the list, while the FinalizerThread is blocked on the faulty object, which in turn should lead to OutOfMemoryError.
So to answer the original question:
what will the Finalizer thread do if there is a infinite loop or deadlock in the Java finalize method.
It will simply sit there and run that infinite loop until OutOfMemoryError.
public class FinalizeLoop {
public static void main(String[] args) {
Thread thread = new Thread() {
#Override
public void run() {
for (;;) {
new FinalizeLoop();
}
}
};
thread.setDaemon(true);
thread.start();
while (true);
}
#Override
protected void finalize() throws Throwable {
super.finalize();
System.out.println("Finalize called");
while (true);
}
}
Note the "Finalize called" if printed only once on the JDK6 and JDK7.
The objects will not be "freed", that is the memory will not be claimed back from them and also resources that are freed in the finalize method will remain reserved throughout.
Basically there is a queue holding all the objects waiting for their finalize() method to be executed. Finalizer thread picks up objects from this queue - runs finalize - and releases the object.
If this thread will be deadlocked the ReferenceQueue Queue will grow up and at some point OOM error will become inexorable. Also the resources will be hogged up by the objects in this queue. Hope this helps!!
for(;;)
{
Finalizer f = java.lang.ref.Finalizer.ReferenceQueue.remove();
f.get().finalize();
}
I build a sample program demonstrate memory leak in java.
public class MemoryLeakTest {
static int depth = 0;
int number=0;
MemoryLeakTest mobj;
MemoryLeakTest(){
number = depth;
if(depth < 6500){
depth++;
mobj = new MemoryLeakTest();
}
}
protected void finalize(){
System.out.println(number + " released.");
}
public static void main(String[] args) {
try{
System.out.println(ManagementFactory.getMemoryMXBean().getHeapMemoryUsage());
System.out.println("Free Memory in starting "+ Runtime.getRuntime().freeMemory());
MemoryLeakTest testObj = new MemoryLeakTest();
System.out.println("Free Memory in end "+ Runtime.getRuntime().freeMemory());
System.out.println(ManagementFactory.getMemoryMXBean().getHeapMemoryUsage());
}
catch(Exception exp){}
finally{
System.out.println("Free Memory"+ Runtime.getRuntime().freeMemory());
System.out.println(ManagementFactory.getMemoryMXBean().getHeapMemoryUsage());
}
}
}
I run it by changing value of N in if(depth < N). An here is the result;
when depth is 1000
init = 16777216(16384K) used = 288808(282K) committed = 16252928(15872K) max = 259522560(253440K)
Free Memory in starting 15964120
Free Memory in end 15964120
init = 16777216(16384K) used = 288808(282K) committed = 16252928(15872K) max = 259522560(253440K)
Free Memory 15964120
init = 16777216(16384K) used = 288808(282K) committed = 16252928(15872K) max = 259522560(253440K)
when depth is 1500
init = 16777216(16384K) used = 288808(282K) committed = 16252928(15872K) max = 259522560(253440K)
Free Memory in starting 15964120
Free Memory in end 15964120
init = 16777216(16384K) used = 288808(282K) committed = 16252928(15872K) max = 259522560(253440K)
Free Memory 15873528
init = 16777216(16384K) used = 379400(370K) committed = 16252928(15872K) max = 259522560(253440K)
when depth is 6000
init = 16777216(16384K) used = 288808(282K) committed = 16252928(15872K) max = 259522560(253440K)
Free Memory in starting 15964120
Free Memory in end 15692784
init = 16777216(16384K) used = 560144(547K) committed = 16252928(15872K) max = 259522560(253440K)
Free Memory 15692784
init = 16777216(16384K) used = 560144(547K) committed = 16252928(15872K) max = 259522560(253440K)
when depth is 6500 (Exception in thread "main" java.lang.StackOverflowError)
init = 16777216(16384K) used = 288808(282K) committed = 16252928(15872K) max = 259522560(253440K)
Free Memory in starting 15964120
Free Memory in end 15676656
init = 16777216(16384K) used = 576272(562K) committed = 16252928(15872K) max = 259522560(253440K)
My questions are;
It is not calling finalize(). Is it memory leak?
There is not change in free memory up to N=1000. But when N=1500 there is 2 different
values for used memory at the end of the program ie 282K and 370K.
Why does it so?
When N=6500, JVM generates error. So why last 2
statements of try{} are executed.
Your program won't "leak" as Java will take care of anything "dangling" out there. That's the benefit of a garbage-collected language.
But what you do have is a StackOverFlow error. Basically, the stack (which is the chain of functions you're in, and how deep that is) is much MUCH smaller than the heap. The heap is "more or less" the size of main memory. Each thread's stack is much much smaller. Basically you're reaching that limit by doing your "Depth" thing.
If you want to test "leaks" (or the idea that you won't have any eventually) try something more like this:
public class MemoryLeakTest {
int number=0;
public MemoryLeakTest mobj;
MemoryLeakTest(int num){
number = num;
}
protected void finalize(){
System.out.println(number + " released.");
}
public static void main(String[] args) {
try{
System.out.println(ManagementFactory.getMemoryMXBean().getHeapMemoryUsage());
System.out.println("Free Memory in starting "+ Runtime.getRuntime().freeMemory());
MemoryLeakTest first = new MemoryLeakTest(0); // Keep a reference to one of them
MemoryLeakTest current = first;
for(int i = 1; i < Int.Parse(args[0]); i++) // forgive me, Java's been a while. This may be C#. But parse the first arg for your number of objects
{
current.mobj = new MemoryLeakTest(i);
current = current.mobj;
}
System.out.println("Free Memory in end "+ Runtime.getRuntime().freeMemory());
System.out.println(ManagementFactory.getMemoryMXBean().getHeapMemoryUsage());
}
catch(Exception exp){}
finally{
System.out.println("Free Memory"+ Runtime.getRuntime().freeMemory());
System.out.println(ManagementFactory.getMemoryMXBean().getHeapMemoryUsage());
}
}
}
That will give you a "chain" of objects all in memory until first goes out of scope.
It is not calling finalize(). Is it memory leak?
No there is no memory leak you always keep accessible reference to your testObj object
and that's the reason why finalizewill never be called in your application.
All you do in your application is to create a huge object graph.
Here you can find an explanation how to create a real memory leak in java.
It is not calling finalize(). Is it memory leak?
Finalize is not guaranteed to be called, it is called when the garbage collector collects the given object but the objects are not guaranteed to be collected before the execution ends.
There is not change in free memory up to N=1000. But when N=1500 there is 2 different values >for used memory at the end of the program ie 282K and 370K. Why does it so?
I think it depends on the execution of the garbage collector and the moments that it gets executed.
When N=6500, JVM generates error. So why last 2 statements of try{} are executed.
This is because you're not catching the exception since StackOverflowError inherits from Error that is not part of the Exception inheritance branch but rather is a brother of Exception, anyway you have no code in the catch, the last two methods of your try are not being executed because the exception has been thrown.
In summary you didn't produce a memory leak, memory leaks happen in java when you have references to objects that are reachable (directly or indirectly) from the execution flow at some point, for example you store objects in a collection that you can reach, or singletons.
The garbage collector itself is smart enough to free object graphs that are not reachable from the program at all.
Hope I could make it clear.
Already most of the answers explained difference between StackOverflowError and memory leak.
There is not change in free memory up to N=1000. But when N=1500 there is 2 different values for used memory at the end of the program ie 282K and 370K. Why does it so?
it is because every time you create new object and previous obj become unreachable(no references, overriding reference) and hence can be freed if required.
So far simplest example to make jvm run out of memory (not leak).
public class PrintSeries {
private static String COMMA = ",";
private StringBuilder buildStream;// = new StringBuilder();
public static void main(String[] args) {
System.out.println(new PrintSeries().convert(10));
System.out.println(new PrintSeries().convert(1000000000));
}
private String convert(int n) {
buildStream = new StringBuilder();
while (n > 1) {
buildStream.append(n-- + COMMA);
}
buildStream.append(n);
return buildStream.toString();
}
}
output
10,9,8,7,6,5,4,3,2,1
Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
at java.util.Arrays.copyOf(Arrays.java:2882)
at java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:100)
at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:390)
at java.lang.StringBuilder.append(StringBuilder.java:119)
at com.cctest.algotest.string.PrintSeries.convert(PrintSeries.java:17)
at com.cctest.algotest.string.PrintSeries.main(PrintSeries.java:10)
This is not evidence of a memory leak. The program is throwing StackOverflowError not OutOfMemoryError. In fact, that is going on is that the constructor is calling itself recursively, and when the number of recursive calls exceeds some large number (between 6,000 and 6,500), you run out of stack space.
It is not calling finalize(). Is it memory leak?
No. The finalize() method is most likely not being called because the GC has not run. And it has not run because you haven't filled the heap. And even if that is not the real explanation, there is no guarantee that the finalize() method will ever be called. The only absolute guarantee you have is that finalize() will be called before the object's memory is reused by the JVM.
There is not change in free memory up to N=1000. But when N=1500 there is 2 different values for used memory at the end of the program ie 282K and 370K. Why does it so?
I'm not sure why that happens, but I don't think it indicates anything significant. (There are all sorts of things that happen under the hood in a JVM that can be sources of non-determinacy in things like memory allocation and usage patterns.)
When N=6500, JVM generates error. So why last 2 statements of try{} are executed.
The statements in the finally is always executed, unless the JVM terminates abruptly. When the StackOverflowError is thrown, it propagates like any other exception, and can be caught and recovered from (in some cases).
Suppose I have following code
package memoryleak;
public class MemoryLeak {
public static int size;
static {
size = (int) (Runtime.getRuntime().maxMemory()*0.6);
}
public static void main(String[] args) throws InterruptedException {
{
byte[] data1 = new byte[size];
}
byte[] data2 = new byte[size];
}
}
This code generates OutOfMemoryError. You can make this code work with one variable allocation (which rewrite stack frame used by first array and make array available for garbage collecting). This puzzle explained here.
{
byte[] data1 = new byte[size];
}
int i = 0;
byte[] data2 = new byte[size];
The question is: why following code still doesn't work?
Object o = new Object();
synchronized (o) {
byte[] data1 = new byte[size];
}
int i = 0;
byte[] data2 = new byte[size];
And following works:
Object o = new Object();
synchronized (o) {
byte[] data1 = new byte[size];
}
int i = 0;
synchronized (o) {
byte[] data2 = new byte[size];
}
My bet is that synchronized adds an element to the frame, causing data1 to move up a slot and not get clobbered by i. synchronized needs to unlock the same object that it locked, even if the local/field changes.
The synchronized code would look something like this:
Object $sync = o;
$sync.lock();
try {
byte[] data1 = new byte[size];
} finally {
$sync.unlock();
}
So taking the last sample of code:
Object o = new Object(); // Slot 0.
synchronized (o) { // Slot 1.
byte[] data1 = new byte[size]; // Slot 2.
}
int i = 0; // Slot 1.
synchronized (o) { // Slot 2. (clobbers data1, was slot 1)
byte[] data2 = new byte[size]; // Slot 3.
}
Puzzles are interesting, but for the pragmatic programmer who does not want to think about (or more importantly depend on) the more arcane aspects of garbage collection, would setting data1 = null as soon as it is no longer needed solve the problem? If so, I'd rather do that then weird synchronized block and dummy variable magic.
Of course, it is sad that the memory does not get freed as soon as the array goes out of scope, which is what people were hoping for in this thread.
This should be fixed in the JVM.
All this behaviour is implementation dependent. The garbage collector runs in its own asynchronous thread that has nothing to do with the synchronization behavior of your program. You simply do not know when the array referenced by data1 will be garbage collected -- you can only hope that it will happen in a "reasonable" amount of time after it goes out of scope/all references to it are gone.
If you are worried about running out of memory in your program, you can explicitly attempt to trigger a garbage collection cycle with System.gc(). But even this does not guarantee that enough memory will be available when you allocate data2. Calling System.gc() is simply a hint to the runtime that you'd like a garbage collection cycle now.
In Java, memory allocation and deallocation is non-deterministic. The garbage collector will run when it runs and you can't make it run at the program level. There's no relevant differences between the code snippets you posted, because gc behaviour is non-deterministic and the precise moment at which it is triggered is implementation and system dependent. Sometimes this is a problem for your application -- if it's an OS or runs in a memory constrained embedded device, for example -- and you will need to code in C++ or some other language where memory management is deterministic. For most of us, though, we just trust that the garbage collector will behave in a reasonably reasonable manner and that is good enough for most purposes -- although, as you see, you can create contrived code that causes problems.
Update: Embarrassing. As a couple of other commenters have reminded me, a garbage collection cycle is explicitly triggered before the jvm throws an OutOfMemory error. However, the behaviour is still non-deterministic: as this link explains, the jvm does not guarantee that all dead objects will be detected in one garbage collection cycle.
You're relying on the GC to collect before your instantiation?
couldn't you do
Object o = new Object();
byte[] data1 = new byte[size];
GC.Collect()
byte[] data2 = new byte[size];