I have found a strange behaviour in my java program here is my code looks like this
public class JavaTest {
private final int dataSize = (int) (Runtime.getRuntime().maxMemory() * 0.6);
public void test() {
{
System.out.println(dataSize);
byte[] data = new byte[dataSize];
}
// for (int i = 0; i < 10; i++) {
// System.out.println("Please be so kind and release memory");
// }
System.out.println(dataSize);
byte[] data2 = new byte[dataSize];
}
public static void main(String[] args) {
JavaTest jmp = new JavaTest();
jmp.test();
}
}
Here when I am commenting the for loop I am getting Exception in thread "main" java.lang.OutOfMemoryError: Java heap space that i can understand the jvm heap sapce is full.
But with that for loop in my code, it executes properly. How comes?
I think it's because you declare byte[] data inside { } block, which means data's scope ends when the code block ends. With loop uncommented, you are probably giving time to garbage collector to free the memory taken by data. And when you comment out the loop, GC doesn't have time to free up that memory yet.
If you remove { } around data declaration, it will also throw OutOfMemoryException even with loop uncommented.
UPDATE
This blog post stated in the comments by #SubOptimal proves this theory wrong, looks like it doesn't have anything to do with time needed by GC to free the memory. I'll quote relevant parts from the blog
The majority of responses were incorrect and suggested that the for() loop either gave the GC time to do its work during the System.out.println()...
Some of my readers realised that it had nothing to do with the System.out.println and that a simple int i = 0; would suffice. If you declare any local variable immediately after the code block, you break the strong reference to the byte[] held in the stack frame 1 before you invoke the new byte[] the second time.
Related
I always thought that a while (true) {...Any code..} would always result in a out of memory error.
But as I go deeper in java it seems it might not be like that.
I'm not able to confirm but if we have a while true that only does calculations, We are not expected to have an out of memory error, only a very detrimental cpu performance, right?
On the other hand if we are always requiring more memory it is expected to have a out of memory error.
I've 3 cases below.
calculations only (I think no memory is being allocated under the hood)
Ever increasing arraylist which it looks an obvious out of memory error
always instanting arraylist with new keyword. I dont know if it causes an out of memory error, because of garbage collector.
I'm not testing im my pc because I only have one, hope someone has the knowledge.
Code
import java.util.*;
public class HelloLeak{
//calculations only, MemoryLeak?
public static void outofmemo1(){
long i = 0;
while (true)
{
i = i * i;
}
}
//adding infinite number of objects, memory leak confirmed.
public static void outofmemo2(){
int i = 0;
List<Integer> l = new ArrayList<>();
while (true)
{
l.add(i);
}
}
//Creating infinite number of ArrayList objects, will garbage collector clear the unused objects or we will get memory leak?
public static void outofmemo3(){
List<Integer> l = new ArrayList<>();
while (true)
{
l = new ArrayList<>();
}
}
public static void main(String []args){
outofmemo1();
//outofmemo2();
//outofmemo3();
}
}
Will do absolutly nothing except ending in an endless loop.
Will crash with an OutOfMemoryError, because you add always a new element to the list, until the heap is filled.
Will be like 1. but you may have spikes up to for example 2GB, then the GC will come, see that there are unused objects, removes them. After that it will spike again, and so on
After a long research , I got to know that String is immutable .String Buffer is more efficient than String if the program involves many computations.
But my question is slightly different from these
I have a function to which I pass a string . The string is actually the text of an article (nearly 3000-5000 charcs) .The function is implemented in threads. I mean to say , there is multiple call of function with different String text each time ..The later stage computations in the functions are too vast . Now when I run my code for a large number of threads, I am getting an error saying : GC Overhead Limit Exceeded . .
Now that I cant reduce the computations in the later stage of functions , my question is will it really help if I change the text type from String to String buffer? Also ,I don’t do any concatenation operation on the text string .
I have posted a small snipet of my code :
public static List<Thread> thread_starter(List<Thread> threads,String filename,ArrayList<String> prop,Logger L,Logger L1,int seq_no)
{ String text="";
if(prop.get(7).matches("txt"))
text=read_contents.read_from_txt(filename,L,L1);
else if(prop.get(7).matches("xml"))
text=read_contents.read_from_xml(filename,L,L1);
else if(prop.get(7).matches("html"))
text=read_contents.read_from_html(filename,L,L1);
else
{
System.out.println("not a valid config");
L1.info("Error : config file not properly defined for i/p file type");
}
/*TODO */
//System.out.println(text);
/*TODO CHANGES TO BE DONE HERE */
if(text.length()>0)
{
Runnable task = new MyRunnable(text,filename,prop,filename,L,L1,seq_no);
Thread worker = new Thread(task);
worker.start();
// Remember the thread for later usage
threads.add(worker);
}
else
{
main_entry_class.file_mover(filename, false);
}
return threads;
}
And i'm calling the above function repeatedly using the following code :
List<Thread> threads = new ArrayList<Thread>();
thread_count=10;
int file_pointer=0;// INTEGER POINTER VARIABLE
do
{
if(file.size()<=file_pointer)
break;
else
{ String file_name=file.get(file_pointer);
threads=thread_starter(threads,file_name,prop,L,L1,seq_no);
file_pointer++;
seq_no++;
}
}while(check_status(threads,thread_count)==true);
And the check status function :
public static boolean check_status(List<Thread> threads,int thread_count)
{
int running = 0;
boolean flag=false;
do {
running = 0;
for (Thread thread : threads) {
if (thread.isAlive()) {
//ThreadMXBean thMxB = ManagementFactory.getThreadMXBean();
//System.out.println(thMxB.getCurrentThreadCpuTime());
running++;
}
}
if(Thread.activeCount()-1<thread_count)
{
flag=true;
break;
}
} while (running > 0);
return flag;
}
If you are getting the error GC Overhead Limit Exceeded then you may try something in between like -Xmx512m first. Also if you have a lot of duplicate strings, you can use String.intern() on them.
You may check this doc:
-XX:+UseConcMarkSweepGC
Check out this link to know what GC Overhead Limit Exceeded error isGC overhead limit exceeded.
As the page suggests, out of memory error occurs when the program spends too much time in garbage collection. So, the problem is not with the number of computations you do...it is with the way you have implemented it. You might have a loop creating too many variables or something like that, so a string buffer might not help you.
How to derive the count of recursion level a recursive function will support without actually executing the function with different complex inputs. For e.g. when I execute below code it displays a level of recursion when the function throws stack overflow error. When I execute the program it displays "Recursion Terminated at 8373".
public class Application {
public static void main(String[] args) throws Exception {
RecursiveExperimenter experimenter = new RecursiveExperimenter();
experimenter.experiment();
}
}
class RecursiveExperimenter {
public void experiment() {
try {
A();
} catch (StackOverflowError e) {
System.out.println("Recursion Terminated at " + counter);
}
}
private void A() {
counter++;
int a = 0, b = 0, c = 0, d = 0;
A();
}
private int counter = 0;
}
How could I have derived that number without actually executing the function but by applying some mathematics? For this questions assume the stack size of the thread to be 1 MB. I am not clear about the overhead of the stack frame itself.
Thanks.
The answer is that this question has no specific answer.
There is no such thing as "the count of recursion level a recursive function will support". Nothing inherent in the concept of recursion imposes any limit on the level of recursion "supported".
In practice, the level of recursion you can achieve depends on the total stack memory available and how much memory each invocation takes up. This will in turn depend on the physical memory limits of the machine, the amount allocated to the JVM, the implementation of the JVM, whether or not the code has been optimized, and probably several other factors. It's even possible for heap memory to be the limiting factor if each invocation allocates large objects on the heap.
In other words, the achievable recursion level for a specific program is 100% dependent on the environment in which the code is run.
I'm going to use a SoftReference-based cache (a pretty simple thing by itself). However, I've came across a problem when writing a test for it.
The objective of the test is to check if the cache does request the previously cached object from the server again after the memory cleanup occurs.
Here I find the problem how to make system to release soft referenced objects. Calling System.gc() is not enough because soft references will not be released until the memory is low. I'm running this unit test on the PC so the memory budget for the VM could be pretty large.
================== Added later ==============================
Thank you all who took care to answer!
After considering all pro's and contra's I've decided to go the brute force way as advised by nanda and jarnbjo. It appeared, however, that JVM is not that dumb - it won't even attempt garbage collecting if you ask for a block which alone is bigger than VM's memory budget. So I've modified the code like this:
/* Force releasing SoftReferences */
try {
final List<long[]> memhog = new LinkedList<long[]>();
while(true) {
memhog.add(new long[102400]);
}
}
catch(final OutOfMemoryError e) {
/* At this point all SoftReferences have been released - GUARANTEED. */
}
/* continue the test here */
This piece of code forces the JVM to flush all SoftReferences. And it's very fast to do.
It's working better than the Integer.MAX_VALUE approach, since here the JVM really tries to allocate that much memory.
try {
Object[] ignored = new Object[(int) Runtime.getRuntime().maxMemory()];
} catch (OutOfMemoryError e) {
// Ignore
}
I now use this bit of code everywhere I need to unit test code using SoftReferences.
Update: This approach will indeed work only with less than 2G of max memory.
Also, one need to be very careful with SoftReferences. It's so easy to keep a hard reference by mistake that will negate the effect of SoftReferences.
Here is a simple test that shows it working every time on OSX. Would be interested in knowing if JVM's behavior is the same on Linux and Windows.
for (int i = 0; i < 1000; i++) {
SoftReference<Object> softReference = new SoftReferencelt<Object>(new Object());
if (null == softReference.get()) {
throw new IllegalStateException("Reference should NOT be null");
}
try {
Object[] ignored = new Object[(int) Runtime.getRuntime().maxMemory()];
} catch (OutOfMemoryError e) {
// Ignore
}
if (null != softReference.get()) {
throw new IllegalStateException("Reference should be null");
}
System.out.println("It worked!");
}
An improvement that will work for more than 2G max memory. It loops until an OutOfMemory error occurs.
#Test
public void shouldNotHoldReferencesToObject() {
final SoftReference<T> reference = new SoftReference<T>( ... );
// Sanity check
assertThat(reference.get(), not(equalTo(null)));
// Force an OoM
try {
final ArrayList<Object[]> allocations = new ArrayList<Object[]>();
int size;
while( (size = Math.min(Math.abs((int)Runtime.getRuntime().freeMemory()),Integer.MAX_VALUE))>0 )
allocations.add( new Object[size] );
} catch( OutOfMemoryError e ) {
// great!
}
// Verify object has been garbage collected
assertThat(reference.get(), equalTo(null));
}
Set the parameter -Xmx to a very
small value.
Prepare your soft
reference
Create as many object as
possible. Ask for the object everytime until it asked the object from server again.
This is my small test. Modify as your need.
#Test
public void testSoftReference() throws Exception {
Set<Object[]> s = new HashSet<Object[]>();
SoftReference<Object> sr = new SoftReference<Object>(new Object());
int i = 0;
while (true) {
try {
s.add(new Object[1000]);
} catch (OutOfMemoryError e) {
// ignore
}
if (sr.get() == null) {
System.out.println("Soft reference is cleared. Success!");
break;
}
i++;
System.out.println("Soft reference is not yet cleared. Iteration " + i);
}
}
You could explicitly set the soft reference to null in your test, and as such simulate that the soft reference has been released.
This avoids any complicated test setup that is memory and garbage collection dependend.
Instead of a long running loop (as suggested by nanda), it's probably faster and easier to simply create a huge primitive array to allocate more memory than available to the VM, then catch and ignore the OutOfMemoryError:
try {
long[] foo = new long[Integer.MAX_VALUE];
}
catch(OutOfMemoryError e) {
// ignore
}
This will clear all weak and soft references, unless your VM has more than 16GB heap available.
Suppose I have following code
package memoryleak;
public class MemoryLeak {
public static int size;
static {
size = (int) (Runtime.getRuntime().maxMemory()*0.6);
}
public static void main(String[] args) throws InterruptedException {
{
byte[] data1 = new byte[size];
}
byte[] data2 = new byte[size];
}
}
This code generates OutOfMemoryError. You can make this code work with one variable allocation (which rewrite stack frame used by first array and make array available for garbage collecting). This puzzle explained here.
{
byte[] data1 = new byte[size];
}
int i = 0;
byte[] data2 = new byte[size];
The question is: why following code still doesn't work?
Object o = new Object();
synchronized (o) {
byte[] data1 = new byte[size];
}
int i = 0;
byte[] data2 = new byte[size];
And following works:
Object o = new Object();
synchronized (o) {
byte[] data1 = new byte[size];
}
int i = 0;
synchronized (o) {
byte[] data2 = new byte[size];
}
My bet is that synchronized adds an element to the frame, causing data1 to move up a slot and not get clobbered by i. synchronized needs to unlock the same object that it locked, even if the local/field changes.
The synchronized code would look something like this:
Object $sync = o;
$sync.lock();
try {
byte[] data1 = new byte[size];
} finally {
$sync.unlock();
}
So taking the last sample of code:
Object o = new Object(); // Slot 0.
synchronized (o) { // Slot 1.
byte[] data1 = new byte[size]; // Slot 2.
}
int i = 0; // Slot 1.
synchronized (o) { // Slot 2. (clobbers data1, was slot 1)
byte[] data2 = new byte[size]; // Slot 3.
}
Puzzles are interesting, but for the pragmatic programmer who does not want to think about (or more importantly depend on) the more arcane aspects of garbage collection, would setting data1 = null as soon as it is no longer needed solve the problem? If so, I'd rather do that then weird synchronized block and dummy variable magic.
Of course, it is sad that the memory does not get freed as soon as the array goes out of scope, which is what people were hoping for in this thread.
This should be fixed in the JVM.
All this behaviour is implementation dependent. The garbage collector runs in its own asynchronous thread that has nothing to do with the synchronization behavior of your program. You simply do not know when the array referenced by data1 will be garbage collected -- you can only hope that it will happen in a "reasonable" amount of time after it goes out of scope/all references to it are gone.
If you are worried about running out of memory in your program, you can explicitly attempt to trigger a garbage collection cycle with System.gc(). But even this does not guarantee that enough memory will be available when you allocate data2. Calling System.gc() is simply a hint to the runtime that you'd like a garbage collection cycle now.
In Java, memory allocation and deallocation is non-deterministic. The garbage collector will run when it runs and you can't make it run at the program level. There's no relevant differences between the code snippets you posted, because gc behaviour is non-deterministic and the precise moment at which it is triggered is implementation and system dependent. Sometimes this is a problem for your application -- if it's an OS or runs in a memory constrained embedded device, for example -- and you will need to code in C++ or some other language where memory management is deterministic. For most of us, though, we just trust that the garbage collector will behave in a reasonably reasonable manner and that is good enough for most purposes -- although, as you see, you can create contrived code that causes problems.
Update: Embarrassing. As a couple of other commenters have reminded me, a garbage collection cycle is explicitly triggered before the jvm throws an OutOfMemory error. However, the behaviour is still non-deterministic: as this link explains, the jvm does not guarantee that all dead objects will be detected in one garbage collection cycle.
You're relying on the GC to collect before your instantiation?
couldn't you do
Object o = new Object();
byte[] data1 = new byte[size];
GC.Collect()
byte[] data2 = new byte[size];