We Keep Coding

Dropbox "java.lang.outofmemoryerror java heap space" when trying to upload a large file [duplicate]

I am getting the following error on execution of a multi-threading program
java.lang.OutOfMemoryError: Java heap space
The above error occured in one of the threads.
Upto my knowledge, Heap space is occupied by instance variables only. If this is correct, then why this error occurred after running fine for sometime as space for instance variables are alloted at the time of object creation.
Is there any way to increase the heap space?
What changes should I made to my program so that It will grab less heap space?

If you want to increase your heap space, you can use java -Xms<initial heap size> -Xmx<maximum heap size> on the command line. By default, the values are based on the JRE version and system configuration. You can find out more about the VM options on the Java website.
However, I would recommend profiling your application to find out why your heap size is being eaten. NetBeans has a very good profiler included with it. I believe it uses the jvisualvm under the hood. With a profiler, you can try to find where many objects are being created, when objects get garbage collected, and more.

1.- Yes, but it pretty much refers to the whole memory used by your program.
2.- Yes see Java VM options
-Xms<size> set initial Java heap size
-Xmx<size> set maximum Java heap size
Ie
java -Xmx2g assign 2 gigabytes of ram as maximum to your app
But you should see if you don't have a memory leak first.
3.- It depends on the program. Try spot memory leaks. This question would be to hard to answer. Lately you can profile using JConsole to try to find out where your memory is going to

You may want to look at this site to learn more about memory in the JVM:
http://developer.streamezzo.com/content/learn/articles/optimization-heap-memory-usage
I have found it useful to use visualgc to watch how the different parts of the memory model is filling up, to determine what to change.
It is difficult to determine which part of memory was filled up, hence visualgc, as you may want to just change the part that is having a problem, rather than just say,
Fine! I will give 1G of RAM to the JVM.
Try to be more precise about what you are doing, in the long run you will probably find the program better for it.
To determine where the memory leak may be you can use unit tests for that, by testing what was the memory before the test, and after, and if there is too big a change then you may want to examine it, but, you need to do the check while your test is still running.

You can get your heap memory size through below programe.
public class GetHeapSize {
public static void main(String[] args) {
long heapsize = Runtime.getRuntime().totalMemory();
System.out.println("heapsize is :: " + heapsize);
}
}
then accordingly you can increase heap size also by using:
java -Xmx2g
http://www.oracle.com/technetwork/java/javase/tech/vmoptions-jsp-140102.html

To increase the heap size you can use the -Xmx argument when starting Java; e.g.
-Xmx256M

Upto my knowledge, Heap space is occupied by instance variables only. If this is correct, then why this error occurred after running fine for sometime as space for instance variables are alloted at the time of object creation.
That means you are creating more objects in your application over a period of time continuously. New objects will be stored in heap memory and that's the reason for growth in heap memory.
Heap not only contains instance variables. It will store all non-primitive data types ( Objects). These objects life time may be short (method block) or long (till the object is referenced in your application)
Is there any way to increase the heap space?
Yes. Have a look at this oracle article for more details.
There are two parameters for setting the heap size:
-Xms:, which sets the initial and minimum heap size
-Xmx:, which sets the maximum heap size
What changes should I made to my program so that It will grab less heap space?
It depends on your application.
Set the maximum heap memory as per your application requirement
Don't cause memory leaks in your application
If you find memory leaks in your application, find the root cause with help of profiling tools like MAT, Visual VM , jconsole etc. Once you find the root cause, fix the leaks.
Important notes from oracle article
Cause: The detail message Java heap space indicates object could not be allocated in the Java heap. This error does not necessarily imply a memory leak.
Possible reasons:
Improper configuration ( not allocating sufficiant memory)
Application is unintentionally holding references to objects and this prevents the objects from being garbage collected
Applications that make excessive use of finalizers. If a class has a finalize method, then objects of that type do not have their space reclaimed at garbage collection time. If the finalizer thread cannot keep up, with the finalization queue, then the Java heap could fill up and this type of OutOfMemoryError exception would be thrown.
On a different note, use better Garbage collection algorithms ( CMS or G1GC)
Have a look at this question for understanding G1GC

In most of the cases, the code is not optimized. Release those objects which you think shall not be needed further. Avoid creation of objects in your loop each time. Try to use caches. I don't know how your application is doing. But In programming, one rule of normal life applies as well
Prevention is better than cure. "Don't create unnecessary objects"

Local variables are located on the stack. Heap space is occupied by objects.
You can use the -Xmx option.
Basically heap space is used up everytime you allocate a new object with new and freed some time after the object is no longer referenced. So make sure that you don't keep references to objects that you no longer need.

No, I think you are thinking of stack space. Heap space is occupied by objects. The way to increase it is -Xmx256m, replacing the 256 with the amount you need on the command line.

To avoid that exception, if you are using JUnit and Spring try adding this in every test class:
#DirtiesContext(classMode = DirtiesContext.ClassMode.AFTER_CLASS)

I have tried all Solutions but nothing worked from above solutions
Solution: In My case I was using 4GB RAM and due to that RAM usage comes out 98% so the required amount if Memory wasn't available. Please do look for this also.If such issue comes upgrade RAM and it will work fine.
Hope this will save someone Time

In netbeans, Go to 'Run' toolbar, --> 'Set Project Configuration' --> 'Customise' --> 'run' of its popped up windo --> 'VM Option' --> fill in '-Xms2048m -Xmx2048m'. It could solve heap size problem.

Why I am getting java.lang.OutOfMemoryError?

I am getting:
java.lang.OutOfMemoryError : Java heap space
Caused by: java.lang.OutOfMemoryError: Java heap space
at java.util.Arrays.copyOf(Arrays.java:2894)
at java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:117)
at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:407)
at java.lang.StringBuilder.append(StringBuilder.java:136)

ltimately you always have a finite max of heap to use no matter what platform you are running on. In Windows 32 bit this is around 2gb (not specifically heap but total amount of memory per process). It just happens that Java happens to make the default smaller (presumably so that the programmer can't create programs that have runaway memory allocation without running into this problem and having to examine exactly what they are doing).
So this given there are several approaches you could take to either determine what amount of memory you need or to reduce the amount of memory you are using. One common mistake with garbage collected languages such as Java or C# is to keep around references to objects that you no longer are using, or allocating many objects when you could reuse them instead. As long as objects have a reference to them they will continue to use heap space as the garbage collector will not delete them.
In this case you can use a Java memory profiler to determine what methods in your program are allocating large number of objects and then determine if there is a way to make sure they are no longer referenced, or to not allocate them in the first place. One option which I have used in the past is "JMP" http://www.khelekore.org/jmp/.
If you determine that you are allocating these objects for a reason and you need to keep around references (depending on what you are doing this might be the case), you will just need to increase the max heap size when you start the program. However, once you do the memory profiling and understand how your objects are getting allocated you should have a better idea about how much memory you need.
In general if you can't guarantee that your program will run in some finite amount of memory (perhaps depending on input size) you will always run into this problem. Only after exhausting all of this will you need to look into caching objects out to disk etc. At this point you should have a very good reason to say "I need Xgb of memory" for something and you can't work around it by improving your algorithms or memory allocation patterns. Generally this will only usually be the case for algorithms operating on large datasets (like a database or some scientific analysis program) and then techniques like caching and memory mapped IO become useful.

The OutOfMemoryError is usually caused by the VM not having enough memory to run your project. Did you run it directly from the command line or did you use an IDE ?
For example, Try running your programm with adding the -Xmx1G option which allocate 1Go of memory heap to your programm, you can of course adjust it to your convenience. the G is for Go and the m is for Mb.

You should give the heap a bigger size for it to work.

Increasing Heap Size upto 6GB

I was looking at the following article:
Increase heap size in Java
Now I have a program that needs about 5GB memory and while doing what was told in the article (that is increasing heap size by using -Xmx5g in the arguments field), I am still getting
Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
My system is Windows 7 (64 bit) with 8GB RAM. Am I doing something wrong? If yes, how shall I proceed to get 5GB of heap memory or it is just not feasible for my system to handle?
Note: I have to do calculations with a 2D matrix that is of 25K*25K size having all non-zero values. Hence I cannot use sparse matrix as well.

OutOfMemoryError is thrown when JVM does not have enough memory for objects being allocated. If you defined heap of 5G this almost definitely mean that you have a kind of memory leak. For example I can write very simple code that will cause OutOfMemoryError at any environment:
List<String> list = new LinkedList<>();
while(true) {
list.add("a");
}
Run this code and wait several seconds. OutOfMemoryError will be thrown. This is because I add strings to list and never clean it.
I believe that something similar happens with your application.
I understand, that it is not so trivial as my example, so you will probably have to use profiler to debug it and understand the reason of your memory leak.
EDIT:
I've just saw that you are working with 25K*25K martrix. It means that you have 625M cells. You have not mentioned the type of the matrix but if it is int that occupies 4 bytes you need 625*4=2500M=2.5G memory, so 5G should be enough.
Please try to analyze what else happens in your program and where your memory is spent.

5G/(25K*25K) ~ 8 bytes.
Generously assuming that you program does not use memory except for that matrix, each matrix element must take no more than 8 bytes.
You should calculate at least approximate memory requirements to check whether it is even possible to handle problem of such size on your hardware. For example, if you need a 2D array of MxN size of double values then you need at least 8*M*N bytes of memory.

Obtaining memory available to JVM at runtime

I'm trying to sort a bunch of data such that that the size of data input to the program can be larger than the memory available to the JVM, and handling that requires external sort which is much slower than Quicksort.
Is there any way of obtaining memory available to the JVM at runtime such that I could use in place sorting as much as possible, and only switch to Mergesort when data input is too large?

Check out these methods on the java.lang.Runtime class:
freeMemory
totalMemory
maxMemory
Example
Runtime rt = Runtime.getRuntime();
System.err.println(String.format("Free: %d bytes, Total: %d bytes, Max: %d bytes",
rt.freeMemory(), rt.totalMemory(), rt.maxMemory()));
Also note that if the total amount of memory is being exhausted you can always start the JVM with a larger amount of heap allocated using the -Xmx JVM argument; e.g.
java -Xmx256M MyClass

You can use the Runtime class to get the amount of memory available.
Runtime r = Runtime.getRuntime();
System.out.println(r.totalMemory());
There are various other memory details you can get from the Runtime object - see Runtime class.

In theory yes, using Runtime.getRuntime().maxMemory().
In practice, there are some problem you need to address:
You need to figure out how many application objects are going to fit in a given number of bytes of memory. AFAIK, there is no simple / efficient way to do this within a running application.
You don't want to try to use all available heap space. If you push your percentage heap residency too high, you risk making the GC horribly inefficient.
The maxMemory() method only tells you how big the heap in virtual memory. The physical size can also be a factor (especially if physical size << virtual size), and there's no portable way to figure that out.
If I was trying to implement this application, I think I'd probably just make the in-memory sort size a configuration parameter or command-line option.

Using the methods of Runtime, as the others suggested, is fine, as long as you take some things into consideration:
1) freeMemory() is a lower bound on the actual available memory, because memory that is unreferenced and ready for GC is considered as used. Running System.gc() before the call may return a more accurate result.
2) totalMemory() can change - it only indicates the current total heap size, and the heap can expand/shrink by the JVM during runtime, depending on its usage. You can use maxMemory() to get the actual maximum.

How to set the maximum memory usage for JVM?

I want to limit the maximum memory used by the JVM. Note, this is not just the heap, I want to limit the total memory used by this process.

use the arguments -Xms<memory> -Xmx<memory>. Use M or G after the numbers for indicating Megs and Gigs of bytes respectively. -Xms indicates the minimum and -Xmx the maximum.

You shouldn't have to worry about the stack leaking memory (it is highly uncommon). The only time you can have the stack get out of control is with infinite (or really deep) recursion.
This is just the heap. Sorry, didn't read your question fully at first.
You need to run the JVM with the following command line argument.
-Xmx<ammount of memory>
Example:
-Xmx1024m
That will allow a max of 1GB of memory for the JVM.

If you want to limit memory for jvm (not the heap size )
ulimit -v
To get an idea of the difference between jvm and heap memory , take a look at this excellent article
http://blogs.vmware.com/apps/2011/06/taking-a-closer-look-at-sizing-the-java-process.html

The answer above is kind of correct, you can't gracefully control how much native memory a java process allocates. It depends on what your application is doing.
That said, depending on platform, you may be able to do use some mechanism, ulimit for example, to limit the size of a java or any other process.
Just don't expect it to fail gracefully if it hits that limit. Native memory allocation failures are much harder to handle than allocation failures on the java heap. There's a fairly good chance the application will crash but depending on how critical it is to the system to keep the process size down that might still suit you.

The NativeHeap can be increasded by -XX:MaxDirectMemorySize=256M
(default is 128)
I've never used it. Maybe you'll find it useful.