We Keep Coding

Java running out of memory on VPS

The program has no apparent memory leaks and while I am observing it run the locally in my machine it works fine. On the VPS it crashes after few hours with a sequence of error messages as shown below.
Exception in thread "Thread-10422" java.lang.OutOfMemoryError: Java heap space
I don't understand why such an error would occur after hours instead of few minutes if there is a memory leak. I used tools such as VisualVM to observe the behavior of the program and the memory runs constant throughout.
Is anyone aware of any ways I can debug this and get to the bottom of this or how to avoid it?
Is there a tool that requires no installation and can observe the memory usage of a process over ssh?
Edit:
There is no stack trace on all the exceptions which is weird. But the error happens in different threads for different classes.
at java.io.BufferedWriter.<init>(BufferedWriter.java:104)
at java.io.BufferedWriter.<init>(BufferedWriter.java:87)
at java.io.PrintStream.init(PrintStream.java:100)
at java.io.PrintStream.<init>(PrintStream.java:142)
at sun.net.www.http.HttpClient.openServer(HttpClient.java:411)
at sun.net.www.http.HttpClient$2.run(HttpClient.java:457)
at java.security.AccessController.doPrivileged(Native Method)
at sun.net.www.http.HttpClient.privilegedOpenServer(HttpClient.java:454)
at sun.net.www.http.HttpClient.openServer(HttpClient.java:521)
at sun.net.www.http.HttpClient.<init>(HttpClient.java:240)
at sun.net.www.http.HttpClient.New(HttpClient.java:321)
at sun.net.www.http.HttpClient.New(HttpClient.java:338)
at sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:935)
at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:914)
at sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:801)
at java.util.HashMap.resize(HashMap.java:479)
at java.util.LinkedHashMap.addEntry(LinkedHashMap.java:431)
at java.util.HashMap.put(HashMap.java:402)
at org.jsoup.nodes.Attributes.put(Attributes.java:58)
at org.jsoup.parser.Token$Tag.newAttribute(Token.java:65)
at org.jsoup.parser.TokeniserState$34.read(TokeniserState.java:791)
at org.jsoup.parser.Tokeniser.read(Tokeniser.java:42)
at org.jsoup.parser.TreeBuilder.runParser(TreeBuilder.java:47)
at org.jsoup.parser.TreeBuilder.parse(TreeBuilder.java:41)
at org.jsoup.parser.HtmlTreeBuilder.parse(HtmlTreeBuilder.java:37)
at org.jsoup.parser.Parser.parseInput(Parser.java:30)
at org.jsoup.helper.DataUtil.parseByteData(DataUtil.java:102)
at org.jsoup.helper.HttpConnection$Response.parse(HttpConnection.java:498)
at org.jsoup.helper.HttpConnection.get(HttpConnection.java:154)
EDIT:
After setting maximum memory I am getting this error
OpenJDK 64-Bit Server VM warning: Attempt to allocate stack guard pages failed.

This clearly indicates that you are running out of the heap space. So you can try by increasing the heap space of the Java virtual machine using coomand java -Xms<initial heap size> -Xmx<maximum heap size>
As per my knowledge, default values are initial: 32M and maximum: 128M. So can make it max value as 256M or 512M.
Have a look at this for information about the Java VM. http://www.oracle.com/technetwork/java/javase/tech/vmoptions-jsp-140102.html

Exception in thread "http-8080-10" java.lang.OutOfMemoryError: Java

I have a Web application running on my 64-bit Windows Server 2003, Oracle 11G database and Apache Tomcat 6.0 Web Server.
Application is on live environment and around 3000 of user using the application I have encountered Java Heap Out Of Memory Error. After increasing Heap space it's resolved.
Now again I am facing same issue, below is the error stack trace:
Exeption in thread "http-8080-10" java.lang.OutOfMemoryError: Java
heap space Aug 23, 2013 8:48:00 PM com.SessionClunter
getActiveSessions Exeption in thread "http-8080-11"
java.lang.OutOfMemoryError: Java heap space Exeption in thread
"http-8080-4" Exeption in thread "http-8080-7"
java.lang.OutOfMemoryError: Java heap space

Your problem could be caused by a few things (at a conceptual level):
You could simply have too many simultaneous users or user sessions.
You could be attempting to process too many user requests simultaneously.
You could be attempting to process requests that are too large (in some sense).
You could have a memory leak ... which could be related to some of the above issue, or could be unrelated.
There is no simple solution. (You've tried the only easy solution ... increasing the heap size ... and it hasn't worked.)
The first step in solving this is to change your JVM options to get it to take a heap dump when a OOME occurs. Then you use a memory dump analyser to examine the dump, and figure out what objects are using too much memory. That should give you some evidence that will allow you to narrow down the possible causes ...

If you keep getting OutOfMemoryError no matter how much you increase the max heap, then your application probably has a memory leak, which you must solve by getting into the code and optimizing it. Short of that, you have no other choice but keep increasing the max heap until you can.
You can look for memory leaks and optimize using completely free tools like this:
Create a heap dump of your application when it uses a lot of memory, but before it would crash, using jmap that is part of the Java installation used by your JVM container (= tomcat in your case):
# if your process id is 1234
jmap -dump:format=b,file=/var/tmp/dump.hprof 1234
Open the heap dump using the Eclipse Memory Analyzer (MAT)
MAT gives suggestions about potential memory leaks. Try to follow those.
Look at the histogram tab. It shows all the objects that were in memory at the time of the dump, grouped by their class. You can order by memory use and number of objects. When you have a memory leak, usually there are shockingly too many instances of some objects that clearly don't make sense all. I often tracked down memory leaks based on that info alone.
Another useful free JVM monitoring tool is VisualVM. A non-free but very powerful tool is JProfiler.

How am I getting a "java.lang.OutOfMemoryError: Java heap space" if my heap isn't full?

I'm debugging a fairly large project I've been working on (but did not originally create) and I've noticed that sometimes it crashes with an OutOfMemoryError. The code is loading a lot of data from files so this isn't entirely surprising in general.
However, what confuses me is that I'm using VisualVM 1.3.4 to profile the program, and it behaves inconsistently. Most times I've run it, the heap gradually expands up to about 2GB (the computer has 16GB of RAM; it's for academic research) with the used heap spiking higher and higher underneath it. Around 2GB, it will crash. The program isn't proccessing more information as time goes on though, so it shouldn't grow the heap to 2GB in just a few minutes.
Sometimes, though, I get a sudden crash after about 30 seconds, with a heap size of 250MB and only about 100MB in use. How am I getting a java.lang.OutOfMemoryError: Java heap space if my heap isn't full?
Edit: I'm using Eclipse to run the program, and I have the VisualVM plugin so it gets launched automatically. Also, I'm using Java 7.

Start the application with the VM argument -XX:+HeapDumpOnOutOfMemoryError.
Analyse the Heap Dump and find out what is causing the issue.
Eclipse MAT is an excellent tool for finding out such issues.

you need to setup the JVMs min and max heap memory
set JAVA_OPTS="-Xms128m -Xmx256m"
something like that but with bigger values like 2G, 4G whatever
LE: As you all know you can't force JVM to run the garbage collector (even though you can ask for it), but there are some ways of convincing it to get rid of some items by null-ing their references. Another thing to watch is the database object that might be lazy initialised. That error could appear when you try to create an object exceding the max heap memory.
Another ideea could be some retarded developer that programatically threw the OutOfMemoryError in some method for some retarded reason. When you reach that part of code, that's what you get (search the project)

There can be at least 2 reasons for the application to crash with OutOfMemoryError.
Your java heap is just too small for the amount of data it needs to process. Then you can either increase it as suggested Matei, or analyze heap dump as suggest Ajay.
Your application leaks memory. Which means that it leaves some unneeded data in memory after processing it. Then increasing heap will not help in the long run. And your options are either heap dump analysis (again) or specialised memory leak detection tool, such as Plumbr

Turned out the crash was caused by using the OpenJDK JRE rather than Oracle's JRE. I don't know exactly what the bug is in OpenJDK that makes it crash like this, but changing to Oracle's JRE ultimately solved the problem.
(I was using OpenJDK because I'm on a Linux computer that someone was using for open-source work before me. When I mentioned the crash to him he had the idea that that might be the cause. He was right.)

Do you have a 32bit operative system without large memory support (PAE on win, huge mem kernel on linux...)? If yes, you may encounter the 2GB memory segment limit per process on 32 bit systems.
As a work around, try to set the JVM parameter -Xss192k to dispose 192kb of stack space per thread, and the parameter -Xmx1024m to use no more than 1GB of heap.

Java "Out of memory error" - heap/system - where to look?

In my Tomcat application I am eventually getting "Out of memory" and "Cannot allocate memory" errors. I suppose it is nothing to do with the heap as it completely fulls up the system memory and I am hardly able to run bash commands.
How this problem is connected to the heap? How can I correctly set up heap size so that the application has enough memory and so it does not consume too much of the system resources?
Strange thing is that "top" command keeps saying that tomcat consumes only 20% of mem and there is still free memory, once the problem happens.
Thanks.
EDIT:
Follow-up:
BufferedImage leaks - are there any alternatives?

Problems with running bash scripts may indicate I/O issues, and this might be the case if your JVM is doing Full GCs all the time (which is the case, if your heap is almost-full).
The first thing to do, is to increase the heap with -Xmx. This may solve the problem, or - if you have a memory leak, it won't, and you will eventually get OutOfMemoryError again.
In this case, you need to analyze memory dumps. See my answer in this thread for some instructions.
Also, it might be useful to enable Garbage Collection Logs (using -Xloggc:/path/to/log.file -XX:+PrintGCDetails) and then analyzing them with GCViewer or HPJmeter.

You can set JVM Heap size by specifying the options
-Xmx1024m //for 1024 MB
Refer this for setting the option forTomcat

If you have 4 GB ram then can allocate 3GB to HEAP -
-Xmx3GB

you can also change the available perm gen size by using the following commands:
-XX:PermSize=128m
-XX:MaxPermSize=256m

How to debug Java OutOfMemory exceptions?

What is the best way to debug java.lang.OutOfMemoryError exceptions?
When this happens to our application, our app server (Weblogic) generates a heap dump file. Should we use the heap dump file? Should we generate a Java thread dump? What exactly is the difference?
Update: What is the best way to generate thread dumps? Is kill -3 (our app runs on Solaris) the best way to kill the app and generate a thread dump? Is there a way to generate the thread dump but not kill the app?

Analyzing and fixing out-of-memory errors in Java is very simple.
In Java the objects that occupy memory are all linked to some other objects, forming a giant tree. The idea is to find the largest branches of the tree, which will usually point to a memory leak situation (in Java, you leak memory not when you forget to delete an object, but when you forget to forget the object, i.e. you keep a reference to it somewhere).
Step 1. Enable heap dumps at run time
Run your process with -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/tmp
(It is safe to have these options always enabled. Adjust the path as needed, it must be writable by the java user)
Step 2. Reproduce the error
Let the application run until the OutOfMemoryError occurs.
The JVM will automatically write a file like java_pid12345.hprof.
Step 3. Fetch the dump
Copy java_pid12345.hprof to your PC (it will be at least as big as your maximum heap size, so can get quite big - gzip it if necessary).
Step 4. Open the dump file with IBM's Heap Analyzer or Eclipse's Memory Analyzer
The Heap Analyzer will present you with a tree of all objects that were alive at the time of the error.
Chances are it will point you directly at the problem when it opens.
Note: give HeapAnalyzer enough memory, since it needs to load your entire dump!
java -Xmx10g -jar ha456.jar
Step 5. Identify areas of largest heap use
Browse through the tree of objects and identify objects that are kept around unnecessarily.
Note it can also happen that all of the objects are necessary, which would mean you need a larger heap. Size and tune the heap appropriately.
Step 6. Fix your code
Make sure to only keep objects around that you actually need. Remove items from collections in a timely manner. Make sure to not keep references to objects that are no longer needed, only then can they be garbage-collected.

I've had success using a combination of Eclipse Memory Analyzer (MAT) and Java Visual VM to analyze heap dumps. MAT has some reports that you can run that give you a general idea of where to focus your efforts within your code. VisualVM has a better interface (in my opinion) for actually inspecting the contents of the various objects that you are interested in examining. It has a filter where you can have it display all instances of a particular class and see where they are referenced and what they reference themselves. It has been a while since I've used either tool for this they may have a closer feature set now. At the time using both worked well for me.

What is the best way to debug java.lang.OutOfMemoryError exceptions?
The OutOfMemoryError describes type of error in the message description. You have to check the description of the error message to handle the exception.
There are various root causes for out of memory exceptions. Refer to oracle documentation page for more details.
java.lang.OutOfMemoryError: Java heap space:
Cause: The detail message Java heap space indicates object could not be allocated in the Java heap.
java.lang.OutOfMemoryError: GC Overhead limit exceeded:
Cause: The detail message "GC overhead limit exceeded" indicates that the garbage collector is running all the time and Java program is making very slow progress
java.lang.OutOfMemoryError: Requested array size exceeds VM limit:
Cause: The detail message "Requested array size exceeds VM limit" indicates that the application (or APIs used by that application) attempted to allocate an array that is larger than the heap size.
java.lang.OutOfMemoryError: Metaspace:
Cause: Java class metadata (the virtual machines internal presentation of Java class) is allocated in native memory (referred to here as metaspace)
java.lang.OutOfMemoryError: request size bytes for reason. Out of swap space?:
Cause: The detail message "request size bytes for reason. Out of swap space?" appears to be an OutOfMemoryError exception. However, the Java HotSpot VM code reports this apparent exception when an allocation from the native heap failed and the native heap might be close to exhaustion
java.lang.OutOfMemoryError: Compressed class space
Cause: On 64-bit platforms a pointer to class metadata can be represented by a 32-bit offset (with UseCompressedOops). This is controlled by the command line flag UseCompressedClassPointers (on by default).
If the UseCompressedClassPointers is used, the amount of space available for class metadata is fixed at the amount CompressedClassSpaceSize. If the space needed for UseCompressedClassPointers exceeds CompressedClassSpaceSize, a java.lang.OutOfMemoryError with detail Compressed class space is thrown.
Note: There is more than one kind of class metadata - klass metadata and other metadata. Only klass metadata is stored in the space bounded by CompressedClassSpaceSize. The other metadata is stored in Metaspace.
Should we use the heap dump file? Should we generate a Java thread dump? What exactly is the difference?
Yes. You can use this heap heap dump file to debug the issue using profiling tools like visualvm or mat
You can use Thread dump to get further insight about status of threads.
Refer to this SE question to know the differenes:
Difference between javacore, thread dump and heap dump in Websphere
What is the best way to generate thread dumps? Is kill -3 (our app runs on Solaris) the best way to kill the app and generate a thread dump? Is there a way to generate the thread dump but not kill the app?
kill -3 <process_id> generates Thread dump and this command does not kill java process.

It is generally very difficult to debug OutOfMemoryError problems. I'd recommend using a profiling tool. JProfiler works pretty well. I've used it in the past and it can be very helpful, but I'm sure there are others that are at least as good.
To answer your specific questions:
A heap dump is a complete view of the entire heap, i.e. all objects that have been created with new. If you're running out of memory then this will be rather large. It shows you how many of each type of object you have.
A thread dump shows you the stack for each thread, showing you where in the code each thread is at the time of the dump. Remember that any thread could have caused the JVM to run out of memory but it could be a different thread that actually throws the error. For example, thread 1 allocates a byte array that fills up all available heap space, then thread 2 tries to allocate a 1-byte array and throws an error.

You can also use jmap/jhat to attach to a running Java process. These (family of) tools are really useful if you have to debug a live running application.
You can also leave jmap running as a cron task logging into a file which you can analyse later (It is something which we have found useful to debug a live memory leak)
jmap -histo:live <pid> | head -n <top N things to look for> > <output.log>
Jmap can also be used to generate a heap dump using the -dump option which can be read through the jhat.
See the following link for more details
http://www.lshift.net/blog/2006/03/08/java-memory-profiling-with-jmap-and-jhat
Here is another link to bookmark
http://java.sun.com/developer/technicalArticles/J2SE/monitoring/

It looks like IBM provides a tool for analyzing those heap dumps: http://www.alphaworks.ibm.com/tech/heaproots ; more at http://www-01.ibm.com/support/docview.wss?uid=swg21190476 .

Once you get a tool to look at the heap dump, look at any thread that was in the Running state in the thread stack. Its probably one of those that got the error. Sometimes the heap dump will tell you what thread had the error right at the top.
That should point you in the right direction. Then employ standard debugging techniques (logging, debugger, etc) to hone in on the problem. Use the Runtime class to get the current memory usage and log it as the method in or process in question executes.

I generally use Eclipse Memory Analyzer. It displays the suspected culprits (the objects which are occupying most of the heap dump) and different call hierarchies which is generating those objects. Once that mapping is there we can go back to the code and try to understand if there is any possible memory leak any where in the code path.
However, OOM doesn't always mean that there is a memory leak. It's always possible that the memory needed by an application during the stable state or under load is not available in the hardware/VM. For example, there could be a 32 bit Java process (max memory used ~ 4GB) where as the VM has just 3 GB. In such a case, initially the application may run fine, but OOM may be encountered as and when the memory requirement approaches 3GB.
As mentioned by others, capturing thread dump is not costly, but capturing heap dump is. I have observed that while capturing heap dump application (generally) freezes and only a kill followed by restart helps to recover.