We Keep Coding

Tuning JVM parameters to avoid java.lang.OutOfMemoryError: PermGen space

I'm getting an error "java.lang.OutOfMemoryError: PermGen space" very often (1-2 times per week) and I cannot localize the problem.
Thread dump shows no blocked processes. And a heap dump cannot be done when this error occurs (even when parameter Heap dump on OOME: enabled).
I've tried many options when restart the JVM but nothing helps.
Maybe somebody knows what can be a problem seeing this screenshots from Java VisualVM?
Tomcat 6, Java 1.6.
Overview JVM parameters:
Monitor:
Monitor (PermGen):

How to find non-heap space memory leak in java?

We have a java webserver that is using eclipse-jetty version 8.1.6. Recently we started noticing Out of memory error.
We had few profiling on the number of threads active. This seems to be reasonable around 100. The process has 5GB max heap memory and 4GB initial heap memory.
Process Details
Environment: Docker(kubernetes)
java.version="1.8.0_91"
java.vm.info="mixed mode"
java.vm.name="Java HotSpot(TM) 64-Bit Server VM"
thread size = 1024K
ulimit is unlimited for max process per user.
Container(Pod) Max memory is allocated to be 8GB
The webserver receives on a average of 350 request per minute. Also we run many such instances behind ELB(kubernetes service). After running for few hours we notice this OOM. The issue is random and it occurs on stress test.
OOM StackTrace:
java.lang.OutOfMemoryError: unable to create new native thread
at java.lang.Thread.start0(Native Method) [na:1.8.0_91]
at java.lang.Thread.start(Thread.java:714) [na:1.8.0_91]
at org.eclipse.jetty.util.thread.QueuedThreadPool.startThread(QueuedThreadPool.java:441) [jetty-util-8.1.16.v20140903.jar:8.1.16.v20140903]
at org.eclipse.jetty.util.thread.QueuedThreadPool.dispatch(QueuedThreadPool.java:366) [jetty-util-8.1.16.v20140903.jar:8.1.16.v20140903]
Since the number of thread count is reasonable. I suspect some memory leak. But I am not able to find the off heap memory size on a docker container.
Is there a way to find it ?
After searching for a while ,I found this bug below in jetty.
How to verify that if the bug is due to below issue without upgrading jetty ?
Related Bug Id: https://bugs.eclipse.org/bugs/show_bug.cgi?id=477817

You could try to add -XX:-HeapDumpOnOutOfMemoryError into your java start parameters and to look into the dump.

Exception in thread "http-8080-10" java.lang.OutOfMemoryError: Java

I have a Web application running on my 64-bit Windows Server 2003, Oracle 11G database and Apache Tomcat 6.0 Web Server.
Application is on live environment and around 3000 of user using the application I have encountered Java Heap Out Of Memory Error. After increasing Heap space it's resolved.
Now again I am facing same issue, below is the error stack trace:
Exeption in thread "http-8080-10" java.lang.OutOfMemoryError: Java
heap space Aug 23, 2013 8:48:00 PM com.SessionClunter
getActiveSessions Exeption in thread "http-8080-11"
java.lang.OutOfMemoryError: Java heap space Exeption in thread
"http-8080-4" Exeption in thread "http-8080-7"
java.lang.OutOfMemoryError: Java heap space

Your problem could be caused by a few things (at a conceptual level):
You could simply have too many simultaneous users or user sessions.
You could be attempting to process too many user requests simultaneously.
You could be attempting to process requests that are too large (in some sense).
You could have a memory leak ... which could be related to some of the above issue, or could be unrelated.
There is no simple solution. (You've tried the only easy solution ... increasing the heap size ... and it hasn't worked.)
The first step in solving this is to change your JVM options to get it to take a heap dump when a OOME occurs. Then you use a memory dump analyser to examine the dump, and figure out what objects are using too much memory. That should give you some evidence that will allow you to narrow down the possible causes ...

If you keep getting OutOfMemoryError no matter how much you increase the max heap, then your application probably has a memory leak, which you must solve by getting into the code and optimizing it. Short of that, you have no other choice but keep increasing the max heap until you can.
You can look for memory leaks and optimize using completely free tools like this:
Create a heap dump of your application when it uses a lot of memory, but before it would crash, using jmap that is part of the Java installation used by your JVM container (= tomcat in your case):
# if your process id is 1234
jmap -dump:format=b,file=/var/tmp/dump.hprof 1234
Open the heap dump using the Eclipse Memory Analyzer (MAT)
MAT gives suggestions about potential memory leaks. Try to follow those.
Look at the histogram tab. It shows all the objects that were in memory at the time of the dump, grouped by their class. You can order by memory use and number of objects. When you have a memory leak, usually there are shockingly too many instances of some objects that clearly don't make sense all. I often tracked down memory leaks based on that info alone.
Another useful free JVM monitoring tool is VisualVM. A non-free but very powerful tool is JProfiler.

java.lang.OutOfMemoryError: Java heap space

My application is deployed on a cluster environment. Recently the server went down with the following stacktrace. It doesn't seem to be coming from the code. It was running all right until recently when this error pop up. No major changes were made to the server. Can someone advise?
java.lang.OutOfMemoryError: Java heap space
at java.lang.AbstractStringBuilder.<init>(AbstractStringBuilder.java:44)
at java.lang.StringBuilder.<init>(StringBuilder.java:69)
at java.io.ObjectStreamClass$FieldReflectorKey.<init>(ObjectStreamClass.java:2106)
at java.io.ObjectStreamClass.getReflector(ObjectStreamClass.java:2039)
at java.io.ObjectStreamClass.initNonProxy(ObjectStreamClass.java:586)
at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1552)
at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1466)
at java.io.ObjectInputStream.readArray(ObjectInputStream.java:1591)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1299)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:348)
at weblogic.utils.io.ChunkedObjectInputStream.readObject(ChunkedObjectInputStream.java:195)
at weblogic.rjvm.MsgAbbrevInputStream.readObject(MsgAbbrevInputStream.java:565)
at weblogic.utils.io.ChunkedObjectInputStream.readObject(ChunkedObjectInputStream.java:191)
at weblogic.rmi.internal.dgc.DGCServerImpl_WLSkel.invoke(Unknown Source)
at weblogic.rmi.internal.BasicServerRef.invoke(BasicServerRef.java:589)
at weblogic.rmi.internal.BasicServerRef$1.run(BasicServerRef.java:479)
at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:363)
at weblogic.security.service.SecurityManager.runAs(Unknown Source)
at weblogic.rmi.internal.BasicServerRef.handleRequest(BasicServerRef.java:475)
at weblogic.rmi.internal.BasicServerRef.access$300(BasicServerRef.java:59)
at weblogic.rmi.internal.BasicServerRef$BasicExecuteRequest.run(BasicServerRef.java:1016)
at weblogic.work.SelfTuningWorkManagerImpl.schedule(SelfTuningWorkManagerImpl.java:126)
at weblogic.rmi.internal.BasicServerRef.dispatch(BasicServerRef.java:321)
at weblogic.rmi.internal.BasicServerRef.dispatch(BasicServerRef.java:918)
at weblogic.rjvm.RJVMImpl.dispatchRequest(RJVMImpl.java:1084)
at weblogic.rjvm.RJVMImpl.dispatch(RJVMImpl.java:1001)
at weblogic.rjvm.ConnectionManagerServer.handleRJVM(ConnectionManagerServer.java:240)
at weblogic.rjvm.ConnectionManager.dispatch(ConnectionManager.java:877)
at weblogic.rjvm.MsgAbbrevJVMConnection.dispatch(MsgAbbrevJVMConnection.java:446)
at weblogic.rjvm.t3.MuxableSocketT3.dispatch(MuxableSocketT3.java:368)
at weblogic.socket.AbstractMuxableSocket.dispatch(AbstractMuxableSocket.java:383)
at weblogic.socket.SocketMuxer.readReadySocketOnce(SocketMuxer.java:872)

You are running out of memory, which suggests one of the following:
you need to give your process more memory (with the -Xmx java command line option); or
you have a memory leak
Without more information, it's hard to say which is the case. The stack trace for an OutOfMemoryError is rarely useful, as it only shows the point at which heap was exhausted; it doesn't show you why your heap is being filled up.

The answer by Simon Nickerson is correct
Just to add, your stack trace begins from weblogic.socket.SocketMuxer.readReadySocketOnce which is the internal weblogic class that accepts incoming requests. So this means the server is not having enough memory to accept requests also.

Are you using the JRockit JVM? If you are you can use JRockit Mission Control and monitor the Java heap usage. You can also use the JRockit Flight Recorder to record JVM events for offline analysis. There is an Oracle webcast on this here: http://www.vimeo.com/22109838. You can skip to 4:54 which is where the overview of JRockit, WLDF and JRF starts.
Keep in mind that when the heap is full it is the NEXT operation that fails with the OutOfMemory Exception, and therefore this stack trace may not indicate any cause of the failure. This simply indicates that when this code ran there wasn't enough heap, not that this code caused the heap to fill up.
** Edits...
Clearly the server is out of memory - at the time of this specific operation. The question is... why? This stack trace doesn't tell you -why- it just indicates that whatever was happening at the time could not complete because there was not enough memory available at that time. This does not mean that it is the cause of the problem.
Sure, you can add more memory but that may not fix the problem - it may only take longer for it to appear.

set catalina.sh/bat
find set JAVA_OPTS=%JAVA_OPTS%
what ever your RAM - adjust but don't give above half of RAM
set JAVA_OPTS=%JAVA_OPTS% %LOGGING_CONFIG% -server -Xms512M -Xmx512M -XX:MaxPermSize=256M

It means that the JVM has run out of all the memory that has been allocated to it. You can change the amount of memory allocated for use by your JVM using the -Xms and -Xmx command line parameters.check the root cause here
OutOfMemoryError in Java is a subclass of java.lang.VirtualMachineError and JVM throws java.lang.OutOfMemoryError when it ran out of memory in heap. OutOfMemoryError in Java can come any time in heap mostly while you try to create an object and there is not enough space in heap to allocate that object

What happens if you specify max heap size greater than available RAM

Asked in an interview. What happens if you specify max heap size (Xmx) greater than available RAM?
I also wonder what happens if you specify min heap size (Xms) greater than available RAM?

The easiest way to find out is try it and see.
Edit:
There are actually at least two answers to the question. Probably on a 64 bit system, as was mentioned, your app could grow and grow in memory usage and start thrashing. On a 32 bit system the story is a little different because the os is not able to give you that much heap space. For instance, if I run an app on Windows XP with 32 bit java with the command line option -Xmx2000m it will die with a message similar to the following:
Invalid maximum heap size: -Xmx2000m
The specified size exceeds the maximum representable size.
Could not create the Java virtual machine.
In Linux with 32 bit java, I get the following with -Xmx3000m:
Could not create the Java virtual machine.
Error occurred during initialization of VM
Could not reserve enough space for object heap
In Linux with 32 bit java, I get the following with -Xmx6000m
Invalid maximum heap size: -Xmx6000m
The specified size exceeds the maximum representable size.
Could not create the Java virtual machine.
Trying this with 64 bit Java, the JVM does allow you to allocate more memory than there is physical RAM, though if you ask for an extremely large amount of memory, the jvm will again fail with an error.

Only if your -Xms (minimum) is larger than available memory will you get an immediate failure on initialization of the JVM
$>java -Xms100g #JVM fails to start
Error occurred during initialization of VM Could not
reserve enough space for object heap
If your -Xmx (maximum) is larger than available memory your JVM does initialize since you are not using memory yet
$>java -Xmx100g #JVM starts up fine
Usage: java [-options] class [args...]
...
If your -Xmx (maximum) is larger than the available memory (total memory to include any virtual memory) you will get a runtime failure if and only if your JVM processes actually tries to use more memory than the machine has.
OpenJDK 64-Bit Server VM warning: INFO: os::commit_memory(0x00007f5feb100000, 927465472, 0) failed; error='Cannot allocate memory' (errno=12)
#
# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (mmap) failed to map 927465472 bytes for committing reserved memory.
# An error report file with more information is saved as:
# /some/file/path/hs_err_pid25.log
It wont 'thrash' until it nears your -Xmx limit, but if that limit is above your available memory you will get the above memory allocation error and your program will terminate before thrashing is even considered. (And that is very dramatic!)

Nothing Dramatic
Although it can happen with certain low-end embedded systems, these days it would be quite rare to see a non-virtual Java environment even in embedded and impossible on a desktop or server.
So, nothing dramatic would happen, but once you use up available RAM, allocating additional (virtual) memory would just unnecessarily delay reclamation (garbage collection) and cause the program to start paging.
If severe, this condition is called "thrashing" and it is not a good thing. Stuff would run slowly.