We Keep Coding

Troubleshooting Java memory usage exception

I'm trying to troubleshoot a Java program that requires increasingly more memory until it cannot allocate any more and then it crashes.
EDIT More information about the program. The program is an indexer going through thousands of documents and indexing them for search. The documents are read from MongoDB and written to MongoDB as well after some processing is performed. During the processing I'm using RocksDB (rocksdb-jni version 5.13.4 from Maven). There is some mentioning in this GitHub issue of RocksDB memory usage growing uncontrollably, but I'm not sure it could be related.
Monitoring the process with visualvm results in the following plot:
but running htop on the machine shows totally different stats:
There is a difference of several GBs of memory that I'm unable to trace the source of.
The program is launched with the following VM arguments:
jvm_args: -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.port=<port> -Djava.rmi.server.hostname=<hostname> -Xmx12G -XX:+UseStringDeduplication
The system has 32GB of RAM and no swap. Out of this 32 GB, ~10GB are always taken by a tmpfs partition, ~8GB by MongoDB and the remaining 12GB are assigned to the program. EDIT The visualvm screenshot above shows 20GB of heap size, because it was from a previous run where I passed -Xmx20G; the behaviour, however, is the same whether I assign 12GB or 20GB to the heap. The behaviour also does not change if I remove the tmpfs partition, freeing 10 more GB of memory: it just takes longer but eventually it will get out of memory.
I have no idea where this memory usage that is not shown in visualvm but appears in htop is coming from. What tools should I use to understand what is going on? The application is running on a remote server, so I would like a tool that only works in the console or can be configured to work remotely, like visualvm.

I always use JProfiler but i hear jetbrains has a good one as well both can connect remotely.
If possible i would try to create a (local) setup where you can freely test it.
in the RocksDB several possible solutions are mentioned, do they work?
RocksDB seems to need some configuration, how did you configure it?
I am not familiar with RocksDB but i see there is some indexing and caching going on. How much data are you processing and what indexes/caching configuration do you have? are you sure this should fit in memory?
as far as i know the memory mismatch is because jni usage is not shown by default by most. There are some flags to improve this NativeMemoryTracking. i can't recall if this will add them to your visualvm overviews as well.
karbos-538 i just noticed this is quite an old issue, i hope this application is already working by now. what part of the question is still relevant to you?

javaws.exe poor performance compared with java.exe in java8

You have to contract for supporting non public available Java releases with Oracle for example Java6 ended with 1.6.0_45 publicly but with support you can use 1.6.0_80. Our aim is to stay supported and public available java platform. Therefore we want to migrate our jnlp based RIA application from Java6 to java8 platform. In migration testing we have experienced performance problem. We try to simulate this problem in Java 8 poor GUI performance compared to Java 6 but we realize that the problem not related the that question.
After the introduction, in order to find out the root cause of problem we profiled the application with java mission control(jmc) bundled with jdk. when we profile application launched at workspace we cannot see any performance degradation. And then we realize that problem occurs only JAVAWS environment. Our application runs as expected at java environment but in javaws environment some GUI's are really slow , it adds 6 seconds more to response time.
My first question is that how we profile java application launched by javaws? Because to use jmc we need to set some JVM arguments (java-vm-args="-XX:+UnlockCommercialFeatures -XX:+FlightRecorder") but javaws doesn't support this JVM parameters. You can find supported list at here
My Second question is that why application doesn't run identical at both environment(java and javaws)? We examined signed and unsigned version of our application and both have same situation. By the way our application uses all-permission.

You can pass any JVM arguments you want to javaws by typing them in:
Java Control Panel > Java > View > Runtime Parameters
As for the poor performance, are you referring to application load time? or is it also slow after the applicatio is started?
If you're talking about load time, then yes javaws is slower (about 3-5 seconds slower) because it has to contact Internet to check if your jar files are black listed or if the certificate is expired. I'm not sure exactly what it is doing in these few seconds but I agree that it is disappointing performance hit and I hope someone can shed more light on it.

Thanks Saeid. Problem is determined. We profiled finally and we found that javaws consume long time to load resources e.g. icons, images. to make a solution we cached the resources and for absence images we return a empty image.
You can see this at
To profile application launched by javaws we make a solution following steps in ubuntu
1. move java executable javam
2. create a script file named java and insert following script
#!/bin/bash
/full/path/of/javam XX:+UnlockCommercialFeatures -XX:+FlightRecorder "$#"
3. run application and check java is launched by ps -ef | grep javam
4. run jmc and profile it.

java heap space problem in remote desktop

I have remote logged into my machine and trying to start tomcat server. But, I get the following error.
Error occurred during initialization of VM
Could not reserve enough space for object heap
Could not create the Java virtual machine.
Apparently there are some memory restrictions when logged into remote desktop. Is there any way I can start the tomcat from RDP?
Thanks

Remote desktop could be causing the problem if you are using a 32-bit system. Java requires a single continuous block of memory to start Java. If you start Java with close to the maximum amount of memory, whether you get that memory as a single block depends on what you have already running on that server.
Solutions include
start Java as a service on start up.
use a 64-bit version of the OS and Java.
use less memory in the JVM, even 100 Mb could make a difference.
increase the amount of main memory in the machine.

In starting up the Tomcat server, you can change the program parameters when running Java. You can add to the VM arguments -Xmx###m to a smaller number that may work on your machine. Also, you may want to see if you're running other memory intensive Java apps.

Problems with jetty crashing intermittently

I'm having problems with jetty crashing intermittently, I'm using Jetty 6.1.24.
I'm running a neo4j Spring MVC webapp, Jetty will stay running for approx 1 hour and then I have to restart Jetty. It is running on small amazon ec2 instance, debian with 1.7gb of RAM.
I start Jetty using java -Xmx900m -server -jar start.jar
I am connecting to the server using putty, when Jetty crashes the putty session disconnects, I cannot see what error caused it to crash.
I would like to be able to see if it is an error generated by Spring, I'm not sure how to log the output from the spring app with Jetty. Or if it is Jetty or a memory issue, what would be the best way to monitor Jetty? I cannot recreate this on my local machine running windows. What do you think would be the best way to approach this? Thanks

This isn't really a programmer question; perhaps it'll be moved over to ServerFault.
You didn't specifically state which operating system you're using, but I'm hazarding a guess at some Linux distribution. You have two options of figuring out what's wrong:
Start your session in screen. Screen will live for as long as the actual machine is powered on, until you reboot the operating system (or you exit screen).
you start screen like this
screen
and you get a new prompt where you can start your program (cd foo, jetty, etc). When you're happy and you just need to go somewhere, you can disconnect the screen by hitting CTRL+A and then CTRL+D. you'll drop back to the place you were before you invoked screen.
To get back to seeing the screen you type screen -R which means to resume an existing screen. you should see jetty again.
The nice thing is that if you lose connection (or you close putty by accident or whatever) then you can use screen -list to get a list of running screens, and then forcibly detach them -D and reattach them to the current putty -R, no harm done!
Use nohup. Nohup more or less detaches the process you're running from the console, so none of its output comes to the terminal. You start your program in the normal fashion, but you add the word nohup to your command.
For example:
nohup ls -l &
After ls -l is complete, your output is stored in nohup.out.

When you say crash do you mean the JVM segfaults and disappears? If that's the case I'd check and make sure you aren't exhausting the machine's available memory. Java on linux will crash when the system memory gets so low the JVM cannot allocate up to its maximum memory. For example, you've set the max JVM memory to 500MB of which it's using 250MB at the moment. However, the Linux OS only has 128MB available. This produces unstable results and the JVM will segfault.
On windows the JVM is more well behaved in this scenario and throws OutOfMemoryError when the system is running low on memory.
Validate how much system memory is available around the time of your crashes.
Verify if other processes on your box are eating up a lot of memory. Turn off anything that could be competing with the JVM.
Run jconsole and connect it to your JVM. That will tell you how memory is being used in your JVM process and give you a history to look back through when it does crash.
Eliminate any native code you might be loading into the JVM when doing this type of testing.
I believe Jetty has some native code to do high volume request processing. Make sure that's not being used. You want to isolate the crashes to Java and NOT some strange native lib. If you take out the native stuff and find it works then you have your answer as to what's causing it. If it continues to crash then it very well could be what I'm describing.
You can force the JVM to allocate all the memory at startup with -Xms900m that can make sure the JVM doesn't fight with other processes for memory. Once it has the full Xmx amount allocated it won't crash. Not a solution, but you can easily test it this way.

When you start java, redirect both outputs (stdout and stderr) to a file:
Using Bash:
java -Xmx900m -server -jar start.jar > stdout.txt 2> stderr.txt
After the crash, inspect those files.
If the crash is due to a signal (like SEGV=segmentation fault), there should be a file dump by the JVM at the location you've started java. For Sun VM (hotspot), it's something like hs_err_pid12121.log (here 12121 is the process ID).

Putty disconnecting STRONGLY hints that the server is running out of memory and starts shutting down processes left and right. It is probably your jetty instance growing too big.
The easiest thing to do now, is adding 1-2 Gb more swap space and do it again. Also note that you can use the jvisualvm to attach to the jetty instance to get runtime information directly.

IBM JRE 1.5 will not startup with the requested 1.5G memory

IBM JRE 5.0 on Windows, when given -Xmx1536m on a laptop with 2GB memory, refuses to start up: error message below. With -Xmx1000m it does start.
Also, it starts fine with -Xmx1536m on other servers and even laptops, so I think that there is something more than just inadequate memory.
Also, when started from within Eclipse (albeit, using the JRE in the IBM 5 JDK in this case) with the same memory parameter, it runs fine.
Any idea what is going on here?
JVMJ9VM015W Initialization error for library j9gc23(2): Failed to instantiate heap. 1536M requested
Could not create the Java virtual machine
Edit:
Does anyone know about the "3GB switch" and if it is relevant here (beyond the obvious fact that approximately that this is a memory limitations problem). How can I tell if it is enabled and what is the most straightforward way to turnit on?

According to IBM DeveloperWorks:
Cause
The system does not have the necessary resources to satisfy the
maximum default heap value required to
run the JVM.
To resolve, here is what it says
Resolving the problem
If you receive
this error message when starting the
JVM, free
memory by stopping other applications
that might be consuming system
resources.
Your JVM doesn't have enough memory resources to create maximum amount of heap space of 1536 MB. Just make sure that you have enough memory to accommodate it.
Also, I believe that in Windows, the maximum heap space is 1000MB? I'm not sure if that's solid, but in Linux/AIX, any Xmx more than 1GB works fine.

The JVM requires that it be able to allocate its memory as a single contiguous block. If you are on a 32-bit system, the maximum available is about 1280M more or less. To get more you must run a 64-bit JVM on a 64-bit OS.
You may be able to get a little more by starting the JVM immediately after rebooting.
As to starting OK on other systems, are those 32 or 64-bit?

Pretty much the maximum you are guaranteed to get on a Windows platform is 1450 MB. Sometimes Windows/Java.exe maps DLLS to addresses in the 1.5-2.0GB range. This doesn't change even if you use the /3GB trick (or you have an OS that supports it). You have to manually rebase the DLLs to get them higher towards the 2GB (or 3GB boundary). It's a real pain in the ass, and I've done it before, but the best I've ever been able to get with and without a combination of /3GB is 1.8G on 32bit Windows.
Best to be done with it and migrate to a 64-bit OS. They're prevalent now-a-days.

I have the same issue in IBM Engineering lifecycle installation:-
Problem:- JVMJ9VM015W Initialization error for library j9gc26(2): Failed to instantiate heap; Could not create the Java virtual machine.
Solution:- I just did it and solve my issue. If you don't have 16GB ram then please don't change the jazz server startup file. If you have 8GB ram then Only do not increase memory size in the server.:
**set JAVA_OPTS=%JAVA_OPTS% -Xmx4G**
**set JAVA_OPTS=%JAVA_OPTS% -Xms4G**
**set JAVA_OPTS=%JAVA_OPTS% -Xmn1G**