We Keep Coding

Solved: JVM hangs forever with no CPU usage or I/O

I'm experiencing this weird problem that the JVM hangs forever very frequently.
I first observed the problem when my Java IDEs frequently hang the entire system GUI. IntelliJ IDEA hangs on indexing almost every single time upon start. Sometimes it proceeds to resolving dependency but always hangs in the end. When this happens, I can type in gnome-terminal, but the commands can't seem to be executed. I can't launch new applications with Alt-F2 or anything alike.
I had to switch to a text console and "killall -9 java" to kill the IDEA process and get control back. "kill -3 java" won't work. The log file contains nothing related, the thread dump is empty. Once the IDE hung, jstack cannot be attached to the process. "jstack -l pid" also hangs. "jstack -F pid" can't attach to the process. Visualvm hangs as well.
The CPU usage by the Java process is 0% and there is no I/O going on.
I've observed the same behavior when using Eclipse. Sometimes it hangs on start up, sometimes upon saving and sometimes upon running a Java application.
Maven / sbt builds executed within text-only ttys cause the same kind of hang, so I guess it's not a window manager / desktop environment / display driver problem.
I highly suspect it's a file system or I/O issue but I have no clue how to debug that. I've tried fsck with no luck, and my system works perfectly fine when not running java programs.
Things I've ruled out:
Permission issues: running IntelliJ with sudo doesn't help, hangs 100% of the time.
Display driver: I've tried both the Nvidia proprietary driver and nouveau, the open source one. Doesn't help.
Window manager / desktop environment: I use Cinnamon, but I've tried running IntelliJ under Unity. Doesn't help.
Java version: I've tried both Oracle Java 7 and Oracle Java 8. I'll probably try OpenJDK but I doubt it would work.
IntelliJ version: I've tried IntelliJ 13 through 14.1. All exhibited the same behavior.
Limited memory: I have 16G RAM with 16G swap space, so memory should not be a limiting factor.
Kernel log doesn't look suspicious. I can't get any kind of log remotely indicating what went wrong.
Any idea?
UPDATE (2015/04/29): The problem seems to have fixed itself after I accidentally kicked the power cable and cold restarted the computer... Still a mystery but IntelliJ is usable as of now.

Some things to check
- The Java IDEs run best with a lot of ram. I usually ask for at least
8G of memory for my dev workstation.
- Make sure you have a stable version of everything, look for known working versions/configurations on Ubuntu
- You have to manually allocate memory in IntelliJ IDEA versions < 14. For example: How to increase IDE memory limit in IntelliJ IDEA on Mac?
- Besides system logs, run tools like top and see what's happening in terms for cpu and ram when running the IDE

I had similar problems a while ago but with Eclipse. The problem was that there was no swap place at all ;) - obviously it should not be a problem with 16GB of RAM.
Could You post JVM arguments for Intellij? And also I have an idea to create another Intellij installation (eg. go back to 14 version) and see if there is similar problem (also compare JVMs settings between these two).
Edit
Ok so try:
use different JRE/JDK. If the problem disappears it will tell us more.
You are on linux so it makes it easy to monitor several things - you said that there is no CPU utilization or hard I/O. But how do you know that? Maybe it will be informative to have some statistics gathered - eg.jstat for JVM itself or for system information (You think that is I/O problem) try:
iostat -hm -p sda 1
Which will print I/O statistics for sda (if you have different discs change device parameter) in 1sec loops (can be also changed). Start this with system and dump output to the file - maybe there is some kind of 'disaster' happening before JVM hangs. Note: iostat sometimes is not available on system itself (on my Linux Mint is not) - install then package sysstat and the command will be available.

Seem to have fixed after a cold restart from an accidental power loss.. Weirdest problem I have ever seen.

javaws.exe poor performance compared with java.exe in java8

You have to contract for supporting non public available Java releases with Oracle for example Java6 ended with 1.6.0_45 publicly but with support you can use 1.6.0_80. Our aim is to stay supported and public available java platform. Therefore we want to migrate our jnlp based RIA application from Java6 to java8 platform. In migration testing we have experienced performance problem. We try to simulate this problem in Java 8 poor GUI performance compared to Java 6 but we realize that the problem not related the that question.
After the introduction, in order to find out the root cause of problem we profiled the application with java mission control(jmc) bundled with jdk. when we profile application launched at workspace we cannot see any performance degradation. And then we realize that problem occurs only JAVAWS environment. Our application runs as expected at java environment but in javaws environment some GUI's are really slow , it adds 6 seconds more to response time.
My first question is that how we profile java application launched by javaws? Because to use jmc we need to set some JVM arguments (java-vm-args="-XX:+UnlockCommercialFeatures -XX:+FlightRecorder") but javaws doesn't support this JVM parameters. You can find supported list at here
My Second question is that why application doesn't run identical at both environment(java and javaws)? We examined signed and unsigned version of our application and both have same situation. By the way our application uses all-permission.

You can pass any JVM arguments you want to javaws by typing them in:
Java Control Panel > Java > View > Runtime Parameters
As for the poor performance, are you referring to application load time? or is it also slow after the applicatio is started?
If you're talking about load time, then yes javaws is slower (about 3-5 seconds slower) because it has to contact Internet to check if your jar files are black listed or if the certificate is expired. I'm not sure exactly what it is doing in these few seconds but I agree that it is disappointing performance hit and I hope someone can shed more light on it.

Thanks Saeid. Problem is determined. We profiled finally and we found that javaws consume long time to load resources e.g. icons, images. to make a solution we cached the resources and for absence images we return a empty image.
You can see this at
To profile application launched by javaws we make a solution following steps in ubuntu
1. move java executable javam
2. create a script file named java and insert following script
#!/bin/bash
/full/path/of/javam XX:+UnlockCommercialFeatures -XX:+FlightRecorder "$#"
3. run application and check java is launched by ps -ef | grep javam
4. run jmc and profile it.

CPU/Memory Tab Not Showing Up When Profiling With Java Visual VM

When I try to profile my java application with Java VisualVM the Threads, CPU and Memory profiling tabs do not show up. I do not have any JVM arguments for my program apart from "-Djava.library.path=lib/native", is there an specific argument that I need to use to enable profiling?
Other information:
Built with Netbeans
Uses LWJGL

Just so this will get removed from the Unanswered list:
The problem was resolved by running the same version of VisualVM as the JVM (64-bits in this case). The successful profiling can be version specific.

Profile Entire Java Program Execution in VisualVM

In Java profiling, it seems like all (free) roads nowadays lead to the VisualVM profiler included with JDK6. It looks like a fine program, and everyone touts how you can "attach it to a running process" as a major feature. The problem is, that seems to be the only way to use it on a local process. I want to be able to start my program in the profiler, and track its entire execution.
I have tried using the -Xrunjdwp option described in how to profile application startup with visualvm, but between the two transport methods (shared memory and server), neither is useful for me. VisualVM doesn't seem to have any integration with the former, and VisualVM refuses to connect to localhost or 127.0.0.1, so the latter is no good either. I also tried inserting a simple read of System.in into my program to insert a pause in execution, but in that case VisualVM blocks until the read completes, and doesn't allow you to start profiling until after execution is under way. I have also tried looking into the Eclipse plugin but the website is full of dead links and the launcher just crashes with a NullPointerException when I try to use it (this may no longer be accurate).
Coming from C, this doesn't seem like a particularly difficult task to me. Am I just missing something or is this really an impossible request? I'm open to any kinds of suggestions, including using a different (also free) profiler, and I'm not averse to the command line.

Consider using HPROF and opening the data file with a tool like HPjmeter - or just reading the resulting text file in your favorite editor.
Command used: javac -J-agentlib:hprof=heap=sites Hello.java
SITES BEGIN (ordered by live bytes) Fri Oct 22 11:52:24 2004
percent live alloc'ed stack class rank self accum bytes objs bytes objs trace name
1 44.73% 44.73% 1161280 14516 1161280 14516 302032 java.util.zip.ZipEntry
2 8.95% 53.67% 232256 14516 232256 14516 302033 com.sun.tools.javac.util.List
3 5.06% 58.74% 131504 2 131504 2 301029 com.sun.tools.javac.util.Name[]
4 5.05% 63.79% 131088 1 131088 1 301030 byte[]
5 5.05% 68.84% 131072 1 131072 1 301710 byte[]
HPROF is capable of presenting CPU usage, heap allocation statistics,
and monitor contention profiles. In addition, it can also report
complete heap dumps and states of all the monitors and threads in the
Java virtual machine.

The best way to solve this problem without modifying your application, is to not use VisualVM at all. As far as other free options are concerned, you could use either Eclipse TPTP or the Netbeans profiler, or whatever comes with your IDE.
If you can modify your application, to suspend it's state while you setup the profiler in VisualVM, it is quite possible to do so, using the VisualVM Eclipse plugin. I'm not sure why you are getting the NullPointerException, since it appears to work on my workstation. You'll need to configure the plugin by providing the path to the jvisualvm binary and the path of the JDK; this is done by visiting the VisualVM configuration dialog at Windows -> Preferences -> Run/Debug - > Launching -> VisualVM Configuration (as shown in the below screenshot).
You'll also need to configure your application to start with the VisualVM launcher, instead of the default JDT launcher.
All application launches from Eclipse, will now result in VisualVM tracking the new local JVM automatically, provided that VisualVM is already running. If you do not have VisualVM running, then the plugin will launch VisualVM, but it will also continue running the application.
Inferring from the previous sentence, it is evident that having the application halt in the main() method before performing any processing is quite useful. But, that is not the main reason for suspending the application. Apparently, VisualVM or its Eclipse plugin does not allow for automatically starting the CPU or memory profilers. This would mean that these profilers would have to be started manually, thereby necessitating the need to suspend the application.
Additionally, it is worth noting that adding the flags: -Xrunjdwp:transport=dt_socket,address=8000,server=y,suspend=y to the JVM startup will not help you in the case of VisualVM, to suspend the application and setup up the profilers. The flags are meant to help you in the case of profilers that can actually connect to the open port of the JVM, using the JDWP protocol. VisualVM does not use this protocol and therefore you would have to connect to the application using JDB or a remote debugger; but that would not resolve the problem associated with profiler configuration, as VisualVM (at least as of Java 6 update 26) does not allow you to configure the profilers on a suspended process as it simply does not display the Profiler tab.

This is now possible with the startup profiler plugin to VisualVM.

The advice with -Xrunjdwp is incorrect. It just enables debugger and with suspend=y it waits for debugger to attach. Since VisualVM is not debugger, it does not help you. However inserting System.in or Thread.sleep() will pause the startup and allows VisualVM to attach to your application. Be sure to read Profiling with VisualVM 1 and Profiling with VisualVM 2 to better understand profiler settings. Note also that instead of profiling, you can use 'Sampler' tab in VisualVM, which is more suitable for profiling entire java program execution. As other mentioned you can also use NetBeans Profiler, which directly support profiling of the application startup.

Can I connect to a jsadebugd process on a remote machine from Eclipse/IDE debugger?

I have a java process running on a Linux box, but it cannot be started/re-started in debug mode. I read about the jsadebugd command and started the daemon, but I want to connect to it from an IDE(possibly Eclipse) debugger or maybe YourKit or jconsole. How can I do this? The remote JVM is 1.6+.

I assume since you mentioned Yourkit and other tool that what you really want to do is look at object state inside your applications. There are a couple of options, though I don't think it gets you the ability to set break-points like Eclipse or another debugger would (though I'd be intersted in hearing what restricts you from starting the process in debug mode - technical? process?)
have you tried connecting with: VisualVM? I don't believe you need to start in debug mode, and it will give you the ability to navigate the object graph, and inspect objects. You can even use it to take heapdumps and do some ad-hoc analysis through them using OQL queries.
If you're running on JDK6+ - have you tried instrumenting with btrace? Some notes from the sailfin team sounded promising, and like DTrace in Solaris, this seem like it would be useful in a variety of situations.

I am not sure if I understand your restrictions correctly but you can start JVM with debugging server (serving JDWP) enabled. See for example "Sun VM Invocation Options" in http://java.sun.com/j2se/1.4.2/docs/guide/jpda/conninv.html
Then you can connect your Eclipse debugger to running JVM. See "Remote debugging" section here http://www.ibm.com/developerworks/library/os-ecbug/

I'm not sure that this is exposed by Eclipse, but here's how to do it with jdb:
http://java.sun.com/javase/6/webnotes/trouble/TSG-VM/html/tooldescr.html#gbmog