I've got two remote servers, both running recent CentOS, both running recent Tomcat6, recent JDK6, and VisualVM 1.3.2.
ssh -X forwarding works on one server - I can start up VisualVM from that machine, it port forwards and runs fine - I see all the JVM processes running on that remote machine as 'local' in VVM.
ssh -X forwarding on the second machine - then running VisualVM - brings up an X windows with VVM in it, but it just shows one 'local' process - the VisualVM itself - and the lower right corner has a bouncing progress bar that says "computing description", and it never ends.
I can't find anything about this anywhere - anyone ever hit this? How do I get past this?
I experienced a similar issue - VisualVM hanging on "computing description", not displaying any local JVMs other than itself. I used "jps" to find all the JVMs running on the system. Used jstack to get the stacks of all those JVMs, including JVisualVM itself. What I found was happening, was that JVisualVM was trying to create an RMI connection to one of the target VMs, and that VM was hanging on the RMI connection attempt. In my case, the reason why it hanged was that I had the JVisualVM profiler attached to that JVM, but JVisualVM then died with a PermGen OOM. Parts of the profiler were still running in the target JVM, but were hanging due to lack of the profiler frontend; thus any attempt at class loading would hang, which would cause the incoming RMI from the new JVisualVM instance to hang also. Restarting the affected JVM resolved the issue.
Without any thread dumps, I can't say whether your issue was anything like mine or not; but if anyone gets this problem again, collecting thread dumps is a good idea. Whatever the root cause is, restarting all JVMs on your box (e.g. reboot) has a reasonable chance of solving it.
If your JVM is paused on a debugging breakpoint then this will cause VisualVM to hang.
I was facing the same issue - then I came across this post https://github.com/oracle/visualvm/issues/82. Then I killed all the JVM/JDK sessions on my machine. Restarted the visual vm - waited a bit and there you go, its not hanging anymore.
In short visual vm hangs when you switch between IP's.
I was using VisualVM in combination with IntelliJ. For me, VisualVM hanged because I was on a VPN connection. The issue resolved after I turned-off VPN.
Related
I'm experiencing this weird problem that the JVM hangs forever very frequently.
I first observed the problem when my Java IDEs frequently hang the entire system GUI. IntelliJ IDEA hangs on indexing almost every single time upon start. Sometimes it proceeds to resolving dependency but always hangs in the end. When this happens, I can type in gnome-terminal, but the commands can't seem to be executed. I can't launch new applications with Alt-F2 or anything alike.
I had to switch to a text console and "killall -9 java" to kill the IDEA process and get control back. "kill -3 java" won't work. The log file contains nothing related, the thread dump is empty. Once the IDE hung, jstack cannot be attached to the process. "jstack -l pid" also hangs. "jstack -F pid" can't attach to the process. Visualvm hangs as well.
The CPU usage by the Java process is 0% and there is no I/O going on.
I've observed the same behavior when using Eclipse. Sometimes it hangs on start up, sometimes upon saving and sometimes upon running a Java application.
Maven / sbt builds executed within text-only ttys cause the same kind of hang, so I guess it's not a window manager / desktop environment / display driver problem.
I highly suspect it's a file system or I/O issue but I have no clue how to debug that. I've tried fsck with no luck, and my system works perfectly fine when not running java programs.
Things I've ruled out:
Permission issues: running IntelliJ with sudo doesn't help, hangs 100% of the time.
Display driver: I've tried both the Nvidia proprietary driver and nouveau, the open source one. Doesn't help.
Window manager / desktop environment: I use Cinnamon, but I've tried running IntelliJ under Unity. Doesn't help.
Java version: I've tried both Oracle Java 7 and Oracle Java 8. I'll probably try OpenJDK but I doubt it would work.
IntelliJ version: I've tried IntelliJ 13 through 14.1. All exhibited the same behavior.
Limited memory: I have 16G RAM with 16G swap space, so memory should not be a limiting factor.
Kernel log doesn't look suspicious. I can't get any kind of log remotely indicating what went wrong.
Any idea?
UPDATE (2015/04/29): The problem seems to have fixed itself after I accidentally kicked the power cable and cold restarted the computer... Still a mystery but IntelliJ is usable as of now.
Some things to check
- The Java IDEs run best with a lot of ram. I usually ask for at least
8G of memory for my dev workstation.
- Make sure you have a stable version of everything, look for known working versions/configurations on Ubuntu
- You have to manually allocate memory in IntelliJ IDEA versions < 14. For example: How to increase IDE memory limit in IntelliJ IDEA on Mac?
- Besides system logs, run tools like top and see what's happening in terms for cpu and ram when running the IDE
I had similar problems a while ago but with Eclipse. The problem was that there was no swap place at all ;) - obviously it should not be a problem with 16GB of RAM.
Could You post JVM arguments for Intellij? And also I have an idea to create another Intellij installation (eg. go back to 14 version) and see if there is similar problem (also compare JVMs settings between these two).
Edit
Ok so try:
use different JRE/JDK. If the problem disappears it will tell us more.
You are on linux so it makes it easy to monitor several things - you said that there is no CPU utilization or hard I/O. But how do you know that? Maybe it will be informative to have some statistics gathered - eg.jstat for JVM itself or for system information (You think that is I/O problem) try:
iostat -hm -p sda 1
Which will print I/O statistics for sda (if you have different discs change device parameter) in 1sec loops (can be also changed). Start this with system and dump output to the file - maybe there is some kind of 'disaster' happening before JVM hangs. Note: iostat sometimes is not available on system itself (on my Linux Mint is not) - install then package sysstat and the command will be available.
Seem to have fixed after a cold restart from an accidental power loss.. Weirdest problem I have ever seen.
I am running a durable Java program on a remote Ubuntu server, where I have root user rights. After some time, the usage on some CPU cores goes up to 100%. The logs show nothing suspicious and the application still works, but with reduced throughput.
How can I debug the JVM so that I can find out the cause of this, while it's still running?
One option is VisualVM, which is included in the JDK starting with Java 1.6. I have found it useful in some situations in the past.
You may connect to local applications or remote applications.
To connect to a remote app, run jstatd on your remote server, and then run VisualVM locally and enter your server's IP address. You should be provided with a list of running Java applications including the one you wish to explore. If you have any trouble listing your application, good documentation is available at the VisualVM website.
Connect to the process with jvisualvm
This tool will allow you to connect to the running process and view all of the threads and their state. This could show you which thread is the culprit merely by looking at which one is awake all the time. You can do a thread dump to see the stack trace for each thread and see what each thread is doing.
It's a very powerful tool for just this kind of debugging. It is distributed with the JDK only, so you will need more than just the JVM runtime installed to have access. Be sure you install the same version of the JDK that the JVM is running.
You will need to have your X display forwarded for this to work.
If you want to see the stack trace on linux just issue kill -SIGQUIT <java-program-pid>. That is one way to see where the the code is executing.
We have trouble with Tomcat 5.5 which stops at night on our production servers (Linux CentOS 4.8) and we have no idea why it stops...
There is no Tomcat's log in catalina.out or any application's log.
We tried different things to find why the server stops:
configure Tomcat to be able to generate a core dump
instrument System.exit() method with javassist to find if the method was called
add a shutdown hook to the JVM (with Runtime.getRuntime().addShutdownHook())
None of them worked, we have no core dump, the Exit method and the shutdown hook are not called.
My conclusions are:
The VM is not terminated properly but crash without any log.
Any idea or log to read to find why Tomcat stops?
1) Make sure you know where stderr is redirected and check if anything got printed there.
2) Check the memory limits on Tomcat and how much free memory does the system have. Review the Linux system logs under /var/log to see if anything suspicious happened during the time. For example, kernel can randomly kill a process (almost) without a trace if the system is running low on memory.
We've ran 5.5 in production for years and never had any unexplained shutdowns, FWIW.
This worked for me.
As suggested here in other answers checked system logs in /var/log/messages but permission denied for me. So, I used dmesg command instead and got this in the logs
"Out of memory: Kill process 14606 (java) score 106 or sacrifice child".
In the output I also noticed Swap Memory free 0 K. Ran top command to confirm the same. So, somehow there was a high memory usage which caused the OS to kill my tomcat process.
After spending hours finally got the reason.
ps -ef | grep tomcat showed that there were several tomcat processes running for the same application. It seems that, earlier tomcat shutdowns might not have been completed successfully and due to some reason the processes were not killed even after the shutdown, which was causing the high memory usage.
So, killed all running tomcat processes using kill. SWAP memory got freed.
Started tomcat again, worked fine. :)
Tomcat 7 has an option inside catalina to prevent the System.exit class call or something similar: http://ci.apache.org/projects/tomcat/tomcat7/docs/security-manager-howto.html .
Maybe there's a similar option for the 5.5 version. Try the documentation.
There are options to redirect the output to the same console that you use to start Tomcat. This information is redirected to logs when you execute on Unix based systems, on Windows, it remains with the console if not redirected.
Most probably there is a stack-overflow exception. This is typical behavior of Tomcat when it happens. For example, you're trying to serialize to JSON or XML beans with cyclic dependencies (but without handling of the cycles).
Everytime I had this issue (several times) it always has been this one. All other stops are usually logged properly (like OutOfMemory etc).
This type of stops leaves no trace anywhere.
In Java profiling, it seems like all (free) roads nowadays lead to the VisualVM profiler included with JDK6. It looks like a fine program, and everyone touts how you can "attach it to a running process" as a major feature. The problem is, that seems to be the only way to use it on a local process. I want to be able to start my program in the profiler, and track its entire execution.
I have tried using the -Xrunjdwp option described in how to profile application startup with visualvm, but between the two transport methods (shared memory and server), neither is useful for me. VisualVM doesn't seem to have any integration with the former, and VisualVM refuses to connect to localhost or 127.0.0.1, so the latter is no good either. I also tried inserting a simple read of System.in into my program to insert a pause in execution, but in that case VisualVM blocks until the read completes, and doesn't allow you to start profiling until after execution is under way. I have also tried looking into the Eclipse plugin but the website is full of dead links and the launcher just crashes with a NullPointerException when I try to use it (this may no longer be accurate).
Coming from C, this doesn't seem like a particularly difficult task to me. Am I just missing something or is this really an impossible request? I'm open to any kinds of suggestions, including using a different (also free) profiler, and I'm not averse to the command line.
Consider using HPROF and opening the data file with a tool like HPjmeter - or just reading the resulting text file in your favorite editor.
Command used: javac -J-agentlib:hprof=heap=sites Hello.java
SITES BEGIN (ordered by live bytes) Fri Oct 22 11:52:24 2004
percent live alloc'ed stack class rank self accum bytes objs bytes objs trace name
1 44.73% 44.73% 1161280 14516 1161280 14516 302032 java.util.zip.ZipEntry
2 8.95% 53.67% 232256 14516 232256 14516 302033 com.sun.tools.javac.util.List
3 5.06% 58.74% 131504 2 131504 2 301029 com.sun.tools.javac.util.Name[]
4 5.05% 63.79% 131088 1 131088 1 301030 byte[]
5 5.05% 68.84% 131072 1 131072 1 301710 byte[]
HPROF is capable of presenting CPU usage, heap allocation statistics,
and monitor contention profiles. In addition, it can also report
complete heap dumps and states of all the monitors and threads in the
Java virtual machine.
The best way to solve this problem without modifying your application, is to not use VisualVM at all. As far as other free options are concerned, you could use either Eclipse TPTP or the Netbeans profiler, or whatever comes with your IDE.
If you can modify your application, to suspend it's state while you setup the profiler in VisualVM, it is quite possible to do so, using the VisualVM Eclipse plugin. I'm not sure why you are getting the NullPointerException, since it appears to work on my workstation. You'll need to configure the plugin by providing the path to the jvisualvm binary and the path of the JDK; this is done by visiting the VisualVM configuration dialog at Windows -> Preferences -> Run/Debug - > Launching -> VisualVM Configuration (as shown in the below screenshot).
You'll also need to configure your application to start with the VisualVM launcher, instead of the default JDT launcher.
All application launches from Eclipse, will now result in VisualVM tracking the new local JVM automatically, provided that VisualVM is already running. If you do not have VisualVM running, then the plugin will launch VisualVM, but it will also continue running the application.
Inferring from the previous sentence, it is evident that having the application halt in the main() method before performing any processing is quite useful. But, that is not the main reason for suspending the application. Apparently, VisualVM or its Eclipse plugin does not allow for automatically starting the CPU or memory profilers. This would mean that these profilers would have to be started manually, thereby necessitating the need to suspend the application.
Additionally, it is worth noting that adding the flags: -Xrunjdwp:transport=dt_socket,address=8000,server=y,suspend=y to the JVM startup will not help you in the case of VisualVM, to suspend the application and setup up the profilers. The flags are meant to help you in the case of profilers that can actually connect to the open port of the JVM, using the JDWP protocol. VisualVM does not use this protocol and therefore you would have to connect to the application using JDB or a remote debugger; but that would not resolve the problem associated with profiler configuration, as VisualVM (at least as of Java 6 update 26) does not allow you to configure the profilers on a suspended process as it simply does not display the Profiler tab.
This is now possible with the startup profiler plugin to VisualVM.
The advice with -Xrunjdwp is incorrect. It just enables debugger and with suspend=y it waits for debugger to attach. Since VisualVM is not debugger, it does not help you. However inserting System.in or Thread.sleep() will pause the startup and allows VisualVM to attach to your application. Be sure to read Profiling with VisualVM 1 and Profiling with VisualVM 2 to better understand profiler settings. Note also that instead of profiling, you can use 'Sampler' tab in VisualVM, which is more suitable for profiling entire java program execution. As other mentioned you can also use NetBeans Profiler, which directly support profiling of the application startup.
I've been running Tomcat 5.5 with Java 1.4 for a while now with a huge webapp. Most of the time it runs fine, but sometimes it will just hang, with no exception generated, and no apparant way of getting it to run again other than re-starting Tomcat. The tomcat instance is allowed a gigabyte of memory on the heap, but rarely exceeds 300 MB. Has anyone else run into this issue, and is there a solution for it?
For clarification: I determined how much memory it is using via Task Manager and via Eclipse (I've also tried running it outside of Eclipse, but get the same problem eventually, though it takes a little longer). With Eclipse, I look at the memory allocated via its little (optional) memory pane and the amount allocated to javaw.exe via the task manager. I use the sysdeo? tomcat plugin for Eclipse.
For any jvm process, force a thread dump. In windows, this can be done with CTRL-BREAK, I believe, in the console window.
In *nix, it is almost always "kill -3 jvm-pid".
This may show if you have threads waiting on db connection pool/thread pool, etc.
Another thing to check out is how many connections you have currently to the JVM -- either use NETSTAT or SysInternals utility such as tcpconn/tcpview (google it).
Also, try to run with the verbose:gc JVM flag. For Sun's JVM, run like "java -verbose:gc". This will show your garbage collections. If it is collecting a lot (FULL COLLECTIONS, expecially) then you probably have a memory leak. The full collections are costly, especially on large heaps like that.
How are you determining that only 300mb are being used?
It sounds like you're hitting a deadlock.
If you can reproduce it in a dev environment then try attaching a debugger once it's happened. Take a look at your threads and see if you have any deadlocks.
If you can't get a debugger to attach you should be able to generate a thread dump, as Dustin pointed out.
Try increasing the logging sensitivity for the Tomcat application server.
http://tomcat.apache.org/tomcat-5.5-doc/logging.html
You can increase the sensitivity to FINEST or ALL for most of them for a few days and see if that helps you catch anything.
I agree with creating multiple thread dumps and viewing them though this: Thread Dump Analyzer