We Keep Coding

How do I debug hanging threads in the JVM?

I am running a durable Java program on a remote Ubuntu server, where I have root user rights. After some time, the usage on some CPU cores goes up to 100%. The logs show nothing suspicious and the application still works, but with reduced throughput.
How can I debug the JVM so that I can find out the cause of this, while it's still running?

One option is VisualVM, which is included in the JDK starting with Java 1.6. I have found it useful in some situations in the past.
You may connect to local applications or remote applications.
To connect to a remote app, run jstatd on your remote server, and then run VisualVM locally and enter your server's IP address. You should be provided with a list of running Java applications including the one you wish to explore. If you have any trouble listing your application, good documentation is available at the VisualVM website.

Connect to the process with jvisualvm
This tool will allow you to connect to the running process and view all of the threads and their state. This could show you which thread is the culprit merely by looking at which one is awake all the time. You can do a thread dump to see the stack trace for each thread and see what each thread is doing.
It's a very powerful tool for just this kind of debugging. It is distributed with the JDK only, so you will need more than just the JVM runtime installed to have access. Be sure you install the same version of the JDK that the JVM is running.
You will need to have your X display forwarded for this to work.

If you want to see the stack trace on linux just issue kill -SIGQUIT <java-program-pid>. That is one way to see where the the code is executing.

Profile Entire Java Program Execution in VisualVM

In Java profiling, it seems like all (free) roads nowadays lead to the VisualVM profiler included with JDK6. It looks like a fine program, and everyone touts how you can "attach it to a running process" as a major feature. The problem is, that seems to be the only way to use it on a local process. I want to be able to start my program in the profiler, and track its entire execution.
I have tried using the -Xrunjdwp option described in how to profile application startup with visualvm, but between the two transport methods (shared memory and server), neither is useful for me. VisualVM doesn't seem to have any integration with the former, and VisualVM refuses to connect to localhost or 127.0.0.1, so the latter is no good either. I also tried inserting a simple read of System.in into my program to insert a pause in execution, but in that case VisualVM blocks until the read completes, and doesn't allow you to start profiling until after execution is under way. I have also tried looking into the Eclipse plugin but the website is full of dead links and the launcher just crashes with a NullPointerException when I try to use it (this may no longer be accurate).
Coming from C, this doesn't seem like a particularly difficult task to me. Am I just missing something or is this really an impossible request? I'm open to any kinds of suggestions, including using a different (also free) profiler, and I'm not averse to the command line.

Consider using HPROF and opening the data file with a tool like HPjmeter - or just reading the resulting text file in your favorite editor.
Command used: javac -J-agentlib:hprof=heap=sites Hello.java
SITES BEGIN (ordered by live bytes) Fri Oct 22 11:52:24 2004
percent live alloc'ed stack class rank self accum bytes objs bytes objs trace name
1 44.73% 44.73% 1161280 14516 1161280 14516 302032 java.util.zip.ZipEntry
2 8.95% 53.67% 232256 14516 232256 14516 302033 com.sun.tools.javac.util.List
3 5.06% 58.74% 131504 2 131504 2 301029 com.sun.tools.javac.util.Name[]
4 5.05% 63.79% 131088 1 131088 1 301030 byte[]
5 5.05% 68.84% 131072 1 131072 1 301710 byte[]
HPROF is capable of presenting CPU usage, heap allocation statistics,
and monitor contention profiles. In addition, it can also report
complete heap dumps and states of all the monitors and threads in the
Java virtual machine.

The best way to solve this problem without modifying your application, is to not use VisualVM at all. As far as other free options are concerned, you could use either Eclipse TPTP or the Netbeans profiler, or whatever comes with your IDE.
If you can modify your application, to suspend it's state while you setup the profiler in VisualVM, it is quite possible to do so, using the VisualVM Eclipse plugin. I'm not sure why you are getting the NullPointerException, since it appears to work on my workstation. You'll need to configure the plugin by providing the path to the jvisualvm binary and the path of the JDK; this is done by visiting the VisualVM configuration dialog at Windows -> Preferences -> Run/Debug - > Launching -> VisualVM Configuration (as shown in the below screenshot).
You'll also need to configure your application to start with the VisualVM launcher, instead of the default JDT launcher.
All application launches from Eclipse, will now result in VisualVM tracking the new local JVM automatically, provided that VisualVM is already running. If you do not have VisualVM running, then the plugin will launch VisualVM, but it will also continue running the application.
Inferring from the previous sentence, it is evident that having the application halt in the main() method before performing any processing is quite useful. But, that is not the main reason for suspending the application. Apparently, VisualVM or its Eclipse plugin does not allow for automatically starting the CPU or memory profilers. This would mean that these profilers would have to be started manually, thereby necessitating the need to suspend the application.
Additionally, it is worth noting that adding the flags: -Xrunjdwp:transport=dt_socket,address=8000,server=y,suspend=y to the JVM startup will not help you in the case of VisualVM, to suspend the application and setup up the profilers. The flags are meant to help you in the case of profilers that can actually connect to the open port of the JVM, using the JDWP protocol. VisualVM does not use this protocol and therefore you would have to connect to the application using JDB or a remote debugger; but that would not resolve the problem associated with profiler configuration, as VisualVM (at least as of Java 6 update 26) does not allow you to configure the profilers on a suspended process as it simply does not display the Profiler tab.

This is now possible with the startup profiler plugin to VisualVM.

The advice with -Xrunjdwp is incorrect. It just enables debugger and with suspend=y it waits for debugger to attach. Since VisualVM is not debugger, it does not help you. However inserting System.in or Thread.sleep() will pause the startup and allows VisualVM to attach to your application. Be sure to read Profiling with VisualVM 1 and Profiling with VisualVM 2 to better understand profiler settings. Note also that instead of profiling, you can use 'Sampler' tab in VisualVM, which is more suitable for profiling entire java program execution. As other mentioned you can also use NetBeans Profiler, which directly support profiling of the application startup.

How create a memory dump of a Java process run as windows service?

How create a memory dump of a Java process run as windows service? With tools like jVisualVM, jstack or jconsole I can not see the process because the service run with a system account.
Edit: With jVisualVM i can see the process but it show only a "Not supported for this JVM."

To use the tools, you need to run then as the same user as the service. If you don't want this security feature, you need to set up JMX for the server will allow remote access (which will also work on the same box as a different user)
If you use taskmgr to create a low level memory dump you will have trouble finding any tools which can make sense of the output. You need to use a Java tool to trigger the dump.

If you are using Windows 7 / Vista, run taskmgr as administrator, find your process, right click, "Create Memory Dump". Your other tools may work correctly if they are run as Administrator too. The process needs to be elevated to take a memory dump of a process that isn't yours.

Utility to view threads in a Java process

I've got a java application on a windows machine which pins the CPU from time to time. Would like to run a utility to get an idea of how many threads etc that application is creating. Is there such a utility?

jconsole is included with the jdk and includes thread/memory/cpu monitoring

You could try jstack, should be part of the jdk.

I've always been partial to YourKit.
There are lots of others though, both open source and commercial. If you do a google search for "java profilers" you should be able to find one to your liking.

get a look at jvisualvm (in the jdk)

Can I connect to a jsadebugd process on a remote machine from Eclipse/IDE debugger?

I have a java process running on a Linux box, but it cannot be started/re-started in debug mode. I read about the jsadebugd command and started the daemon, but I want to connect to it from an IDE(possibly Eclipse) debugger or maybe YourKit or jconsole. How can I do this? The remote JVM is 1.6+.

I assume since you mentioned Yourkit and other tool that what you really want to do is look at object state inside your applications. There are a couple of options, though I don't think it gets you the ability to set break-points like Eclipse or another debugger would (though I'd be intersted in hearing what restricts you from starting the process in debug mode - technical? process?)
have you tried connecting with: VisualVM? I don't believe you need to start in debug mode, and it will give you the ability to navigate the object graph, and inspect objects. You can even use it to take heapdumps and do some ad-hoc analysis through them using OQL queries.
If you're running on JDK6+ - have you tried instrumenting with btrace? Some notes from the sailfin team sounded promising, and like DTrace in Solaris, this seem like it would be useful in a variety of situations.

I am not sure if I understand your restrictions correctly but you can start JVM with debugging server (serving JDWP) enabled. See for example "Sun VM Invocation Options" in http://java.sun.com/j2se/1.4.2/docs/guide/jpda/conninv.html
Then you can connect your Eclipse debugger to running JVM. See "Remote debugging" section here http://www.ibm.com/developerworks/library/os-ecbug/

I'm not sure that this is exposed by Eclipse, but here's how to do it with jdb:
http://java.sun.com/javase/6/webnotes/trouble/TSG-VM/html/tooldescr.html#gbmog