I'm wondering if it's possible to get a handle on running instances of a given class. The particular issue that triggered this was an application that doesn't exit nicely because of a number of running threads.
Yes, I know you can daemonize the theads, and they won't then hold up the application exit. But it did get me to wondering if this was possible. The closest thing I can is the classloaders (protected!) findLoadedClass, although you'd have to run through your own classloader to do this.
On a related note, is this how profiling tools manage to track object handles? by running through their own custom classloaders? or is there some nice tricky way that I'm not seeing?
You can indeed get a stack trace of all running Threads dumped to the stdout by using kill -QUIT <pid> on a *NIX like OS, or by running the application in a Windows console and pressing Ctrl-Pause (as another poster notes.)
However, it seems like you're asking for programmatic ways to do it. So, assuming what you really want is the set of all Threads who's current call stacks include one or more methods from a given class...
The best thing I can find that doesn't involve calls into the JVMTI is to inspect the stacks of all running threads. I haven't tried this, but it should work in Java 1.5 and later. Keep in mind that this is, by definition, not AT ALL thread-safe (the list of running threads - and their current stack traces - are going to constantly change underneath you...lots of paranoid stuff would be necessary to actually make use of this list.)
public Set<Thread> findThreadsRunningClass(Class classToFindRunning) {
Set<Thread> runningThreads = new HashSet<Thread>();
String className = classToFindRunning.getName();
Map<Thread,StackTraceElement[]> stackTraces = Thread.getAllStackTraces();
for(Thread t : stackTraces.keySey()) {
StackTraceElement[] steArray = stackTraces.get(t);
for(int i = 0;i<steArray.size();i++) {
StackTraceElement ste = steArray[i];
if(ste.getClassName().equals(className)) {
runningThreads.add(t);
continue;
}
}
}
return runningThreads;
}
Let me know if this approach works out for you!
From this page,
A Java profiler uses a native interface to the JVM (the JVMPI for Java <=1.4.2 or the JVMTI for Java >=1.5.0) to get profiling information from a running Java application.
This amounts to some Sun supplied native code that gives a profiler hooks into the JVM.
If you only need a quick stack trace of your running application to find out which threads are blocking the JVM exit, you can send it the QUIT signal.
$ kill -QUIT <pid of java process>
Under Windows you need to run the application in a console window (using java.exe, not javaw.exe), then you can press Ctrl-Pause to generate the stack dump. In both cases it’s written to stdout.
I use Visual VM to monitor the JVM. Has lots of features, give it a try.
I think you need a Java thread dump. This should list all threads with what they are doing at the moment. I had a similar problem myself, which a thread dump helped solve (Quartz scheduler threads, which did not exit when Tomcat finished).
Related
Situation:
I have a Java1.7 process running in CentOS6 with multiple threads. The process currently halts (i.e., stuck in some kind of loop or waiting function). Due to the complexity nature of the program, it is difficult to do a routine debug in, for instance, Eclipse (more explanation in the background section below). Therefore, I'd like to debug the code by tracing the current running stack.
Question:
Is there a Linux command that would allow me to print the stack to identify the thread/method that is currently running such that I can find which method is causing the halt?
Background:
The reasons for not being able to debug in Eclipse:
It is a MapReduce program typically run on multiple computers.
Even if I use run on one computer, it still involves multiple threads running simultaneously.
Most importantly, the "halting bug" occurs randomly (i.e., not being able to reproduce). So my best shot is to identify the current running method that caused the bug.
P.S. My approach may be completely wrong, so feel free to correct me and point me in the right direction.
Thanks for your help.
You can use JStack to get the current thread dump. It should give you currently running threads and their stack traces.
It will even do more for you - should there be any deadlocks present it will tell you about them.
Apart from that you can also use JVisualVM to monitor your application in real time (you can check threads in real time there and take thread dumps from it).
From RedHat:
Following are methods for generating a Java thread dump on Unix:
1) Note the process ID number of the Java process (e.g. using top, a
grep on ps -axw, etc.) and send a QUIT signal to the process with the
kill -QUIT or kill -3 command. For example:
kill -3 JAVA_PID
I want to exit a java process and free all the resources before it finishes its normal running, if a certain condition is meet. I dont however want to quit JVM, as I have other java programs running at the same time. Does return; do the above, or is there a better way to do it?
Thanks.
There is one JVM process per running Java application. If you exit that application, the process's JVM gets shut down. However, this does not affect other Java processes.
You need to understand the JVM mechanism and clarify the terminology.
Let's use the following as datum for the terminology.
Threads are divisions of concurrently processed flows within a process.
A process is an OS level thread. The OS manages the processes. A process is terminated by sending a termination signal to the OS management. The signal may be sent by the process itself or by another process that has the applicable privilege.
Within a process, you can create process level threads. Process level threads are normally facilitated by the process management of the OS, but they are initiated by the process and terminated by the process. Therefore, process level threads are not the same as processes.
An application is a collection of systems, programs and/or threads that cooperate in various forms. A program or process within an application may terminate without terminating the whole application.
Within the context of JVM terminology, program may be one of the following.
A program is run per JVM process. Each program consumes one JVM process and is invoked by supplying the classpath of java bytecode and specifying the main entry point found in the classpath. When you terminate a java program, the whole jvm process that ran that program also terminates.
A program is run per process level thread. For example, an application run within a tomcat or JEE server is run as a thread within the JEE process. The JEE process is itself a program consuming one JVM process. When you terminate an application program, the JEE process does not terminate.
You may initiate process level threads within a java program. You may write code that terminates a thread but that would not terminate the process (unless it is the last and only running thread in the process). The JVM garbage collection would take care of freeing of resources and you do not need to free resources yourself after a process level thread is terminated.
The above response is simplified for comprehension. Please read up on OS design and threading to facilitate a better understanding of processes and the JVM mechanism.
If the other threads running concurrently are not daemon threads, leaving main will not terminate the VM. The other threads will continue running.
I completely missed the point though.
If you start each program in a separate JVM, calling System.exit() in one of them will not influence the others, they're entirely different processes.
If you're starting them through a single script or something, depending on how it is written, something else could be killing the other processes. Without precise information about how you start these apps, there's really no telling what is going on.
#aix's answer is probably apropos to your question. Each time you run the java command (or the equivalent) you get a different JVM instance. Calling System.exit() in one JVM instance won't cause other JVM instances to exit. (Try it and see!)
It is possible to create a framework in which you do run multiple programs within the same JVM. Indeed this is effectively what you do when you run a "bean shell". The same sort of thing happens when your "programs" are services (or webapps, or whatever you call them) running in some application server framework.
The bad news is that if you do this kind of thing, there is no entirely reliable way make an individual "program" go away. In particular, if the program is not designed to be cooperative (e.g. if it doesn't check for interrupts), you will have to resort to the DEPRECATED Thread.stop() method and friends. And those methods can have nasty consequences for the JVM and the other programs running in it.
In theory, the solution to that problem is to use Isolates. Unfortunately, I don't think that any mainstream JVMs support Isolates.
Some common usecases leading these kind of requirements can be solved through tools like Nailgun, or Drip.
Nailgun allows you to run what appears to be multiple independent executions of a commandline program, but they all happen in the same JVM. Therefore repeated JVM start-up time does not have to be endured. If these execution interact with global state, then the JVM will get polluted in time and things start to break up.
Drip will use a new JVM for each execution, but it always keeps a precreated JVM with the correct classpath and options ready. This is less performant, but it can guarantee correctness through isolation.
I am running a Java process with Xmx2000m, the host OS is linux centos, jdk 1.6 update 22. Lately I have been experiencing a weird behavior in the process, it becomes totally unresponsive with no apparent reason, no logs, no errors, nothing.. I am using jconsole to monitor the processor, heap and Perm memory are not full, threads and loaded classes are not leaking..
Explanation anyone?
I doubt anyone can give you an explanation since there are lots of possible reasons and not nearly enough information. However, I suggest that you jstack the process once it's hung to figure out what the threads are doing, and take it from there. It sounds like a deadlock or thrashing of some sort.
Do a thread dump. If you have access to the foreground process on Linux, use ctrl-\. Or use jstack to dump stack remotely. Or you can actually poke it through JMX via jconsole at MBeans/java.lang/Threading/Operations/dumpAllThreads.
Without knowing more about your app, it's hard to speculate about the cause. Presumably your threads are either a) blocked or b) exited. If they are blocked, they could be waiting for I/O on a database or other operation OR they could be waiting on a lock or monitor (deadlocked). If a deadlock exists, the thread dump will tell you which threads are deadlocked, which lock, and (in Java 6) annotate the stack with where locks have been taken. You can also search for deadlocks with the JMX method, available through jconsole at MBeans/java.lang/Threading/Operations/find[Monitor]DeadlockedThreads().
Or your threads may have received unhandled exceptions and exited. Check out Thread's uncaughtExceptionHandlers or (better) use Executors in java.util.concurrent.
And finally, the other classic source of pauses in Java is GC. Run with -verbose:gc and other GC flags to see if it's doing a full GC collection. You can also turn this on dynamically in jconsole by flipping the flag at MBeans/java.lang/Memory/Attributes/Verbose.
Agree with aix, but would like to add a couple of recommendataions.
1. check your system. Run top to see whether the system itself is healthy, CPU is not 100% and memory is available. If not, fix this.
2. application may freeze as a result of dead lock. Check this.
Ok here are some updates I wanted to share:
There is an incompatability between NTPL (Linux’s new thread library) and the Java 1.6+ JVM. A random bug causes the JVM to hang and eat up 100% CPU.
To work around it set LD_ASSUME_KERNEL=2.4.1 before running the JVM, export LD_ASSUME_KERMEL=2.4.1 . This disables NTPL: problem solved!
But for compatibility reasons, I'm still looking for a solution that uses NTPL.
Threads can be traced using jvisualvm and jconsole, and deadlocks can be avoided too. Note that there are several network services each with separate thread pools, and they all become unreachable.
Check the jvisualvm of the process right before the crash.
http://www.jadyounan.com/wp-content/uploads/2010/12/process.png
Could you elaborate more on what you are doing ? 2000 for memory is rather a lot.
I've looked through couple of articles here such as:
java stack dump on windows
Thread dump programmatically /JDI (Java Debugger Interface)
But didnt catch the exact answer.
The problem:
There is a Java5 Application on Windows that's runs as a service (so we dont have a console where we are able to use Ctrl+Break for Dumping).
And sometimes Application hangs and we need a thread dump.
We've tried "jstack" but it doesnt work in our env (we found out that its Java6 only compatible).
So we made a C++ app that calls thread dump via .dll call method attaching to the Java app process, and because of this it needs Local Admin rights, that is not so good.
So we'd like other options that works without admin rights and works with Java 5 without lots of rework of existing code.
Method with Printing in LOOP thread dumps (Thread.getAllStackTraces()) is not an option because we need to refactor lots of applications in order to make it work.
So that just an util that works from "outside" of applications would be a best option.
Thanks in advance!
One option is to dump all the information using jmap, and then analyzing it using other tool.
jmap -dump:format=b,file=<filename>.hprof <jvm_pid>
I am not sure, buy I think it will work on Java 5.
References:
HPjmeter-like graphical tool to view -agentlib:hprof profiling output
You can attach to the process with JConsole to detect deadlocks and get stack traces of the threads. For more information, see here: http://java.sun.com/developer/technicalArticles/J2SE/jconsole.html
I like to generate a thread dump programmatically. I've learned that there a basically two ways to do it:
Use the "Java Virtual Machine Tool Interface" JVM-TI
Use the higher abstracted "Java Debugger Interface" JDI
For the JVM-TI I was able to find some useful information, but I would have to write a JNI-DLL which, at least for the moment, I would like to avoid. With the JDI I can use Java and it seems I'm able to use it from within the application. But I wasn't able to find some kind of tutorial or HOWTO for it. The only documentation I could find, were the Java-Docs http://java.sun.com/j2se/1.5.0/docs/guide/jpda/jdi/ which isn't very helpful, because it doesn't show me how to use this classes.
So, does anybody know of a good tutorial/book I could read?
Thx for any help!
There is a third way: Thread.getAllStackTraces()
http://java.sun.com/javase/6/docs/api/java/lang/Thread.html#getAllStackTraces()
This is much easier than the debugger interface...
You can get just about all the Thread info you need including deadlocks from http://java.sun.com/javase/6/docs/api/java/lang/management/ThreadMXBean.html
Thread.getAllStackTraces() dumps only the execution trace of all the threads, but doesn't give the information of object locks that have been obtained by a particular thread or the lock on which a particular thread has been waiting. Basically, we'll not be able to nail down deadlocks with this.
Did you consider the remote alternative ? I.e. VisualVM
jps and jstack are also useful tools included in JDK 5, providing a quick command line method for obtaining stack traces of all current threads.
This article suggest JDI is also used as a remote tool.
So I am not sure you can triggers a thread dump within your own program, instead you find a way to send to yourself a SIGQUIT signal (kill -3) on Unix platforms, or press the Ctrl-\ key on Unix or Ctrl-Break on Windows platforms.
Plus, JDI wasn't intended to be used to debug the same process in which the JDI client is running. Still this thread I just linked to is the closest I have found to actually use JDI within the same program.