Running kill -QUIT on a Unix system will trigger a thread dump. I know this because I have done this hundreds of times.
However, another developer tells me he has seen this "crash the JVM" and using twiddle or the JMX API is "safer".
I'm struggling to find any references online to kill -QUIT behaving this way.
Can anyone confirm that it could actually kill the java process/cause the JVM to quit?
(Obviously one way for it to do this would if someone didn't correctly type "-QUIT" :-))
In 12 years I have never seen kill -QUIT crash a JVM. But as Disco 3 says, if you're doing a thread dump while the JVM is in distress (which is when you usually do thread dumps), it may (possibly?) crash with an OutOfMemoryError. But anything could crash a JVM in that situation. I wouldn't hesitate to use kill -QUIT, but you may find jstack more useful because it will dump the thread dump to your stdout rather than the JVM's.
Related
Situation:
I have a Java1.7 process running in CentOS6 with multiple threads. The process currently halts (i.e., stuck in some kind of loop or waiting function). Due to the complexity nature of the program, it is difficult to do a routine debug in, for instance, Eclipse (more explanation in the background section below). Therefore, I'd like to debug the code by tracing the current running stack.
Question:
Is there a Linux command that would allow me to print the stack to identify the thread/method that is currently running such that I can find which method is causing the halt?
Background:
The reasons for not being able to debug in Eclipse:
It is a MapReduce program typically run on multiple computers.
Even if I use run on one computer, it still involves multiple threads running simultaneously.
Most importantly, the "halting bug" occurs randomly (i.e., not being able to reproduce). So my best shot is to identify the current running method that caused the bug.
P.S. My approach may be completely wrong, so feel free to correct me and point me in the right direction.
Thanks for your help.
You can use JStack to get the current thread dump. It should give you currently running threads and their stack traces.
It will even do more for you - should there be any deadlocks present it will tell you about them.
Apart from that you can also use JVisualVM to monitor your application in real time (you can check threads in real time there and take thread dumps from it).
From RedHat:
Following are methods for generating a Java thread dump on Unix:
1) Note the process ID number of the Java process (e.g. using top, a
grep on ps -axw, etc.) and send a QUIT signal to the process with the
kill -QUIT or kill -3 command. For example:
kill -3 JAVA_PID
i have a java multi-threaded program that is running. i am running it on a tomcat server. when the threads are still running, some executing tasks, some still waiting for some thing to return and all kinds of things, assume i stop the server all of a sudden in this scenario.. when i do i get a warning on the tomcat terminal saying a thread named x is still running and the server is being stopped so this might lead to a memory leakage. what is the OS actually trying to tell me here? can someone help me understand this?? i am running this program on my system several times and i have stopped the server abruptly 3 times and i have seen this message when ever i do that. have i runined my server? (i mean my system). did i do something very dangerous????
please help.
Thanks in advance!
when i do i get a warning on the tomcat terminal saying a thread named x is still running and the server is being stopped so this might lead to a memory leakage. what is the OS actually trying to tell me here?
Tomcat (not the OS) is surmising from this extra thread that some part of your code forked a thread that may not be properly cleaning itself up. It is thinking that maybe this thread is forked more than once and if your process runs for a long time, it could fill up usable memory which would cause the JVM to lock up or at least get very slow.
have i ruined my server? (i mean my system). did i do something very dangerous????
No, no. This is about the tomcat process itself. It is worried that this memory leak may stop its ability to do its job as software -- nothing more. Unless you see more than one thread or until you see memory problems with your server (use jconsole for this) then I would only take it as a warning and a caution.
It sounds like your web server is forking processes which are not terminated when you stop the server. Those could lead to a memory leak because they represent processes that will never die unless you reboot or manually terminate them with the kill command.
I doubt that you will permanently damage your system, unless those orphaned processes are doing something bad, but that would be unrelated to you stopping the server. You should probably do something like ps aux | grep tomcat to find the leftover processes and do the following
Kill them so they don't take up more system resoures.
Figure out why they are persisting when the server is stopped. This sounds like a misbehaving server.
Is there a possibility that kill -3 / quit PID prints nothing i.e. an empty thread dump? We heard a story from a support engineer and was wondering if some experts could validate.
This is on Java 6_26 on RHEL 5
I have only seen this when the server redirects to stdout, like JBoss, and stdout has been redirected to /dev/null because whoever set up the server thought that everything going to stdout was already going to a named log file.
The console output of JVM thread dump on some servers is redirected to a log file. In case of Tomcat Server it is usually Catalina.out.
I have seen the behavior you describe in a standalone Java application (Oracle JDK 1.6.20+, Linux), but I can't tell how to reproduce this behavior consistently. It may have been after an OutOfMemoryError in one of the threads but I'm not sure any more.
I also think that what I got was not just an empty dump, but that the command actually froze and didn't return me to the shell until I pressed ctrl+C after waiting for a while. Either way, I'm sure that the behavior of jstack was exactly the same as of kill -3. When it happened, the app was in such bad shape that it didn't react to normal kill and only kill -9 worked on it. There were no redirections and under normal circumstances the app reacted to kill -3 as it should.
It is possible to kill a thread that is in state RUNNING in a non programatically way?
I know that top command in *nix can show threads. Can I kill the thread in the OS?
I'd like to know if there is a way to link a thread to a process so I can kill only that specific thread and not the application.
We had a bug in our code that kept a thread in state RUNNING in a synchronized method. The thread kept the lock on the object "hanging" the application.
The bug is fixed. But I wonder if is possible.
The short answer is "maybe, but you should not and most of the time it won't work either".
The long answer is:
"Maybe..."
Some JVM implementation map java threads to OS threads and some do not. If the JVM does a mapping to a native OS thread, you might be able to kill that thread with some process tool that the OS provides (like kill on *nix). If the JVM does green threads, meaning it doesn't map a Java thread to an OS level thread, then you are basically out of luck using OS level tools. Luckily only very few JVM implementations do this. An approach that can be used regardless in which way the JVM organizes it's threads, is using the java debugger. This article describes the procedure of doing it: http://www.rhcedan.com/2010/06/22/killing-a-java-thread/.
"but you should not do it"
Killing a thread on the OS level will almost certainly leave the JVM in an undefined state (read "jvm might crash or delete all files on your disk or do whatever it fricking pleases to do"). Even when going the debugger way, only a very small amount of java applications (read "no application made on this planet") will properly handle the event that an outside application is killing one of it's threads. As a result these applications will be put in an undefined state (read "application might crash or delete all files on your disk or do whatever it fricking pleases to do").
"and most of the time it won't work either"
If the thread is really stuck with some blocked IO etc, then killing the thread won't work, it will just not respond. If a program is stuck it's probably better to kill the whole program, find the issue with the program and fix it instead of killing a single thread.
For all your doubts on killing a thread, refer this:
http://download.oracle.com/javase/1.4.2/docs/guide/misc/threadPrimitiveDeprecation.html
On linux, there is a tkill(int tid, int sig) command, similar to kill.
On windows, ProcessExplorer can do it from gui, don't know if there is anything with cli.
I am running a Java process with Xmx2000m, the host OS is linux centos, jdk 1.6 update 22. Lately I have been experiencing a weird behavior in the process, it becomes totally unresponsive with no apparent reason, no logs, no errors, nothing.. I am using jconsole to monitor the processor, heap and Perm memory are not full, threads and loaded classes are not leaking..
Explanation anyone?
I doubt anyone can give you an explanation since there are lots of possible reasons and not nearly enough information. However, I suggest that you jstack the process once it's hung to figure out what the threads are doing, and take it from there. It sounds like a deadlock or thrashing of some sort.
Do a thread dump. If you have access to the foreground process on Linux, use ctrl-\. Or use jstack to dump stack remotely. Or you can actually poke it through JMX via jconsole at MBeans/java.lang/Threading/Operations/dumpAllThreads.
Without knowing more about your app, it's hard to speculate about the cause. Presumably your threads are either a) blocked or b) exited. If they are blocked, they could be waiting for I/O on a database or other operation OR they could be waiting on a lock or monitor (deadlocked). If a deadlock exists, the thread dump will tell you which threads are deadlocked, which lock, and (in Java 6) annotate the stack with where locks have been taken. You can also search for deadlocks with the JMX method, available through jconsole at MBeans/java.lang/Threading/Operations/find[Monitor]DeadlockedThreads().
Or your threads may have received unhandled exceptions and exited. Check out Thread's uncaughtExceptionHandlers or (better) use Executors in java.util.concurrent.
And finally, the other classic source of pauses in Java is GC. Run with -verbose:gc and other GC flags to see if it's doing a full GC collection. You can also turn this on dynamically in jconsole by flipping the flag at MBeans/java.lang/Memory/Attributes/Verbose.
Agree with aix, but would like to add a couple of recommendataions.
1. check your system. Run top to see whether the system itself is healthy, CPU is not 100% and memory is available. If not, fix this.
2. application may freeze as a result of dead lock. Check this.
Ok here are some updates I wanted to share:
There is an incompatability between NTPL (Linux’s new thread library) and the Java 1.6+ JVM. A random bug causes the JVM to hang and eat up 100% CPU.
To work around it set LD_ASSUME_KERNEL=2.4.1 before running the JVM, export LD_ASSUME_KERMEL=2.4.1 . This disables NTPL: problem solved!
But for compatibility reasons, I'm still looking for a solution that uses NTPL.
Threads can be traced using jvisualvm and jconsole, and deadlocks can be avoided too. Note that there are several network services each with separate thread pools, and they all become unreachable.
Check the jvisualvm of the process right before the crash.
http://www.jadyounan.com/wp-content/uploads/2010/12/process.png
Could you elaborate more on what you are doing ? 2000 for memory is rather a lot.