I have J2SE application running in 1.5 java VM in RHEL OS. One of the task of the application is to create 3 infinitely running user threads, during startup. The purpose is to check for a request of a particular type in backend DB table and do corresponding operations.
As we observed, the long running threads suddenly stops running, but still the application is alive and JVM process can be seen, in ps -ef|grep java
Can someone throw light on why threads which are created to run in infinite loop, stops suddenly? Any ideas on how to detect this issue and possible resolution will be of great help
With Regards,
Krishna
I would suggest sending a Ctrl+Break to your app, dumping the threads and analysing the output. Perhaps your threads are waiting for some input (IO). Perhaps they're deadlocked. Perhaps they've exited with an uncaught exception. The thread dump will tell you what's going on (and it helps if you name your threads in advance so you can identify them in the dump).
Perhaps you are having unhandled exceptions.
First of all, you should log all your thread activity (you can use log4j to achieve this).
You can also override the uncaughtException method of the ThreadGroup class and create alerts for when a thread dies due to an exception that has not been caught.
Related
If a Java process hangs (due to bug in JNI (faces deadlock), Can it result in blocking of entire JVM? i.e. all processes and threads getting blocked?
due to bug in JNI Yes. If you call into native code a bug can easily bring down the entire JVM (or block everything).
No. The thread or threads in deadlock will remain blocked, but other threads can run independently in other programs or even in the same program. Obviously deadlocks should be avoided whenever possible, but the affected threads will only be the threads in deadlock and any and all threads waiting on the completion of those threads.
Normally every application has it's own JVM instance running. So you could not crash other applications by trying to crash your current JVM . However some application share one JVM like a Web server for instance.
Another scenario would be if it crashes anything on an operating system level. Well then everything related to that will shut down.
I have code (runtime between 1 and 2 seconds) which I want to run everytime the computer shuts itself down (logging off a user out of a another system on another server).
I have already tried to add it to the ShutdownHook but the execution of the overgiven Thread is to slow - the thread gets killed before all commands could be executed.
Also tried to add it to a finally statement, that didn't help anything too.
What is IYO common practise for that case?
THX a lot.
You will not get this to work robustly. ShutdownHook will only be called if the JVM is closed down gracefully. Ungraceful closes include the OS killing the process, or even a power failure.
An alternative in this instance is to maintain a "heartbeat" signal between your computer and the server which, if severed, logs the user off the server.
I m using windows 7 OS. I have around 6 threads in my application. For the purpose of testing the alerts to check the health of the threads, i need to kill the threads manually and check if the alerts are working properly. Can we kill a thread like how we kill a process with its pid?
Dan Woods documented how to kill a thread in this blog entry...
https://web.archive.org/web/20160302023213/http://www.rhcedan.com/2010/06/22/killing-a-java-thread
The steps he performed involved using a debugger (JDB) and injecting an exception in the thread's execution. Specifically...
Ensure that your java program is started with the following parameters:
-Dcom.sun.management.jmxremote.port=50199
-Dcom.sun.management.jmxremote.authenticate=false
-Dcom.sun.management.jmxremote.ssl=false
-Xrunjdwp:transport=dt_socket,address=50100,server=y,suspend=n
This will allow us to attach the java debugger to the running process, after
we identify which Thread is causing the problem. Also, make sure that
you have your iptables setup appropriately so as to only allow
connections on 50100 and 50199 from the hosts/workstations that you manage.
Identify the offending thread:
Kill the thread. In this example, the ThreadName is “btpool0-0?. Fire up the java debugger (also shipped with the JDK distribution), and attach to the running JVM…
[root#host ~]# jdb -attach 50100
Get a list of the running threads — this will also give us the thread id as the JVM sees it:
> threads
--snip--
(org.mortbay.thread.BoundedThreadPool$PoolThread)0x25cb
btpool0-0 running
--snip--
The thread id that we’re going to kill is “0x25cb”. The first step of killing the thread is to jump into it, and suspend it…
thread 0x25cb
btpool0-0[1] suspend 0x25cb
btpool0-0[1] step
Step completed: <... snip ...>
btpool0-0[1] kill 0x25cb new java.lang.Exception()
killing thread: btpool0-0
btpool0-0[1] instance of
com.site.package.name(name='btpool0-0', id=9675) killed btpool0-0[1]
Exit the java debugger, and you’re done!
There is no safe way to "kill" a thread without killing the process it is in. It not something you would do deliberately. For testing purposes I would add code to your application to support this.
It's not true. You can always attach to the JVM process with GDB and do a call pthread_kill if you know the thread id. You only need to translate from the java thread dump (do a kill -3) which gives you a hex id, (native id), then look into the list of threads in GDB (info threads) and locate the real thread id.
This is proven to work.
As Peter says, you can't do this safely. Indeed on some platforms Thread.kill is not even implemented. However:
If this is just for testing, a unit test that called Thread.kill would be reasonable ... assuming it worked on the test platforms where it needed to work. (A "loud" comment in the source code would be in order to help people porting the unit test ...)
Another alternative is to add some code to the thread runnable that allows your unit tests to tell it to die. If the thread code needs to be (almost) production code for this to work, you could create a subclass that overrides something so that it "breaks" in a way that suits your purposes ... for testing. In fact, this approach allows you to cause the threads "break" in controlled ways, potentially allowing you to test different aspects of your alerting code.
You can't do it from outside (OS or debugger), you'll have to write your own Thread watchdog that can interact with the user and kill the thread you want.
Try to look here for how to handle signals with java
In java you can not kill the like unix . Either you can interrupt the tread in java or you can kill the process in unix .
Wait for some time in the thread and kill the thread in the code - simple way.
As mentioned in a previous post by buzz3791, it works by using jdb. However the change that I noticed is, you can't kill the thread, but you can interrupt or suspend the thread.
#jdb -attach 50100
threads --This will show all threads running on the jvm under Groups and Reference Handler section.
Groups
Reference Handler
:(com.orientechnologies.orient.server.network.protocol.binary.ONetworkProtocolBinary)0x36c1:
OrientDB (/xxx.xxx.xxx.xxx:123) <- BinaryClient (/xxx.xxx.xxx.xxx:678)
thread 0x36c1-- This will be the thread id that can be picked from one of the threads in the thread that you wish to kill/interrupt and run this interrupt 0x36c1
OrientDB (/xxx.xxx.xxx.xxx:123) <- BinaryClient (/xxx.xxx.xxx.xxx:678)[1] interrupt 0x36c1
you can try multiple times the same interrupt command and if it is already interrupted, it will show that the thread id is invalid. Thus you know that the thread is killed, this can be verified by looking at the stack trace and confirmed.
Tested this on the OrientDB database server with jdk 8 and it works.
It is possible to kill a thread that is in state RUNNING in a non programatically way?
I know that top command in *nix can show threads. Can I kill the thread in the OS?
I'd like to know if there is a way to link a thread to a process so I can kill only that specific thread and not the application.
We had a bug in our code that kept a thread in state RUNNING in a synchronized method. The thread kept the lock on the object "hanging" the application.
The bug is fixed. But I wonder if is possible.
The short answer is "maybe, but you should not and most of the time it won't work either".
The long answer is:
"Maybe..."
Some JVM implementation map java threads to OS threads and some do not. If the JVM does a mapping to a native OS thread, you might be able to kill that thread with some process tool that the OS provides (like kill on *nix). If the JVM does green threads, meaning it doesn't map a Java thread to an OS level thread, then you are basically out of luck using OS level tools. Luckily only very few JVM implementations do this. An approach that can be used regardless in which way the JVM organizes it's threads, is using the java debugger. This article describes the procedure of doing it: http://www.rhcedan.com/2010/06/22/killing-a-java-thread/.
"but you should not do it"
Killing a thread on the OS level will almost certainly leave the JVM in an undefined state (read "jvm might crash or delete all files on your disk or do whatever it fricking pleases to do"). Even when going the debugger way, only a very small amount of java applications (read "no application made on this planet") will properly handle the event that an outside application is killing one of it's threads. As a result these applications will be put in an undefined state (read "application might crash or delete all files on your disk or do whatever it fricking pleases to do").
"and most of the time it won't work either"
If the thread is really stuck with some blocked IO etc, then killing the thread won't work, it will just not respond. If a program is stuck it's probably better to kill the whole program, find the issue with the program and fix it instead of killing a single thread.
For all your doubts on killing a thread, refer this:
http://download.oracle.com/javase/1.4.2/docs/guide/misc/threadPrimitiveDeprecation.html
On linux, there is a tkill(int tid, int sig) command, similar to kill.
On windows, ProcessExplorer can do it from gui, don't know if there is anything with cli.
We have a Java App that connects via RMI to another Java app.
There are multiple instances of this app running at the same time, and after a few days an instance just stops processing... the CPU is in 0 and I have an extra thread listening to an specific port that helps to shutdown the App.
I can connect to the specific port but the app doesn't do anything.
We're using Log4j to keep a log and nothing is written, so there aren't any exceptions thrown.
We also use c3p0 for the DB connections.
Anyone have ideas?
Thanks,
I would suggest starting with a thread dump of the affected application.
You need to see what is going on on a thread by thread basis. It could be that you have a long running thread, or other process which is blocking other work from being done.
Since you are running linux, you can get your thread dump with the following command
kill -3 <pid>
If you need help reading the output, please post it in your original question.
If nothing is shown from the thread dump, other alternatives can be looked at.
Hum... I would suggest using JMetter to stress the Application and take note of anything weird that might be happening (such as Memory Leaks, Deadlocks and such). Also review the code for any Exceptions that might interrupt the program (or System.exit() calls). Finally, if other people have access to the computer, makes sense to check if the process wasn't killed manually somehow.