I'm trying to debug an issue. We have few Threads that work on data from a BoundedLinkedQueue. After processing one record, the current thread executes Thread.currentThread().yield().
Now, once a while it is observed that one of the thread just vanishes ! I have traced the logs to find that such a "vanishing" thread works till the yield statement. After that no traces of that thread are found - nor any errors or exceptions are thrown near the last log seen for the thread.
Can anyone give any pointers for debugging directions ? Is the usage of the yield correct ? Is the yield a reliable statement ? This is because I found out this article suggesting to avoid the yield statement ? Has anyone seen such a condition before ?
Edit: On some research, it seems that try/catch may miss some exceptions and those would be just put into System.err which may not be noticeable in a multi-threaded environment. Thanks to #JVerstry for the pointer, I have set uncaughtexceptionhandler for the Thread. The build and run process takes long. Will update more once I have something concrete.
Here are few links that talk about UncaughtExceptionHandler:
Thread.UncaughtExceptionHandler
Catching Uncaught Exceptions in JDK 1.5
Java theory and practice: Hey, where'd my thread go ?
Know the JVM Series 1 - The Uncaught Exception Handler
As pointed out in the article you linked, yield doesn't define whether or not the current quantum is interrupted. If you yield right before thread exit, the scheduler just might complete the quantum for the thread causing the thread to exit immediately.
Yield does not make threads vanish. It is possible that your thread throws an exception and that is not caught. Did you implement an uncaught exception handler? If not, then I recommend you do so. It would explain your problem (unless the thread ends up naturally and your code does not do what you think it should do).
What occurs after the yield? Will the thread exit or will it attempt to process another piece of data from the queue?
You should verify that the what is being called after the yield is actually being called with logging.
How do you know the thread has exited? Have you verified by using by looking at a stack trace (Use Jstack)?
Lastly why are you using yield at all? I assume your BoundedLinkedQueue allows threads to retrieve data in a thread-safe manner, or blocks if the queue is empty. Why not just let the JVM manange thread scheduling?
We were able to get a thread dump when this re-occurred and seems that the thread was just blocking on a JDBC call forever - a bug in the jdbc jar. We just replaced the jar with the latest version and seems to have solved it. Thanks all for the valuable inputs - made me learn a lot of new things. Also, now put a query time out to prevent blocking forever.
Related
I'm wondering how to stop an unresponsive thread in Java, such that it's really dead.
First of all, I'm well aware of Thread.stop() being deprecated and why it should not be used; there are already many excellent answers on this topic, cf. [1][2]. So, the question more precisely is, whether it's actually technically possibly to kill a thread which code is not controlled by us but possibly hostile and not responding to interrupts.
In the simplest case, a hostile thread would be running while(true);, but it could as well be using up memory or other system resources to do more damage. Calling interrupt() on that thread is obviously ineffective. What about calling stop() instead?
I have run this in a debugger and, in fact, the thread really disappears. But is this approach reliable? The hostile thread could be prepared for this case; think of try{run();}catch(ThreadDeath t){run();} where it catches the ThreadDeath that is produced when we call stop() and recursively calls itself again.
As an outside observer, we cannot see what is going on; Thread.stop() always runs silently. Worst of all, the usual diagnostics won't work anymore (tried this while debugging on Corretto 1.8.0_275 Windows x64): Thread.getState() always returns RUNNABLE regardless of success in killing the thread, same goes for Thread.isAlive() (always true).
It may not be possible, at least not reliably in every scenario.
IF I understand the mechanism correctly (and there is some uncertainty there), if the code executes in such a way that there are no safepoints during the execution (for example in counted loops), it is not possible for the JVM to signal to the thread that it should stop (the thread never polls for an interrupt).
In such a scenario, you need to kill the JVM process, rather than the thread.
Some extra reading:
How to get Java stacks when JVM can't reach a safepoint
Counted loops
In a nutshell, there's no 100% reliable way to stop a Thread the way you'd like it.
Why?
This is an explanation for others who don't know why, anyone who knows the issue can skip this.
The way how threads are intended to be terminated forcefully is with the interruption state of the Thread. A Thread should be terminated with its interrupt() method is called which sets a boolean flag to true.
When the interruption flag is set to true, the Thread should terminate itself with virtually no delay.
Anyway the Thread can choose to simply ignore this and keep on running.
This is when the stop() method can be called that forces the Thread to terminate. The problem is that this method messes up concurrency, can damage objects and the program can be corrupted without a warning for the user. See Why the stop() method is deprecated?
At the end I could think of two possible ways, one is basically your way, the other one is safer but more complicated.
As an example, a hostile third party .jar which contains a Thread that refuses to terminate can cause these problems.
Quick & Dirty
This solution isn't completely safe but based on the usage this may be acceptable unless you really like security.
Try to first to call the interrupt() method on the Thread and give it a bit time to terminate.
If the Thread doesn't respond, you can either:
terminate the program and warn the user to not run that Thread again.
stop() the thread and hope for the best.
Complicated & Safe
The safest solution I can think of is creating a whole new process to run the Thread in. If the Thread doesn't want to terminate after interrupt(), you can just end the process with System.exit(-1) and let the OS handle it.
You need Inter Process Communication to communicate with the other process and that makes it a lot more complicated but also safer.
Related
How do you kill a Thread in Java?
What is an InterruptedException in Java? (Disclaimer: I've answered it)
What does java.lang.Thread.interrupt() do?
For me isAlive returns false if the process finishes due to Thread.stop.
I've made the following example, and it successfully kills the errant thread.
import java.util.Arrays;
public class BrokenThreads{
static boolean[] v = { true };
public static void call(){
try{
while(true){
Thread.sleep(200);
}
} catch ( Throwable td){
System.out.println("restarting");
call();
}
}
public static void main(String[] args) throws Exception{
Thread a = new Thread( BrokenThreads::call);
a.start();
Thread.sleep(500);
System.out.println( Arrays.toString( a.getStackTrace() ) );
while(v[0]){
a.stop();
System.out.println(a.getStackTrace().length);
v[0] = a.isAlive();
}
System.out.println("finished normally");
System.out.println( Arrays.toString( a.getStackTrace() ) );
}
}
Note that "getStackTrace" takes time, and you can see the stacktrace accumulate as recursive calls are made, until two stops happen quick enough to end the thread.
This uses two techniques to see if the thread has stopped. isAlive and the depth of the stack trace.
I think the question describes a scenario that is the reason why Thread.stop() is deprecated since ages now, but was not yet removed … just to have a 'last resort option', to be used only when being really desperate and being aware of all the negative impact.
But that call to Thread.stop() must be build into the code somehow, same as any alternative one may think about – so why not just fix the code for the thread? Or, if that is not possible because that code comes with a third party library without source code, why not replacing that library instead?
Ok, during testing, your own code may go wild, and you need an emergency break – for that, Thread.stop() is still good enough if you do not want to kill the whole JVM (what would be the better option in most of the cases). But again, you have to build this into the code before you start the test …
But in production, there should never be a thread that does not stop when receiving an interrupt. So there should be no need for a replacement of Thread.stop().
This can potentially open a can of worms like memory access violations which will kill the JVM itelf.
What you could do is isolate the thread, running .finalize() on it, then forcing the JVM to run GC operations such as Runtime.gc(), System.runFinalization() while forcing interruptions on that particular thread in order to bypass it's resurrection behavior.
I think .finalize() is effectively deprecated since java11 or maybe sooner, so it probably won't help you much.
If you really want to secure your runtime during it's operational cycles, your best bet would be to find a way to essentially map out your configuration before you start it, and have monitoring tools set up which cross-check against that map and monitor the integrity of your runtime while looking for injected classes and/or threads. ... this is assuming of course, you're attempting to guard against "virus-like" attacks in your jvm ... which is not exactly unheard of but still pretty rare.
If you're running some code off the internet, you could simply solve the issue with a call hierarchy inspection and figure out what spawns that problematic thread.
Note: If that invasive thread is calling native dll code which is looping back into it's caller, then your JVM will crash if you mark sections of it's address space as garbage collected.
I have multi threads that want to put a value in a TreeSet<Long>, in a part of code. The values are almost unique because they are System.nanoTime(). I clean the TreeSets periodically. Problem is that sometimes my threads got blocked in TreeSet.add() function. I used jconsole to watch thread states, threads are in RUNNABLE state and the stack trace shows this:
java.util.TreeMap.put(TeeMap.java:567)
java.util.TreeSet.add(TreeSet.java:255)
... //my program stack trace
I'm using jdk 1.7.0_60 for running program. Also I should mention in this situation cpu usage become 100%. My question is why the threads got blocked and how can I fix the situation? I looked at TreeMap code, but I didn't figure out problem, but I think problem relates to while loop in TreeMap.put().
As it was mentioned in comments, the problem is that TreeSet is not thread safe and if we want to modify it (add or remove data) in multi threads, it must be synchronized externally.
In my JavaFX 8 app, I have a thread that runs a loop and occasionally updates the UI. Somewhere in this thread, there's an exception causing me grief. Finding it wouldn't be a problem, except exceptions on worker threads don't seem to be logged to System output the same as those on the main program thread.
What I mean is, when I run the program, if something breaks in main(), IntelliJ prints the exception message and a stack trace to the Run view (standard out, AFAIC). On the other hand, if something breaks in my thread, I don't get the same verbose output. In fact, most of the time, I don't get anything at all - just a mysteriously non-functioning program.
Is there another step I need to take to capture errors in worker threads? Is my environment misconfigured? This makes debugging next to impossible - any help appreciated!
Between Samuel's comment and some searching, I've got my answer; Runnables tend to consume their errors, so they're never thrown up to the main thread. This means that with no error handling, they simply disappear and the thread is broken.
The solution in my case is to add try/catch blocks in my task. This is dumping errors the way I expect now.
(I guess your answer/comment means I should turn my comment into an answer.)
Depending on your threading model exceptions are not propagated to the main thread, they just kill the thread they are thrown from. You might benefit from adding a try/catch to your "run" method when starting the thread.
I have a program where I compile java code a user types into a text field, and then run it. A run the code in a seperate thread, so that the GUI they use to input the source code doesn't get locked up.
The GUI has an abort button that should stop the thread. My issue is that I need to stop the compiling thread no matter what is going on inside of it, which means I must account for a case where the thread is caught in an infinite loop (due to user error), and it cannot properly end itself using a safe flag. I've read up on many solutions that involve using a flag of some kind, but they aren't available to me because of this looping issue. I need to have the thread stop and the memory it's using freed (I can't just let it sit in the background forever, unless that is the only solution left). Any advice or alternative solutions? Hopefully some fresh perspectives could help squash this issue.
Edit:
Here's a sample bit of user submitted code:
public class RunMe extends SomethingThatRuns {
public void run() {
int i = 0;
while (i = 0) {
//Prepare to get stuck!
}
}
}
I'll compile this class, and then run it. This is where it will get stuck, and the run() method can never finish, or even loop to check a flag.
You can run it in a new JVM so you can kill it when you want.
Thinking about security this may be a good thing to do too.
Call stop() on the thread.
Yes, this is a deprecated method. However, it really shouldn't be "deprecated", it should be "dangerous." In some circumstances, however, there's really no choice but to use it, and the invocation of an "agent" provided by a user is one of those cases.
Make sure that your program doesn't use any data that are manipulated by this user thread; or, if you do, devise some transactional mechanism to exchange data safely between the threads.
Even this method isn't guaranteed to terminate the thread. For example, the user can catch the resulting Throwable and ignore it. Or, the thread implementation might not respond to stop() calls if the thread is in some native code. But it's your best chance.
The core issue here is the fact that the code even allows an infinite loop to be entered as part of user error. Fix that, and everything else will become easier to deal with.
Properly-behaving threads should usually terminate themselves gracefully when there's no work to do (or return quietly to a thread pool to ask for more work, if that's your application's design). If you feel like you need to have one thread forcefully kill another then you've likely got a fundamental design issue. It's fine to have one thread tell another, "Hey, you should terminate now so that I can join with you..." because that allows your threads to clean things up as they finish. Forcefully destroying threads just isn't the right way to manage these situations.
You can use them to insert a interrputed check in every loop and maybe in other places too.
I can see two options:
As you compile the user code you can edit it before. You may use
ANTLR to parse and modify the code.
There are bytecode manipulation frameworks like ASM that allow you to manipulate code that is already
compiled.
I don't think it is easy but it might be a way.
interupt(); the Thread in the gui
and in the code that the thread runs regularly check for Thread.interrupted() and throw an exception when you do especially inside loops
At a high level, you are asking how one thread might go about stopping another thread. To that end, see this SO question Stopping a Thread in Java?.
I am using a thread pool for my task. After completion of each task I am destroying the thread using Thread.stop() and Thread.destroy(). But after running my application (in Eclipse) for around 30 min. I am getting a Memory out of bound error.
Please suggest me how to kill the thread.
If you're using a thread pool, you shouldn't be terminating the thread to start with - the whole point of a thread pool is to reuse threads.
If you don't want to reuse the thread, then just start a new thread instead of using a thread pool - and just let the thread die, instead of calling stop or destroy. These methods are deprecated for good reason - they basically shouldn't be called.
It's not really clear how this would cause an out of memory exception though - is there any reason why you're focusing on threading as the probable cause?
To reinforce what #Jon Skeet said, it is a REALLY BAD IDEA to call the deprecated Thread.stop() or Thread.destroy() methods.
According to the javadoc, Thread.destroy() was fundamentally dangerous and was never implemented. The original idea was simply to kill the thread and break all of its monitor locks. If it happened to be in the middle of updating a shared data structure, the data structure would be left in an indeterminate state. Other threads waiting for the killed thread to notify some object would wait for ever.
Thread.stop() causes a ThreadDeath exception to be raised at an unexpected (to the code that was hit) place. It is a little bit more orderly than killing a thread, but unless all of the stopped thread (including anything that it calls) is carefully written with finally blocks to notify waiters, restore data structures, etc, you have the same problem.
Refer to Java Thread Primitive Deprecation for the whole story.
When the task is complete, the thread run should return. Do nothing more. That will take care of things.
In debug mode the threads are not cleared by the garbage collector.
Try to run the app instead of run in debug mode and everything should be fine.