A bug in a third party library is causing an infinite loop in a worker thread on a JBoss instance of mine. Do you know of a way to kill this "stuck" thread without restarting the server? We'd like to be able to recover from this until a fix is deployed, preferably without having to restart.
I've seen a few people mention using Thread.interrupt() - if I were to code my own MBean, how would I get a handle to the thread in question in order to interrupt it?
Update: Wasn't able to solve using any of these methods. I did come across another thread about the same issue that had a link to why Thread.stop() is deprecated. Someone else has asked a similar question with similar results. It seems like more sophisticated containers should provide this kind of health mechanism, but I guess their hands are tied w/r/t the JVM.
I had a similar bug (infinite loop) in a 3rd party lib. I ended up applying the fix myself (while waiting for the people from the 3rd party lib to fix their mess) and then I placed the modified .class in my .war, making sure it is loaded before the bogus .class (the bogus one being inside the bogus 3rd party .jar).
It is not nice but it works, see my question here:
Order of class loading from a .war file
What I mean is this: if you have to wait for the people responsible for the 3rd party bugged lib to fix their stuff, you can potentially be waiting a very long time. We couldn't afford that. We needed a fix ASAP. So we ended up applying a patch/hack to their code.
You could for example add a boolean check inside the infinite loop and then forcing the loop to exit when you want the bogus thread to "die".
Note that I haven't used the deprecated Thread stop() since ten years and I really didn't want to use it in the above case.
I suppose the most difficult part is to identify the hanging thread. You provide no info about it, but perhaps you can build some rules around the thread's name or its current stack trace.
If you can identify the thread by its name, I would get all threads in the VM by getting my own thread group with Thread.currentThread().getThreadGroup(), then walk up the thread group hierarchy by calling getParent() on the thread group until it returns null. You now have the top level thread group. You can now fill a preallocated array with all threads using the enumerate(Thread[] list) method on the top level thread group.
If you need the stack traces anyway to identify the thread, you can also use the static utility method Map<Thread,StackTraceElement[]> Thread.getAllStackTraces() to get all threads. Computing the stack traces is however quite expensive, so this might not be the best solution if you don't actually need them.
After identifying the thread you must call the stop() method on it. Interrupting it won't help, unless the implementation of the running code actually evaluates the thread's interrupted flag and behaves as you expect it to. Not that the stop() method is deprecated and that using it may have many funny side effects. You can find more details in the API documentation.
You could use the discouraged myThread.stop() method. But then it is very likely the Thread is still referenced there, so you should use some reflection magic to remove all references to this thread from the components holding it.
How to find the Thread? Use Thread.getThreadGroup() and ThreadGroup.getThreadGroup() to go up to the root ThreadGroup(), and then use the iterate() functions to go through all threads.
Try my jkillthread which tries to do something like this.
Related
I know this subject has been discussed here before, and we have utilized past conversations to attempt to resolve the DbMaxReadersExceededException that we are still experiencing. We are using version 2.5.1 of ObjectBox. We are also, heavily, using RxJava threads while manipulating our BoxStore DB. At any moment in time, potentially a handful of RxJava threads are running, accessing the DB. Threads are constantly spawning, executing and terminating.
This is a very "non-standard" use of Android. Our App is running on a non-cell phone device, that sits on a wall and is expected to be available 24x7. 95% of the RxJava threads that access the BoxStore DB are short lived, get in / get out threads, that retrieve information and present to the device user. We do have a few longer lived background RxJava threads, that talk to an external DB over the internet to keep the local DB up to date. But these threads to spawn, execute and terminate. Theses threads run in the background at regular intervals. These background threads are not associated with a Fragment nor Activity; therefore the common way of cleaning up, using a CompositeDisposable, is not utilized.
We are seeing that readers are accumulating, despite many attempts to resolve the situation. We have also noticed that threads, that have run to termination, marked as isAlive and appear to be part of the RxJava thread pool, also accumulate. We have observed this using Thread.getAllStackTraces() and printing out this information regularly. Separate issue I am not trying to resolve with this post (I am concentrating on the DbMaxReadersExceededException issue, but they may be related).
The readers accumulate as the result of .find() calls on a Query that is build; based upon analysis of when a reader occurs. That is not surprising, but sometimes a .find() causes a new reader and sometimes it does not. I do not understand this behavior, and I am not sure if that is a telling sign or not. But it does result in the accumlation of active readers everytime the RxJava thread that accessed a given Box is invoked.
Any help / assistance offered will be greatly appreciated. Please ask any questions about anything that I may have accidental left out.
Things that we have tried, based upon other posts that I have read, include:
Collect Disposables from RxJava background threads and dispose
We have tried collecting the Disposable generated by the .subscribe() from these background threads, and added a timer to .dispose() of them sometime (5 seconds) after the thread that was using this object terminates (run to completion).
Utilized BoxStore.diagnose()
We have written code to utilize BoxStore.diagnose() to be able to periodically watch the reader accumulation.
Tried BoxStore.closeThreadResources()
We have added BoxStore.closeThreadResources() calls when an RxJava thread terminates to cleanup any BoxStore resources that may be active.
Tried Box.closeThreadResources()
We have tried adding Box.closeThreadResources() calls closer to when the Box is accessed in order to access and then clean up ASAP.
Tried breaking down .method() sequence and added .close() calls to itermediate objects
We have tried breaking down the .method() call sequence that terminates with the .find() call and then .close() or .closeThreadResources() the intermediate objects along the way.
Tried combinations of the above
We have tried a combination of all of the above.
Wrote method to be able to monitor RxJava threads using Thread.getAllStackTraces() - RxJava threads seem to accumulate
We have written a method that helps us monitor RxJava threads using Thread.getAllStackTraces().
We have tried to manually invoke the Garbage Collector
We added code, after the .dispose(), mentioned above, to cause a manual Garbage Collection (System.gc()).
As far as I know, we have tried every suggestion that I have seen posted on this and other forms, regarding this issue. We are at a loss as to what to do or try next. I did see something about a package called RxObjectBox, but I have not pursued this any further.
Should we:
Look at restructuring our RxJava thread access?
Do we need to look closer at RxObjectBox?
Is there a known problem with ObjectBox 2.5.1 that we should be using a later version?
What haven't we tried that we should?
This is my situation:
I am studying a large codebase, running on Java1.7, not very easy to move around, lots of interfaces, deep inheritance trees, lots of threads etc.
I put a breakpoint in some place, but this object is running in a Thread that was spawned somewhere. I need to find that place.
there are too many .run() and .start() hits to look for individually (and to narrow down by the class is difficult too as there are many classes/inheritance (and I don't know the codebase yet)).
So my questions is, is there a way, having a Thread stopped in a breakpoint (intelliJ, but I can use eclipse too) to find out where it was started??
thanks
Maybe you can put breakpoint into Thread.start().
To avoid mutltiple invocation of breakpoint, maybe it make sense to place breakpoint with conditional logic, for example checking global boolean flag. For example, you suspect, that your code invokes right before some event, when event happens, put global flag to true.
No.
I don't think there's a way out of this without some brute force effort.
I would trace back to the Runnable that was started (through the stack trace), then get that class' inheritance and interface hierarchy, then look for run() and start() methods on all those classes. Unless someone has just gone nuts with inheritance, it shouldn't take that long.
Breaking on the code in your object tells you which thread it is and its call stack can tell you which Runnable you should be looking for. I'm assuming you've already gotten this far and that it's not enough to find all the references to this Thread/Runnable. In that case you can write a wrapper class for java.lang.Thread that does an instanceof/type check in the run() and setting your breakpoint there
I think first you have to get the Runnable that is run. That's simple as it's always the first line of your stack trace. (Of course you need the concrete class and not the one that defines the run method.) Once you have the class it should be easy to find the instantiation. Then it should simple to follow to the point where the thread is started. Did I miss something?
By way of some library, I find myself calling this function twice concurrently on a single instance (using the implementation returned by Executors.newSingleThreadScheduledExecutor). The Runnable passed to the second call seems not to execute, neither immediately nor on the next scheduled slot, and no exception is raised. If I serialize the two calls (did this very crudely and unintentionally by putting a breakpoint before the second caller's scheduling call), then the second runnable is executed with no issue.
I'm new to this interface, but it doesn't seem like these scheduling functions are designed to be reentrant. But I can't find anything in the various documentation describing what should happen here.
Well, small test case doesn't reproduce the problem, so I have no reason to believe the function isn't reentrant. What actually fixed the problem was to remove all this from request time to server start. There were some other signs, like the breakpoint temporary fix I mentioned in the OP, that point to some awful timing issue, somewhere in my stack.
I'm working with threads but after a time, most of them stop doing their job. The first thing I thought was a deadlock, but all are with state RUNNING.
I suppose there is an error in my logic or a new characteristic that I not realized and I must handle (it's a webcrawler).
Is it possible to get the current executing method or operation? I want this to see where my threads are trapped.
EDIT: I think that is something I need to handle or there is error in my logic because this happens after a time executing, not imeddiatly after the start.
A debugger is the way to go. This is what they are designed for.
Java debuggers with threading support are built into both the Eclipse and Netbeans IDEs.
Make VM to dump the threads (Ctrl-Break). Find your threads in the list. Look at the topmost stacktrace method. Done.
You can get the current stack trace in Java. You will get an array of StackTraceElement elements.
The first item in the array is the currently executing method.
See the following question for how to get the stack trace:
Get current stack trace in Java
Code might look like:
StackTraceElement[] trace = Thread.currentThread().getStackTrace();
StackTraceElement yourMethod = trace[1];
System.out.println(yourMethod.getMethodName());
You have 2 options:
Use debug to get some understanding that was executed and what not.
Use a lot of logmessages (you can also produce stacktraces in that messages)
Thread dumps are the right solution for the problem. If you want to do it programmatically within the process (some kind of monitoring logic), then java.lang.management.ThreadMXBean provides access to all threads along with their current stacks at the time.
It is, throw an exception, catch it immediately and save the stack. This is about as performant as asking an elephant to fly overseas but it's possible since it sort of extracts the current call stack to something you can work with.
However, are you sure you haven't run into a livelock?
Do you suppose your web crawler program is in a loop processing the same urls. Add some high level logging so each thread writes what it's processing.
How can I find out who created a Thread in Java?
Imagine the following: You use ~30 third party JARs in a complex plugin environment. You start it up, run lots of code, do some calculations and finally call shutdown().
This life-cycle usually works fine, except that on every run some (non-daemonic) threads remain dangling. This would be no problem if every shutdown was the last shutdown, I could simply run System.exit() in that case. However, this cycle may run several times and it's producing more garbage every pass.
So, what should I do? I see the threads in Eclipse's Debug View. I see their stack traces, but they don't contain any hint about their origin. No creator's stack trace, no distinguishable class name, nothing.
Does anyone have an idea how to address this problem?
Okay, I was able to solve (sort of) the problem on my own: I put a breakpoint into
Thread.start()
and manually stepped through each invocation. This way I found out pretty quickly that Class.forName() initialized lot of static code which in return created these mysterious threads.
While I was able to solve my problem I still think the more general task still remains unaddressed.
I religiously name my threads (using Thread(Runnable, String), say), otherwise they end up with a generic and somewhat useless name. Dumping the threads will highlight what's running and (thus) what's created them. This doesn't solve 3rd party thread creation, I appreciate.
EDIT: The JavaSpecialist newsletter addressed this issue recently (Feb 2015) by using a security manager. See here for more details
MORE: A couple of details for using the JavaSpecialist technique: The SecurityManager API includes "checkAccess(newThreadBeingCreated)" that is called on the thread creator's thread. The new thread already has its "name" initialized. So in that method, you have access to both the thread creator's thread, and the new one, and can log / print etc. When I tried this the code being monitored started throwing access protection exceptions; I fixed that by calling it under a AccessController.doPriviledged(new PrivilegedAction() { ... } where the run() method called the code being monitored.
When debuging your Eclipse application, you can stop all thread by clicking org.eclipse.equinox.launcher.Main field in the debug view.
Then from there, for each thread you can see the stack trace and goes up to the thred run method.
Sometimes this can help and sometimes not.
As Brian said, it a good practice to name threads because it's the only way to easily identify "who created them"
Unfortunately it doesn't. Within Eclipse I see all the blocking threads, but their stack traces only reflect their internal state and (apparently) disclose no information about the location of their creation. Also from a look inside the object (using the Variables view) I was unable to elicit any further hints.
For local debugging purposes, one can attach a debugger to a Java application as early as possible.
Set a non-suspending breakpoint at the end of java.lang.Thread#init(java.lang.ThreadGroup, java.lang.Runnable, java.lang.String, long, java.security.AccessControlContext, boolean) that will Evaluate and log the following:
"**" + getName() + "**\n" + Arrays.toString(Thread.currentThread().getStackTrace())
This will out the thread name and how the thread is created (stacktrace) that one can just scan through.