Trying to make sense of my application hanging, it was all running fine then hung.
I then attached Yourkit Profiler which highlighted two threads that had not changed for some time and appeared to have hang.
The interesting thing is both threads were performing the same task but on different data, however the method is simply a cpu intensive task which isn't sharing data, performing I/O or connecting to anything else.
public int[][] computeAssignments(float[][] matrix)
So with this in mind I cannot understand what could be causing them to just stop
Both threads are run as part of an an ExecutorService
What approach should I take to resolve this.
Update
As Glen guessed the problem turned out to be completely unrelated to threads. It is just that they both happened to hit the same problem, the code they were calling could go into a recursive loop if a particular set of data was provided. A good hint another poster gave was to check in profiler whether the threads were in Runnable or Waiting state, only if they were in waiting state would they actually be deadlocked.
My guess would be that you have an infinite loop there.
Related
Lets say I have written a infinite write loop but didn't have statement inside it? Will it create any issue like memory will be full etc or JVM will stop responding after sometime?
Why would you do something like that?
To answer, it wouldn't consume endless memory but Cpu usage could be a pain with really no instruction at all.
At minimum, you should help CPU preemption allowing the Thread to yield:
Thread.yield();
You can read this in Java Api Javadoc:
A hint to the scheduler that the current thread is willing to yield its current use of a processor. The scheduler is free to ignore this hint.
Yield is a heuristic attempt to improve relative progression between threads that would otherwise over-utilise a CPU. Its use should be combined with detailed profiling and benchmarking to ensure that it actually has the desired effect.
It is rarely appropriate to use this method. It may be useful for debugging or testing purposes, where it may help to reproduce bugs due to race conditions. It may also be useful when designing concurrency control constructs such as the ones in the java.util.concurrent.locks package.
An infinite loop might and probably will result in 100% CPU core utilization. Depending what you mean by "write loop" a similar technique is called Busy Waiting or Spinning.
spinning as a time delay technique often produces unpredictable or even inconsistent results unless code is implemented to determine how quickly the processor can execute a "do nothing" loop, or the looping code explicitly checks a real-time clock
You'll certainly keep one hardware thread busy. It wont create any objects, so memory isn't a direct issue as such.
However, the context is important.
If it is a high priority thread, the system may become unresponsive. This is implementation specific. Twenty years ago I wrote an infinite loop that made a Windows NT system unresponsive. (I think this was a TCP proxy and only happened when an IBM 3090 running CICS sent an empty keep alive frame to a 3270 terminal. Good times.)
If the thread is holding any locks, that wont be released.
If the thread does something useful, that useful thing wont happen. For instance if you were to write the loop in a finaliser (and the system only has one finaliser thread), no other object will get finalised and therefore not garbage collected either. The application may behave peculiarly. It'salways fun to run random code on the finaliser thread.
I was wondering if there is some way in Java to stop thread immediately. I don't want to check its interrupted status, I need to stop it immediately. That's because in thread's run method there are many calculations and to achieve what I want using interrupted I would have to inject status check everywhere. So is there some way to interrupt thread immediately? Maybe stop() method? I know it's is said it shouldn't be used because of deadlocks but if it could solve my problem (even if it would cause another ;) ) I could use it. So? P.S. I know there were other, similar questions but everywhere people give similar questions to interrupted() which doesn't suit me.
The question/answer that #Alya'aGamal points to is the right one.
If your app's design assumes that forcibly stopping a thread, or a process, or a program (like using kill -9 or stopping it via the Task Manager on Windows) is an okay thing to do, then you really need to justify that, because it sounds like a bad design choice. If you used someone else's app and the only way to close it on demand was to forcibly stop it, wouldn't you think that was a rather major flaw?
If you have long-running loops or algorithms and it's important to be able to stop them at an arbitrary point then you MUST put some kind of regular status or signal check in place in order to do this properly.
Always design your apps in a way such that there is a nice, friendly, graceful way for them to exit from all situations other than things outside your control (e.g. another app starts saturating the CPU, a hard disk dies, a RAM chip gets fried, a meteor hits the Earth, etc.)
As others have said, it's not a good idea to just kill a thread, which is why the stop() method has been deprecated. It's just too easy to introduce deadlocks this way. There are other reasons why stopping a thread externally is bad, but I won't get into them here.
Status checks really are the only other way to go, but I can understand why you'd want to avoid them. Checks add overhead and make the code cumbersome if your run() method has many lines of code... but there's simply no other thread-safe way to stop a thread.
That said - there are four components of deadlock: mutual exclusion, hold and wait, non-pre-emption, and circular wait. If you can guarantee that any one of these conditions will never be met inside your run() method, then you will never encounter a deadlock by calling stop().
To start with it is a huge application and the problem concerns many lines so I can't really attach any code.
After a change that consists mainly of clearing and re-adding elements to some collection, swing GUI of the application freezes. That freeze doesn't happen while that added code is executed but some time afterwards. What makes it strange is that no thread is suspended.
My question is whether the infinite loop is the only explanation of this problem. It feels unlikely to me that this is the case because the added code finishes without problems.
There may be some unsynchronized collection access issue, but I don't see it leading to that situation. It doesn't look also like we are dealing with a deadlock coming from synchronization problem since no thread is suspended.
In the end it is a deadlock.
My team leader told me that threads waiting on monitor (on "synchronized") aren't shown in eclipse as suspended. He found some two threads and asked to pause them. Then I saw they are waiting for each other to release the occupied monitors.
The deadlock isn't fault of the code I entered. It's just that the change I made revealed wrong synchronization in some other place.
Thank you all for trying to help me, I really appreciate it. It is my first question on stackoverflow and I was surprised how fast had you reacted.
Huge application + manipulating collections -> garbage collector kicks in?
Some related reading here at SO:
side effect for increasing maxpermsize and max heap size
And Oracle article about GC tuning:
http://www.oracle.com/technetwork/java/gc-tuning-5-138395.html
It may be because of waiting (due to some heavy processing) in Event dispatcher thread in swing where events are executed. Ideally you should execute any resource intensive task in a separate thread so that UI doesn't freeze
You might already know this, but just for the sake of it i would say that if you have not used a Swing worker in your long running processes within your app, this would be a ideal situation to use it.
In an open source application I'm participating, we've got a bug, where the application doesn't always close properly. That's what I'd like to solve.
Experience has shown that this happens most of the time when threads and processes are being started, but not managed correctly (e.g. a thread is waiting on a socket connection, the application is being shut down and the thread keeps on waiting).
With this in mind I've searched for '.start()' in the entire source and found 53 occurrences (which scared me a bit).
As a first step, I wanted to create a helper class (ThreadExecutor) where the current code 'thread.start()' would be replaced by 'ThreadExecutor.Execute(thread)' to have a) only a few changes in the existing source and b) a single class where I can easily check which threads don't end as they should. To do this I wanted to
add the thread to be executed to a list called activeThreads when calling the Execute method
start the thread
remove it from the activeThreads list when it ends.
This way I'd have an up to date list of all executing threads and when the app hangs on shutdown I could see in there which thread(s) is(are) causing it.
Questions:
What do you think about the concept? I'm usually coding c# and know how I'd do it using .NET with workers, but am not too sure what's best in Java (I'd like to modify as few lines of code as possible in the existing source).
If the concept seems ok, how can I get notified of a thread terminating. I'd like to avoid having an additional thread checking every once in a while what the state of all threads contained in activeThreads is, to remove them if they terminated.
Just to clarify: Before figuring out how to terminate the application properly, what I'm asking here is what's the best/easiest way to find which threads are at cause for certain test cases which are pretty hard to reproduce.
I would attempt to analyze your application's behavior before changing any code. Code changes can introduce new problems - not what you want to do if you're trying to solve problems.
The easiest way to introspect the state of your application with regard to which threads are currently running is to obtain a thread dump. You said that your problem is that the application hangs on shutdown. This is the perfect scenario to apply a thread dump. You'll be able to see which threads are blocked.
You can read more about thread dumps here.
Try to make all threads daemon(when all remaining threads are daemon the JVM terminates). Use thread.setDaemon(true) before starting each thread.
You could try to look into your application using jvisualvm (which is shipped with the jdk, find it in the bin folder of your jdk). JVisualVM can connect to your application and display a lot of interesting information, including which processes are still running. I'd give that a shot before starting down the road you describe.
Here is some documentation on JVisualVM should you need it.
The best way in java is to use Thread pools instead of Threads directly (although using threads directly is accepted). Thread pools accept Runnable objects, which you can see as Tasks. The idea is that most threads would do a small task and then end, because making a Thread is expensive and harder to manager you can use the threadpool, which allows things like 'ThreadPoolExecutor.awaitTermination()`. If you have more tasks than Threads in the pool, remaining tasks will just be queued.
Changing a Thread into a Runnable is easy, and you can even execute a runnable on a Thread you make yourself.
Note that this might not work for threads that run a long time, but your question seems to suggest that they will eventually finish.
As for your second question, the best way to find out which threads are running at a certain point is to run the application in a debugger (such as Eclipse) and pause all threads on a breakpoint in the close function.
I would try the trial edition of jprofiler or something similar, which gives you a lot of insight into what your application and its threads actually do.
Don't change the code yet, but try to reproduce and understand when this happens.
Create yourself a static thread pool.
static ExecutorService threads = Executors.newCachedThreadPool();
For every start of thread change:
new Thread(new AThread()).start();
to
threads.submit(new AThread ());
When your code exits, list all running threads with:
List<Runnable> runningThreads = threads.shutdownNow();
for ( Runnable t : runningThreads ) {
System.out.println("Thread running at shutdown: "+t.toString());
}
This will not only shut down all running threads, it will list them out for you to see what their issue is.
EDIT: Added
If you want to keep track of all running threads use:
Future f = threads.submit(new AThread ());
and store it in a list somewhere. You can then find out about its state with calls like:
f.isDone();
... etc.
I'm using a java.util.concurrent.ExecutorService that I obtained by calling Executors.newSingleThreadExecutor(). This ExecutorService can sometimes stop processing tasks, even though it has not been shutdown and continues to accept new tasks without throwing exceptions. Eventually, it builds up enough of a queue that my app shuts down with OutOfMemoryError exceptions.
The documentation seem to indicate that this single thread executor should survive task processing errors by firing up a new worker thread if necessary to replace one that has died. Am I missing something?
It sounds like you have two different issues:
1) You're over-feeding the work queue. You can't just keep stuffing new tasks into the queue, with no regard for the consumption rate of the task executors. You need to figure out some logic for knowing when you to block new additions to the work queue.
2) Any uncaught exception in a task's thread can completely kill the thread. When that happens, the ExecutorService spins up a new thread to replace it. But that doesn't mean you can ignore whatever problem is causing the thread to die in the first place! Find those uncaught exceptions and catch them!
This is just a hunch (cuz there's not enough info in your post to know otherwise), but I don't think your problem is that the task executor stops processing tasks. My guess is that it just doesn't process tasks as fast as you're creating them. (And the fact that your tasks sometimes die prematurely is probably orthogonal to the problem.)
At least, that's been my experience working with thread pools and task executors.
Okay, here's another possibility that sounds feasible based on your comment (that everything will run smoothly for hours until suddenly coming to a crashing halt)...
You might have a rare deadlock between your task threads. Most of the time, you get lucky, and the deadlock doesn't manifest itself. But occasionally, two or more of your task threads get into a state where they're waiting for the release of a lock held by the other thread. At that point, no more task processing can take place, and your work queue will pile up until you get the OutOfMemoryError.
Here's how I'd diagnose that problem:
Eliminate ALL shared state between your task threads. At first, this might require each task thread making a defensive copy of all shared data structures it requires. Once you've done that, it should be completely impossible to experience a deadlock.
At this point, gradually reintroduced the shared data structures, one at a time (with appropriate synchronization). Re-run your application after each tiny modification to test for the deadlock. When you get that crashing situation again, take a close look at the access patterns for the shared resource and determine whether you really need to share it.
As for me, whenever I write code that processes parallel tasks with thread pools and executors, I always try to eliminate ALL shared state between those tasks. As far as the application is concerned, they may as well be completely autonomous applications. Hunting down deadlocks is a drag, and in my experience, the best way to eliminate deadlocks is for each thread to have its own local state rather than sharing any state with other task threads.
Good luck!
My guess would be that your tasks are blocking indefinitely, rather than dying. Do you have evidence, such as a log statement at the end of your task, suggest that your tasks are successfully completing?
This could be a deadlock, or an interaction with some external process that is blocking.
Although you don't leave enough detail to be sure, the first thing I'd try is to have your tasks catch "Exception" at the top level and log the message.
I know it doesn't seem right, but occasionally (depending on a lot of variables) I've worked on code where stuff happening in a thread throws an exception and it is never logged, or it just doesn't show up on the console--yet the "executing" code exits out of it's top level loop or whatever code is causing your task to run.
I guess I'm just saying, make sure your tasks are not throwing an exception out.