I have a method in Java that call several other methods. This method is being called from several threads, in a fixed thread pool. The number of workers is the same as the number of available processors (cores).
public void query() {
method1();
method2();
}
When I profile the program execution using VisualVM the times of method1() and method2() times are very short, but query() self-time is very long. But the method has no other code apart from the two calls. There might be synchronization inside method1() and method2(), but nothing obvious in code that I have control of.
When I reduce the number of workers in the pool to 1, this self-time is almost gone. Both single-threaded and multi-threaded execution times of the whole program are almost the same. I think it means that my method query() is waiting for something.
There are no deadlocks, the execution finishes fine. The two methods method1() and method2() call a lot of other things including library classes in obfuscated jars, so it is not easy for me to debug it. However the query() method is called directly from the worker threads, using java.util.concurrent.ExecutorService.
Issue a kill command at level 3 against the running process. All threads will dump a stack trace to the standard out and the app will continue running.
kill -3 <pid>
Note, you wont see anything on the console where you issued the kill command. The Java app itself will have the output. You might need to check logs, depending on where the app is redirecting its output.
I have found the problem in a proxy class that was wrapping another class in a custom locking mechanism.
I went on creating a series of Thread Dumps. Since I was using JVisualVM for profiling, I created a handful of Thread Dumps during the process. Ctrl+Break worked too, same as kill -3 <pid> mentioned by Synesso in his answer.
I used the Thread Dump Analyzer mentioned in the comments to analyze them. I did not know what to look for first, but thanks to the linking of objects and monitors in the TDA, I found something like this:
"pool-9-thread-32" #304 prio=5 os_prio=0 tid=0x000000002a706800 nid=0x348c waiting for monitor entry [0x000000003f06e000]
java.lang.Thread.State: BLOCKED (on object monitor)
at com.example.MyClass.method1(MyClass.java:400)
- waiting to lock <0x0000000680837b90> (a com.example.DifferentClass)
at com.example.MyClass.query(MyClass.java:500)
... omitted ...
at java.util.concurrent.FutureTask.run(FutureTask.java:270)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:618)
at java.lang.Thread.run(Thread.java:745)
Locked ownable synchronizers:
- <0x000000075bc59aa8> (a java.util.concurrent.ThreadPoolExecutor$Worker)
DifferentClass extends abstract MyClass and there is a call from method1() to DifferentClass, where a DTO object is passed to a method that does a lot of processing, logging and finally saving to a database. The proxy class was used during the creation of one of the database handling classes.
Your best option is to find a way to get a stack trace of the running program. Here is one possible way.
I suggest running the program using the debug mode in your IDE and put break points next to what may appear to be the problem. Then step into (e.g. F7 in Netbeans) at the point where the program makes a delay. You can step in all the way to obfuscated code though you may not be able to fix the issue there. However you will have known where the delay is.
Related
I have multi threads that want to put a value in a TreeSet<Long>, in a part of code. The values are almost unique because they are System.nanoTime(). I clean the TreeSets periodically. Problem is that sometimes my threads got blocked in TreeSet.add() function. I used jconsole to watch thread states, threads are in RUNNABLE state and the stack trace shows this:
java.util.TreeMap.put(TeeMap.java:567)
java.util.TreeSet.add(TreeSet.java:255)
... //my program stack trace
I'm using jdk 1.7.0_60 for running program. Also I should mention in this situation cpu usage become 100%. My question is why the threads got blocked and how can I fix the situation? I looked at TreeMap code, but I didn't figure out problem, but I think problem relates to while loop in TreeMap.put().
As it was mentioned in comments, the problem is that TreeSet is not thread safe and if we want to modify it (add or remove data) in multi threads, it must be synchronized externally.
I have a swing application that freezes after some (random) time. I have taken 5 thread snapshots every 10 seconds after it freezes and they all contain these exact same lines:
"AWT-EventQueue-0" prio=6 tid=0x0000000009949000 nid=0x7bec waiting on condition [0x000000000ebbc000]
java.lang.Thread.State: RUNNABLE
at java.math.BigInteger.valueOf(Unknown Source)
at java.math.BigDecimal.inflate(Unknown Source)
at java.math.BigDecimal.add(Unknown Source)
at uk.co.xx.xxx.xxxx.XXX$4.get(XXX.java:171)
Note that no other thread in the thread dump is in XXX.java.
The corresponding code line (XXX.java:171) looks somewhat inoffensive:
a = a.add(b.multiply(c, MATH_CONTEXT), MATH_CONTEXT);
where:
a, b and c are local BigDecimal variables.
MATH_CONTEXT is a public final static variable, only accessed within XXX.java
My questions (answering any of them would be great help)
Is this evidence of a deadlock or liveness issue (the thread does not seem to make progress but it is in RUNNABLE state)?
Is this the likely reason for the freeze or should I look somewhere else?
What would be the next step to solve the problem?
Is this evidence of a deadlock or liveness issue (the thread does not seem to make progress but it is in RUNNABLE state)?
I doubt it. Since the program freezes, there clearly is an issue. However, I doubt there's a deadlock involving the code you've shown.
Is this the likely reason for the freeze or should I look somewhere else?
I think it's likely that this is a red herring, and the problem lies elsewhere.
What would be the next step to solve the problem?
I personally would look into potential memory allocation and garbage collection issues. In particular, I would make sure the program isn't spending all of its time collecting garbage and therefore failing to make progress.
To do this, I'd use a memory profiler.
While I am at it, I would also monitor the overall CPU and memory usage of the process, and page fault statistics (to see if there's excessive swapping).
I do no see a solution, but i can describe the steps i would start with.
I would try to attache a Profiler and check, whether the memory is growing, because the system could swap memory and that why it seems to hang, but i does not.
The profiler also tells you, if the Thread is really hanging and if so, where it seem to hang.
For profiling i use VisualVM.
In an open source application I'm participating, we've got a bug, where the application doesn't always close properly. That's what I'd like to solve.
Experience has shown that this happens most of the time when threads and processes are being started, but not managed correctly (e.g. a thread is waiting on a socket connection, the application is being shut down and the thread keeps on waiting).
With this in mind I've searched for '.start()' in the entire source and found 53 occurrences (which scared me a bit).
As a first step, I wanted to create a helper class (ThreadExecutor) where the current code 'thread.start()' would be replaced by 'ThreadExecutor.Execute(thread)' to have a) only a few changes in the existing source and b) a single class where I can easily check which threads don't end as they should. To do this I wanted to
add the thread to be executed to a list called activeThreads when calling the Execute method
start the thread
remove it from the activeThreads list when it ends.
This way I'd have an up to date list of all executing threads and when the app hangs on shutdown I could see in there which thread(s) is(are) causing it.
Questions:
What do you think about the concept? I'm usually coding c# and know how I'd do it using .NET with workers, but am not too sure what's best in Java (I'd like to modify as few lines of code as possible in the existing source).
If the concept seems ok, how can I get notified of a thread terminating. I'd like to avoid having an additional thread checking every once in a while what the state of all threads contained in activeThreads is, to remove them if they terminated.
Just to clarify: Before figuring out how to terminate the application properly, what I'm asking here is what's the best/easiest way to find which threads are at cause for certain test cases which are pretty hard to reproduce.
I would attempt to analyze your application's behavior before changing any code. Code changes can introduce new problems - not what you want to do if you're trying to solve problems.
The easiest way to introspect the state of your application with regard to which threads are currently running is to obtain a thread dump. You said that your problem is that the application hangs on shutdown. This is the perfect scenario to apply a thread dump. You'll be able to see which threads are blocked.
You can read more about thread dumps here.
Try to make all threads daemon(when all remaining threads are daemon the JVM terminates). Use thread.setDaemon(true) before starting each thread.
You could try to look into your application using jvisualvm (which is shipped with the jdk, find it in the bin folder of your jdk). JVisualVM can connect to your application and display a lot of interesting information, including which processes are still running. I'd give that a shot before starting down the road you describe.
Here is some documentation on JVisualVM should you need it.
The best way in java is to use Thread pools instead of Threads directly (although using threads directly is accepted). Thread pools accept Runnable objects, which you can see as Tasks. The idea is that most threads would do a small task and then end, because making a Thread is expensive and harder to manager you can use the threadpool, which allows things like 'ThreadPoolExecutor.awaitTermination()`. If you have more tasks than Threads in the pool, remaining tasks will just be queued.
Changing a Thread into a Runnable is easy, and you can even execute a runnable on a Thread you make yourself.
Note that this might not work for threads that run a long time, but your question seems to suggest that they will eventually finish.
As for your second question, the best way to find out which threads are running at a certain point is to run the application in a debugger (such as Eclipse) and pause all threads on a breakpoint in the close function.
I would try the trial edition of jprofiler or something similar, which gives you a lot of insight into what your application and its threads actually do.
Don't change the code yet, but try to reproduce and understand when this happens.
Create yourself a static thread pool.
static ExecutorService threads = Executors.newCachedThreadPool();
For every start of thread change:
new Thread(new AThread()).start();
to
threads.submit(new AThread ());
When your code exits, list all running threads with:
List<Runnable> runningThreads = threads.shutdownNow();
for ( Runnable t : runningThreads ) {
System.out.println("Thread running at shutdown: "+t.toString());
}
This will not only shut down all running threads, it will list them out for you to see what their issue is.
EDIT: Added
If you want to keep track of all running threads use:
Future f = threads.submit(new AThread ());
and store it in a list somewhere. You can then find out about its state with calls like:
f.isDone();
... etc.
I am continuing my path to deep understanding of Java Thread. Unfortunately my Java Certification didn't cover that part, so the only way of learning is to post a series of dumb questions. With so many years of Java Development, I am sometimes wondering how much I still have to learn :-)
In particular my attention is now with the reference handler thread.
"Reference Handler" daemon prio=10 tid=0x02da3400 nid=0xb98 in Object.wait() [0x0302f000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x1aac0320> (a java.lang.ref.Reference$Lock)
at java.lang.Object.wait(Object.java:485)
at java.lang.ref.Reference$ReferenceHandler.run(Unknown Source)
- locked <0x1aac0320> (a java.lang.ref.Reference$Lock)
Now some questions are following, for some of them I know the answer, but I am not posting it, because I would like to hear someone else opinions:
What is the Reference Handler thread supposed to do ?
A thread dump should be considered bottom up, why does the stack trace start with locked, shouldn't the lock statement appears at least after the thread has run ?
What does "Native Method" means ?
Why "Unknown Source", in which case the thread dump cannot recall the source code ?
Lastly the waiting on and locked has the same , why ?
as usual, I kindly ask to answer all the questions, so that I can mark answered.
I suspect it handles running finalizers for the JVM. It's an implementation detail and as such not specified in the JVM spec.
This only means that the java.lang.ref.Reference$Lock was locked in the method mentioned in the line preceding it (i.e in ReferenceHandler.run().
"Native Method" simply means that the method is implemented in native (i.e. non-Java) code (think JNI).
Unknown Source only means that the .class file doesn't contain any source code location information (at least for this specific point). This can happen either when the method is a synthetic one (doesn't look like it here), or the class was compiled without debug information.
When a thread waits on some object, then it must have locked that object at some point down the call trace, so you can't really have a waiting on without a corresponding locked.
1) The Finalizer Thread calls finalizer methods.
The Reference Thread has a similar purpose.
http://www.java2s.com/Open-Source/Java-Document/6.0-JDK-Core/lang/java/lang/ref/Reference.java.htm
The OpenJDK source states its is a
High-priority thread to enqueue pending References
The GC creates a simple linked list of references which need to be processed and this thread quickly adds them to a proper queue. The reason this is done in two phases is that the GC does nothing but find the References, this thread calls code which handles those references e.g. Call Cleaners, and notifies ReferenceQueue listeners.
2) A lock is acquired for a synchronized method before it is entered.
3-5) covered by Joachim ;)
Wow, too deep for me. I can only answer one or two of your questions.
"Native Method" simply means the implementation of that method is in some native (i.e. C or C++) library. Once the call stack has "gone native", the JVM can no longer monitor it. No way for it to provide additional stack information.
"Unknown Source" likely means the code was compiled with optimization turned on and debugging info turned off (-g flag?). This eliminates the file/line information from the .class file.
I am writing a Java application in which i m writing a thread program to read a file. Every time the program is run it will create a thread and read the file. Its time consuming, I know the file will never change so I want to make a daemon thread which will read the file only once and store it in a String.
The am facing several problems-
1) Once i start the daemon thread, How do i access it again?
2)If I want to stop the daemon thread, how do i do it?
Please help.
thanks,
I think you are confused with the way a daemon thread works. A daemon thread doesn't prevent the application from quitting if it's the only thread running; user threads do. If you know the file is never going to change, why not load it once without using any thread? Also, after the file loading is done by your daemon thread (i.e. the run() method completes normally), it'll automatically be handled by your runtime unless you've got an infinite loop in your run() method. IMO posting a bit of code would help the cause here.
I guess, your daemon will live in a different virtual machine, and in this case, you can't access the String from your application.
If you want to access your thread, just keep a reference to the Thread object. Likewise, if you want to access your string keep a reference to the String object. These references could be stored as static variables, but they don't have to be.
Stopping a thread directly via Thread.stop() or Thread.suspend() is deprecated. See this article for a description of why, and a "proper" way to stop threads.