I've been caught by yet another deadlock in our Java application and started thinking about how to detect potential deadlocks in the future. I had an idea of how to do this, but it seems almost too simple.
I'd like to hear people's views on it.
I plan to run our application for several hours in our test environment, using a typical data set.
I think it would be possible to perform bytecode manipulation on our application such that, whenever it takes a lock (e.g. entering a synchronized block), details of the lock are added to a ThreadLocal list.
I could write an algorithm that, at some later point, compares the lists for all threads and checks if any contain the same pair of locks in opposite order - this would be reported as a deadlock possibility. Again, I would use bytecode manipulation to add this periodic check to my application.
So my question is this: is this idea (a) original and (b) viable?
This is something that we talked about when I took a course in concurrency. I'm not sure if your implementation is original, but the concept of analysis to determine potential deadlock is not unique. There are dynamic analysis tools for Java, such as JCarder. There is also research into some analysis that can be done statically.
Admittedly, it's been a couple of years since I've looked around. I don't think JCarder was the specific tool we talked about (at least, the name doesn't sound familiar, but I couldn't find anything else). But the point is that analysis to detect deadlock isn't an original concept, and I'd start by looking at research that has produced usable tools as a starting point - I would suspect that the algorithms, if not the implementation, are generally available.
I have done something similar to this with Lock by supplying my own implementation.
These days I use the actor model, so there is little need to lock the data (as I have almost no shared mutable data)
In case you didn't know, you can use the Java MX bean to detect deadlocked threads programmatically. This doesn't help you in testing but it will help you at least better detect and recover in production.
ThreadMXBean threadMxBean = ManagementFactory.getThreadMXBean();
long[] deadLockedThreadIds = threadMxBean.findMonitorDeadlockedThreads();
// log the condition or even interrupt threads if necessary
...
That way you can find some deadlocks, but never prove their absence. I'd better develop static checking tool, a kind of bytecode analizer, feeded with annotations for each synchronized method. Annotations should show the place of the annotated method in the resource graph. The task is then to find loops in the graph. Each loop means deadlock.
Related
I am trying understand a rationale behind biased locking and making it a default. Since reading this blog post, namely:
"Since most objects are locked by at most one thread during their lifetime, we allow that thread to bias an object toward itself"
I am perplexed... Why would anyone design a synchronized set of methods to be accessed by one thread only? In most cases, people devise certain building blocks specifically for the multi-threaded use-case, and not a single-threaded one. In such cases, EVERY lock aquisition by a thread which is not biased is at the cost of a safepoint, which is a huge overhead! Could someone please help me understand what I am missing in this picture?
The reason is probably that there are a decent number of libraries and classes that are designed to be thread safe but that are still useful outside of such circumstances. This is especially true of a number of classes that predate the Collections framework. Vector and it's subclasses is a good example. If you also consider that most java programs are not multi threaded it is in most cases an overall improvement to use a biased locking scheme, this is especially true of legacy code where the use of such Classes is all to common.
You are correct in a way, but there are cases when this is needed, as Holger very correctly points in his comment. There is so-called, the grace period when no biased-locking is attempted at all, so it's not like this will happen all the time. As I last remember looking at the code, it was 5 seconds. To prove this you would need a library that could inspect Java Object's header (jol comes to my mind), since biased locking is hold inside mark word. So only after 5 seconds will the object that held a lock before will be biased towards the same lock.
EDIT
I wanted to write a test for this, but seems like there is one already! Here is the link for it
Can anyone please explain to me the consequences of mutating a collection in java that is not thread-safe and is being used by multiple threads?
The results are undefined and somewhat random.
With JDK collections that are designed to fail fast, you might receive a ConcurrentModificationException. This is really the only consequence that is specific to thread safety with collections, as opposed to any other class.
Problems that occur generally with thread-unsafe classes may occur:
The internal state of the collection might be corrupted.
The mutation may appear to be successful, but the changes may not, in fact, be visible to other threads at any given time. They might be invisible at first and become visible later.
The changes might actually be successful under light load, but fail randomly under heavy load with lots of threads in contention.
Race conditions might occur, as was mentioned in a comment above.
There are lots of other possibilities, none of them pleasant. Worst of all, these things tend to most commonly reveal themselves in production, when the system is stressed.
In short, you probably don't want to do that.
The most common outcome is it looks like it works, but doesn't work all the time.
This can mean you have a problem which
works on one machine but doesn't on another.
works for a while but something apparently unrelated changes and your program breaks.
whenever you have a bug you don't know if it's a multi-threading issue or not if you are not using thread safe data structures.
What can happen is;
you rarely/randomly get an error and strange behaviour
your code goes into an infinite loop and stops working (HashMap used to do this)
The only option is to;
limit the amount of state which is shared between threads, ideally none at all.
be very careful about how data is updated.
don't rely on unit tests, you have to understand what the code doing and be confident it will be behave correctly in all possible situations.
The invariants of the data structure will not be guaranteed.
For example:
If thread 2 does a read whilst thread 1 is adding to the DS thread 1 may consider this element added while thread 2 doesn't see that the element has been added yet.
There are plenty of data structures that aren't thread-safe that will still appear to function(i.e. not throw) in a multi threaded environment and they might even perform correctly under certain circumstances(like if you aren't doing any writes to the data structure).
To fully understand this topic exploring the different classes of bugs that occur in concurrent systems is recommended: this short document seems like a good start.
http://pages.cs.wisc.edu/~remzi/OSTEP/threads-bugs.pdf
I know that thread safety of java sockets has been discussed in several threads here on stackoverflow, but I haven't been able to find a clear answer to this question - Is it, in practice, safe to have multiple threads concurrently write to the same SocketOutputStream, or is there a risk that the data sent from one thread gets mixed up with the data from another tread? (For example the receiver on the other end first receives the first half of one thread's message and then some data from another thread's message and then the rest of the first thread's message)
The reason I said "in practice" is that I know the Socket class isn't documented as thread-safe, but if it actually is safe in current implementations, then that's good enough for me. The specific implementation I'm most curious about is Hotspot running on Linux.
When looking at the Java layer of hotspot's implementation, more specifically the implementation of socketWrite() in SocketOutputStream, it looks like it should be thread safe as long as the native implementation of socketWrite0() is safe. However, when looking at the implemention of that method (j2se/src/solaris/native/java/net/SocketOutputStream.c), it seems to split the data to be sent into chunks of 64 or 128kb (depending on whether it's a 64bit JVM) and then sends the chunks in seperate writes.
So - to me, it looks like sending more than 64kb from different threads is not safe, but if it's less than 64kb it should be safe... but I could very well be missing something important here. Has anyone else here looked at this and come to a different conclusion?
I think it's a really bad idea to so heavily depend on the implementation details of something that can change beyond your control. If you do something like this you will have to very carefully control the versions of everything you use to make sure it's what you expect, and that's very difficult to do. And you will also have to have a very robust test suite to verify that the multithreaded operatio functions correctly since you are depending on code inspection and rumors from randoms on StackOverflow for your solution.
Why can't you just wrap the SocketOutputStream into another passthrough OutputStream and then add the necessary synchronization at that level? It's much safer to do it that way and you are far less likely to have unexpected problems down the road.
According to this documentation http://www.docjar.com/docs/api/java/net/SocketOutputStream.html, the class does not claim to be thread safe, and thus assume it is not. It inherits from FileOutputStream, which normally file I/O is not inherently thread safe.
My advice is that if the class is related to hardware or communications, it is not thread safe or "blocking". The reason is thread safe operations consume more time, which you may not like. My background is not in Java but other libraries are similar in philosophy.
I notice you tested the class extensively, but you may test it all day for many days, and it may not prove anything, my 2-cents.
Good luck & have fun with it.
Tommy Kwee
In highly concurrent systems, it can be difficult to be confident that your usage of locks is correct. Specifically, deadlocks can result if locks are acquired in an order that was not expected while being acquired in the proper order in another thread.
There are tools (e.g. Coverity) which can do static analysis on a code base and look for "unusual" locking orders. I'd like to explore other options to meet my needs.
Are there any light-weight* tools for instrumenting Java code which can detect cases where locks are being acquired in an order other than expected? I am okay with explicitly calling out locking orders via comments / annotations.
Free and/or open-source solutions preferred. Please also comment if there are non-instrumentation approaches to this problem.
* For my purposes, light-weight means...
If it is instrumentation, I can still run my program with the same ballpark performance. 30-50% degradation is acceptable, I suppose.
I don't have to spend half the day interacting with the tool just to get an "okay" out of it. Ideally I should only notice that I'm using it when there's a problem.
If it is instrumentation, it should be easy to disable for production environments.
It shouldn't clutter my code at every synchronize statement. As previously mentioned, I'm okay with explicitly commenting/annotating the objects or classes of objects which get locked with relative orderings.
I have not used AspectJ so cannot vouch for how easy it is to use. I have used ASM to create a custom code profiler, this was about 2 days work. The effort to instrument synchronization should be similar. AspectJ should be quicker and easier once you are up to speed with aspects.
I have implemented deadlock detecting trace for our c++ based server. Here is how I did it:
When ever acquiring or releasing a lock I traced:
<time> <tid> <lockid> <acquiring|releasing> <location in code>
This extra trace affected performance quite drastically and was not usable in production.
So when a possible deadlock was discovered in production I used the log file to figure out what was happening around the deadlock. Then reproduced this functionality in a test environment with my tracing turned on.
Then I ran a script on the log file to see if deadlock was possible and how. I used an awk script, using this algoritm:
Foreach line
if acquiring
add lockid to list of current locks for this thread
add each pair of locks in this list to a set lock pairs for this thread. eg for list of Lock A -> Lock B -> Lock C generate the pairs (Lock A, Lock B), (Lock A, Lock C), (Lock B, Lock C)
if releasing
remove current lockid from tail of list for this thread
For each lock pair search all other threads for the reverse lock pairs, each match is a potential deadlock so print the pairs and threads affected
Instead of making the algorithm smarter I then desk checked that the lock acquisition to see if it was a real deadlock.
I did this after failing to find the cause of a deadlock for a number of days, it took a few more days to implement and a few hours to find the deadlock.
If you are considering this approach in Java things to consider are:
Do you only use synchronized to protect your critical sections? Are you using the classes in java.lang.concurrent? (these might require special handling/instrumentation)
How easy is it to print the code location with aspects/ASM? I used __FILE__ and __LINE__ in c++. ASM will give you the class name, method name and signature.
You cannot instrument the locks used to protect your tracing/logging.
You can streamline your instrumentation if you use a log file per thread and thread local storage for the file object.
How do you uniquely identify objects you synchronize on? Maybe toString() and System.identityHashCode() would be enough, but might require more. I used the address of the object in C++.
You can use AspectJ, which is relatively easy to learn and will allow you to setup your own customized and simplified way of monitoring your threads and any locks they access.
Doesn't get you all the way there, but a good start is to use the JCIP annotations, and FindBugs catches a few things.
I'm wondering what good ways there would be make assertions about synchronization or something so that I could detect synchronization violations (while testing).
That would be used for example for the case that I'd have a class that is not thread-safe and that isn't going to be thread-safe. With some way I would have some assertion that would inform me (log or something) if some method(s) of it was called from multiple threads.
I'm longing for something similar that could be made for AWT dispatch thread with the following:
public static void checkDispatchThread() {
if(!SwingUtilities.isEventDispatchThread()) {
throw new RuntimeException("GUI change made outside AWT dispatch thread");
}
}
I'd only want something more general. The problem description isn't so clear but I hope somebody has some good approaches =)
You are looking for the holy grail, I think. AFAIK it doesn't exist, and Java is not a language that allows such an approach to be easily created.
"Java Concurrency in Practice" has a section on testing for threading problems. It draws special attention to how hard it is to do.
When an issue arises over threads in Java it is usually related to deadlock detection, more than just monitoring what Threads are accessing a synchronized section at the same time. JMX extension, added to JRE since 1.5, can help you detect those deadlocks. In fact we use JMX inside our own software to automatically detect deadlocks an trace where it was found.
Here is an example about how to use it.
IntelliJ IDEA has a lot of useful concurrency inspections. For example, it warns you when you are accessing the same object from both synchronised and unsynchronised contexts, when you are synchronising on non-final objects and more.
Likewise, FindBugs has many similar checks.
As well as #Fernando's mention of thread deadlocking, another problem with multiple threads is concurrent modifications and the problems it can cause.
One thing that Java does internally is that a collection class keeps a count of how many times it's been updated. And then an iterator checks that value on every .next() against what it was when the interator was created to see if the collection has been updated while you were iterating. I think that principle could be used more generally.
Try ConTest or Covertity
Both tools analyze the code to figure out which parts of the data might be shared between threads and then they instrument the code (add extra bytecode to the compiled classes) to check if it breaks when two threads try to change some data at the same time. The two threads are then run over and over again, each time starting them with a slightly different time offset to get many possible combinations of access patterns.
Also, check this question: Unit testing a multithreaded application?
You might be interested in an approach Peter Veentjer blogged about, which he calls The Concurrency Detector. I don't believe he has open-sourced this yet, but as he describes it the basic idea is to use AOP to instrument code that you're interested in profiling, and record which thread has touched which field. After that it's a matter of manually or automatically parsing the generated logs.
If you can identify thread unsafe classes, static analysis might be able to tell you whether they ever "escape" to become visible to multiple threads. Normally, programmers do this in their heads, but obviously they are prone to mistakes in this regard. A tool should be able to use a similar approach.
That said, from the use case you describe, it sounds like something as simple as remembering a thread and doing assertions on it might suffice for your needs.
class Foo {
private final Thread owner = Thread.currentThread();
void x() {
assert Thread.currentThread() == owner;
/* Implement method. */
}
}
The owner reference is still populated even when assertions are disabled, so it's not entirely "free". I also wouldn't want to clutter many of my classes with this boilerplate.
The Thread.holdsLock(Object) method may also be useful to you.
For the specific example you give, SwingLabs has some helper code to detect event thread violations and hangs. https://swinghelper.dev.java.net/
A while back, I worked with the JProbe java profiling tools. One of their tools (threadalyzer?) looked for thread sync violations. Looking at their web page, I don't see a tool by that name or quite what I remember. But you might want to take a look. http://www.quest.com/jprobe/performance-home.aspx
You can use Netbeans profiler or JConsole to check the threads status in depth