synchronised()
{
//line1
//line2
// line: waiting indefinitely for DB connection
//line 3
}
How to ensure that other threads get access to these lines and the thread currently deadlocked here, somehow releases the lock?
Related
I'm investigating the strange issue with tomcat shutdown process: after runnig shutdown.sh the java process still appears(I check it by using ps -ef|grep tomcat)
The situation is a bit complicated, because I have very limitted access to the server(no debug, for example)
I took thread dump(by using kill -3 <PID>) and heap dump by using remote jConsole and Hotspot features.
After looking into thread dump I found this:
"pool-2-thread-1" #74 prio=5 os_prio=0 tid=0x00007f3354359800 nid=0x7b46 waiting on condition [0x00007f333e55d000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000c378d330> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:748)
So, my understanding of the problem is follows: There is a resource(DB connection or something else) which is used in CachedThreadpool, and this resource is now locked,
and prevent to thread pool-2-thread-1 to stop. Assuming that this thread isn't deamon - JVM cannot gracefully stop.
Is there a way to find out which resource is locked, from where is it locked and how to avoid that? Another question is - how to prevent this situation?
Another this is: what the adress 0x00007f333e55d000 is for?
Thanks!
After a wail of struggling I found out, that the freezing thread always have the same name pool-2-thread-1. I grep all the sources of the project to find any places where any scheduled thread pool is started: Executors#newScheduledThreadPool. After a huge amount of time I logged all that places, with thread name in thread pool.
One more restart server and gotcha!
I found out one thread pool, that is started with only one thread and used following code:
private void shutdownThreadpool(int tries) {
try {
boolean terminated;
LOGGER.debug("Trying to shutdown thread pool in {} tries", tries);
pool.shutdown();
do {
terminated = pool.awaitTermination(WAIT_ON_SHUTDOWN,TimeUnit.MILLISECONDS);
if (--tries == 0
&& !terminated) {
LOGGER.debug("After 10 attempts couldn't shutdown thread pool, force shutdown");
pool.shutdownNow();
terminated = pool.awaitTermination(WAIT_ON_SHUTDOWN, TimeUnit.MILLISECONDS);
if (!terminated) {
LOGGER.debug("Cannot stop thread pool even with force");
LOGGER.trace("Some of the workers doesn't react to Interruption event properly");
terminated = true;
}
} else {
LOGGER.info("After {} attempts doesn't stop", tries);
}
} while (!terminated);
LOGGER.debug("Successfully stop thread pool");
} catch (final InterruptedException ie) {
LOGGER.warn("Thread pool shutdown interrupted");
}
}
After that the issue was solved.
So, if i have understood this correctly, a thread goes into waiting state when we call wait on an object and it goes into blocked state when it is waiting for a lock on an object(like when trying to get into a synchronized block or method).
How does I/O-methods like read() put a Thread in a blocked state though? I understand WHY it has to be in a blocked state, waiting for data that it can read but i'm also interested in HOW. How does the JVM notify the thread that it can continue when data in the resource its trying to read, is available again?
It doesn't change the state of the thread to BLOCKED
public static void main(String[] args) throws IOException {
Thread main = Thread.currentThread();
new Thread(() -> {
for (int i = 0; i < 10; i++) {
System.out.println(main + " is in "+main.getState()+" state");
try {
Thread.sleep(100);
} catch (InterruptedException e) {
throw new AssertionError(e);
}
}
}).start();
System.in.read();
}
prints
Thread[main,5,main] is in RUNNABLE state
Thread[main,5,main] is in RUNNABLE state
Thread[main,5,main] is in RUNNABLE state
Thread[main,5,main] is in RUNNABLE state
Thread[main,5,main] is in RUNNABLE state
Thread[main,5,main] is in RUNNABLE state
Thread[main,5,main] is in RUNNABLE state
Thread[main,5,main] is in RUNNABLE state
Thread[main,5,main] is in RUNNABLE state
Thread[main,5,main] is in RUNNABLE state
instead the OS doesn't return from the read until there is some data and the OS decides whether and when to context switch the thread/process.
How does the JVM notify the thread that it can continue when data in the resource its trying to read, is available again?
The OS wakes the thread when there is more data or the stream has been closed. The JVM doesn't get involved.
This depends on the native platform.
In POSIX, a call to read usually blocks until data is available, but there are many other reasons to return, such as an end-of-file was reached, the file descriptor was closed, the operation timed out or a signal interrupted the operation.
In Windows, the most closely related function is ReadFile.
The gory details, refering to Java 8 update 112 b15:
FileInputStream.read calls the native FileInputStream.read0, implemented natively through JNI in Java_java_io_FileInputStream_read0, which calls readSingle, which calls IO_Read.
In POSIX, IO_Read is defined as handleRead, which calls read. The RESTARTABLE macro loops while there's an error and the errno is EINTR.
In Windows, IO_Read is defined as handleRead, which calls ReadFile.
My Daemon create and runs a function on a different thread this function runs many other functions. I want to check before each function if the Daemon was closed and if not then i will perform the function.
How can i know if the Daemon was stopped?
Just like any other thread, you have to get the thread instance and then call:
thread.isAlive()
class Daemon extends Thread()
{
private boolean started;
public void Daemon() {
started = false;
}
public void run() {
started = true;
// rest of your code.
}
public boolean isStoped() {
return started && !isAlive();
}
}
Use isStoped() to know when your thread has been stoped. isAlive() alone is not enough because a thread which has not been started will return false.
When code running in some thread creates a new Thread object, the new thread has its priority initially set equal to the priority of the creating thread, and is a daemon thread if and only if the creating thread is a daemon.
If you don't know if the thread is daemon or not then use isDaemon,
isDaemon
public final boolean isDaemon()
Tests if this thread is a daemon thread.
Returns:
true if this thread is a daemon thread; false otherwise.
Then you can ask the Thread for its current status by calling:
Thread.State ts = thread.getState();
and you should get one of the follwing:
A thread state. A thread can be in one of the following states:
NEW
A thread that has not yet started is in this state.
RUNNABLE
A thread executing in the Java virtual machine is in this state.
BLOCKED
A thread that is blocked waiting for a monitor lock is in this state.
WAITING
A thread that is waiting indefinitely for another thread to perform a particular action is in this state.
TIMED_WAITING
A thread that is waiting for another thread to perform an action for up to a specified waiting time is in this state.
TERMINATED
A thread that has exited is in this state.
Reference: http://docs.oracle.com/javase/7/docs/api/java/lang/Thread.html#isDaemon()
I have a web app being served by jetty + mysql. I'm running into an issue where my database connection pool gets exhausted, and all threads start blocking waiting for a connection. I've tried two database connection pool libraries: (1) bonecp (2) hikari. Both exhibit the same behavior with my app.
I've done several thread dumps when I see this state, and all the blocked threads are in this state (not picking on bonecp, I'm sure it's something on my end now):
"qtp1218743501-131" prio=10 tid=0x00007fb858295800 nid=0x669b waiting on condition [0x00007fb8cd5d3000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x0000000763f42d20> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
at com.jolbox.bonecp.DefaultConnectionStrategy.getConnectionInternal(DefaultConnectionStrategy.java:82)
at com.jolbox.bonecp.AbstractConnectionStrategy.getConnection(AbstractConnectionStrategy.java:90)
at com.jolbox.bonecp.BoneCP.getConnection(BoneCP.java:553)
at com.me.Foo.start(Foo.java:30)
...
I'm not sure where to go from here. I was thinking that I would see some stack traces in the thread dump where my code was stuck doing some lengthly operation, not waiting for a connection. For example, if my code looks like this:
public class Foo {
public void start() {
Connection conn = threadPool.getConnection();
work(conn);
conn.close();
}
public void work(Connection conn) {
.. something lengthy like scan every row in the database etc ..
}
}
I would expect one of the threads above to have a stack trace that shows it working away in the work() method:
...
at com.me.mycode.Foo.work()
at com.me.mycode.Foo.start()
but instead they're just all waiting for a connection:
...
at com.jolbox.bonecp.BoneCP.getConnection() // ?
at com.me.mycode.Foo.work()
at com.me.mycode.Foo.start()
Any thoughts on how to continue debugging would be great.
Some other background: the app operates normally for about 45 minutes, mem and thread dumps show nothing out of the ordinary. Then the condition is triggered and the thread count spikes up. I started thinking it might be some combination of sql statements the app is trying to perform which turn into some sort of lock on the mysql side, but again I would expect some of the threads in the stack traces above to show me that they're in that part of the code.
The thread dumps were taken using visualvm.
Thanks
Take advantage of the configuration options for the connection pool (see BoneCPConfig / HikariCPConfig). First of all, set a connection time-out (HikariCP connectionTimeout) and a leak detection time-out (HikariCP leakDetectionThreshold, I could not find the counterpart in BoneCP). There might be more configuration options that dump stack-traces when something is not quite right.
My guess is that your application does not always return a connection to the pool and after 45 minutes has no connection in the pool anymore (and thus blocks forever trying to get a connection from the pool). Treat a connection like opening/closing a file, i.e. always use try/finally:
public void start() {
Connection conn = null;
try {
work(conn = dbPool.getConnection());
} finally {
if (conn != null) {
conn.close();
}
}
}
Finally, both connection pools have options to allow JMX monitoring. You can use this to monitor for strange behavior in the pool.
I question the whole design.
If you have a waiting block in a multithreaded netIO, you need a better implementation of the connection.
I suggest you take a look at non blocking IO (Java.nio, channels package), or granulate your locks.
I just went through the source code of ThreadPoolExecutor found that it will interrupt all idle workers once the time is up to the set value of keepAliveTime and allowCoreThreadTimeOut is true.
It's a little strange to me it can only invoke the interrupt method when runState >= SHUTDOWN:
The code below is from the method getTask() of ThreadPoolExecutor.
Runnable getTask() {
...
if (workerCanExit()) {
if (runState >= SHUTDOWN) // Wake up others
interruptIdleWorkers();
return null;
}
}
Does that mean all the idle threads can only be interrupted when the runState >= SHUTDOWN (SHUTDOWN, STOP or TERMINATED)? That's to say they will be not interrupted when the state is RUNNING.
You are right. This getTask() method in the ThreadPoolExecutor is called on to gets the next task for a worker thread to run. This code block is only executed when the method call has not identified any Runnable task for execution. So, if nothing is found to execute, it must check for the shutdown state.
from java doc of workerCanExit()
Check whether a worker thread that fails to get a task can exit. We allow a worker thread to die if the pool is stopping, or the queue is empty, or there is at least one thread to handle possibly non-empty queue, even if core timeouts are allowed.
As an example, configure ThreadPoolExecutor as : corePoolSize=1, maxPoolSize=5, workQueueSize=1, keepAliveTime=60s, allowCoreThreadTimeOut=false.
When you offer 5 tasks(each task is time-consuming) concurrently, one of the 5 tasks will enter into workQueue, other 4 tasks will be handled immediately by 4 worker threads newly created almost at the same time.
At this time, the sum of worker threads is 4(workerThreadCount=4). Once one thread completes its task, it will be waiting on workQueue by invoking blocking method, workQueue.take() or workQueue.poll(keepAliveTime). As for which blocking method will be invoked is decided by the workerThreadCount.
For example(Hypothesis), at a time point, workerThread-0 is handling task-0; task-1 is staying in workQueue; workerThread-1 is handling task-2; workerThread-2 is handling task-3; workerThread-3 is handling task-4, and workerThreadCount==4.
workerThread-3 completes task-4, and this time [workerThreadCount==4] > [corePoolSize==1], it will get next task(task-1) from workQueue by workQueue.poll(keepAliveTime). Then, go on handling task-1, and this time workerThreadCount==4. The code segment of ThreadPoolExecutor.java as following:
while (task != null || (task = getTask()) != null) {
task.run();
}
private Runnable getTask() {
boolean timed = allowCoreThreadTimeOut || wc > corePoolSize;
Runnable r = timed ? workQueue.poll(keepAliveTime, TimeUnit.NANOSECONDS) : workQueue.take();
}
then workerThread-0 completes task-0, because of [workerThreadCount==4] > [corePoolSize==1], it still get next task from workQueue by workQueue.poll(keepAliveTime). But, this time workQueue is empty, so workerThread-0 will be blocked on it, and the state of workerThread-0 is TIMED_WAITING. Once keepAliveTime elapsed, workQueue.poll(keepAliveTime) will return null, next, workerThread-0 will return from Runnable.run(), and turn into TERMINATED. This time workerThreadCount==3.
then workerThread-1 completes task-2, it will return by the same way as workerThread-0. This time workerThreadCount==2.
then workerThread-3 completes task-1, it will return by the same way as workerThread-1. This time workerThreadCount==1.
then workerThread-2 completes task-3, but this time [workerThreadCount==1] not more than [corePoolSize==1], so it will be blocked when get the next task from workQueue by workQueue.take() until there is a available task in workQueue. And its state is WAITING.
NOTE : the source code comes from JDK8.
Exactly. A correct task(if interruptions are allowed) must checks itself for interrupted flag and terminate (i.e. return from run()).