I'm having trouble getting over 100 threads to run simultaneously. When I do a thread dump, I noticed that many of them are in parked status, i.e.
parking to wait for <0x00000000827e1760> (java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject).
The program runs fine with about 25 threads or less. Is there a way ti identify what's causing the concurrent lock, and/or prevent it? This was running in a fixed pool size of 200 using the Executor service.
Apologies for the lack of code - it's proprietary and there's a lot to be changed to obfuscated it.
Are you using some sort of ThreadPoolExecutor such as the ones provided by java.util.concurrent.Executors class? Perhaps you are facing a case of tasks being finished by silently uncaught exceptions. The dump fragment looks like an inactive pooled thread and one reason to get an inactive thread (which should be active) is an exception throwed up but surrounded by the default thread pool implementation.
LockSupport.park()
In thread pools, THREADS waiting for a TASK are locked out by LockSupport.park();. See java.util.concurrent.locks.AbstractQueuedSynchronizer source from openjdk :
public final void await() throws InterruptedException {
// code omitted
while (!isOnSyncQueue(node)) {
LockSupport.park(this);
if ((interruptMode = checkInterruptWhileWaiting(node)) != 0)
break;
}
// code omitted
}
It means that the TASK which the THREAD were executing finished (abruptaly or not) and now the thread is waiting for another task to execute (see java.util.concurrent.ThreadPoolExecutor openjdk source):
private Runnable getTask() {
// ...
Runnable r = timed ?
workQueue.poll(keepAliveTime, TimeUnit.NANOSECONDS) :
workQueue.take(); <== the thread is blocked here
// ...
}
As one can see, the thread is locked out in the call workQueue.take();.
Thus, shortly, threads in "parked status" are just waiting for new tasks after the previous ones have finished.
Why does my task is no longer running?
The most reasonable cause of a finished task is the regular end of the run(). The task flow finishes and then the task is released by the respective owner thread. Once the thread releases the task, it is ready to execute another task as long there is one.
A straightforward way to check this scenario is by logging something in the end of the run() method:
class MyRunnable implements Runnable {
public void run() {
while(/*some condition*/) {
// do my things
}
log.info("My Runnable has finished for now!");
}
}
If log a message is not enough you can call a method of another object instead.
Exceptions under the wood
Another (most) probable cause is an uncaught exception thrown during the task execution. Within a thread pool, an unchecked exception like this will abruptaly stop the method execution and (surprisely) be swallowed into a java.util.concurrent.FutureTask object. In order to avoid things like this, I use the following idiom:
class MyRunnable implements Runnable {
public void run() {
while(/*some condition*/) {
try {
// do my things
} catch (Throwable throwable) {
handle(throwable);
}
}
log.info("My Runnable has finished for now!");
}
private void handle(Throwable throwable) {
// ...
}
}
or depending on the logic/performance requirements I also use:
public void run() {
try {
while(/*some condition*/) {
// do my things
}
} catch (Throwable throwable) {
handle(throwable);
}
System.out.println("My Runnable has finished for now!");
}
The code below exemplify the issues commented here in action:
package mypocs;
import java.util.concurrent.Executors;
import java.util.concurrent.ThreadPoolExecutor;
public class ExceptionSwallowingInThreadPoolsPoC {
public static void main(String[] args) {
ThreadPoolExecutor threadPoolExecutor = (ThreadPoolExecutor) Executors.newFixedThreadPool(10);
final Object LOCK = new Object();
threadPoolExecutor.submit(() -> {
while (true) {
synchronized (LOCK) {
System.out.println("Thread 'A' never ends");
}
Thread.sleep(1000L);
}
});
threadPoolExecutor.submit(() -> {
int lifespan = 3;
while (lifespan > 0) {
synchronized (LOCK) {
System.out.println("Thread 'B' is living for " + lifespan + " seconds");
}
lifespan--;
try {
Thread.sleep(1000L);
} catch (InterruptedException e) {
e.printStackTrace();
}
}
System.out.println("Thread 'B' finished");
});
threadPoolExecutor.submit(() -> {
int lifespan = 3;
while (lifespan > 0) {
synchronized (LOCK) {
System.out.println("Thread 'C' is living for " + lifespan + " seconds");
}
lifespan--;
if (lifespan < 1) {
throw new RuntimeException("lifespan reached zero");
}
try {
Thread.sleep(1000L);
} catch (InterruptedException e) {
e.printStackTrace();
}
}
System.out.println("Thread 'C' finished");
});
while (true) {
try {
Thread.sleep(1000L);
} catch (InterruptedException e) {
e.printStackTrace();
}
synchronized (LOCK) {
System.out.println("==== begin");
System.out.println("getActiveCount: " + threadPoolExecutor.getActiveCount());
System.out.println("getCompletedTaskCount: " + threadPoolExecutor.getCompletedTaskCount());
System.out.println("getPoolSize: " + threadPoolExecutor.getPoolSize());
System.out.println("==== end");
}
}
}
}
The code should output something like:
Thread 'A' never ends
Thread 'B' is living for 3 seconds
Thread 'C' is living for 3 seconds
Thread 'C' is living for 2 seconds
==== begin
getActiveCount: 3
getCompletedTaskCount: 0
getPoolSize: 3
==== end
Thread 'B' is living for 2 seconds
Thread 'A' never ends
==== begin
getActiveCount: 3
getCompletedTaskCount: 0
getPoolSize: 3
==== end
Thread 'C' is living for 1 seconds
Thread 'B' is living for 1 seconds
Thread 'A' never ends
Thread 'B' finished
==== begin
getActiveCount: 1
getCompletedTaskCount: 2
getPoolSize: 3
==== end
Thread 'A' never ends
Thread 'A' never ends
...
The class (ConditionObject) you are referring to is used to lock objects from being accessed concurrently by multiple threads. The Javadoc doesn't describe the thread state you mention, but here is my guess:
Your locked object is being blocked by one thread so long, that the other threads start to pile up on the lock. Once the thread holding the lock releases it, the next thread continues the aquire the lock. Until that new thread has done his work, new threads pile up behing the lock.
If my guess is right, then could:
reduce the time that each thread spends in the lock, or
distribute the threads on different locked things (if your problem permits that), or
you use an implementation that doesn't require locking.
Without knowing your problem domain, I hope that the information above is enough to point you into some direction that might be of help for you.
Related
I am trying to do a manual retry. But I feel the code is not thread safe.
Can anyone please provide suggestion on how to make it thread safe
while (retryCounter < maxRetries) {
try {
//TODO - add delay with config
Thread.sleep(3000);
t.run();
break;
} catch (Exception e) {
retryCounter++;
//TODO - Add audit logs
if (retryCounter >= maxRetries) {
LOG.info("Max retries exceeded");
//TODO - remove exception add audit logs
throw new RuntimeException("Max retry exceeded");
}
}
}
Thanks in advance
As pointed out by # SteffenJacobs, t.run does not execute the run logic on a separate thread but rather on the thread that makes the invocation. If you replace t.run with t.start then the run logic will be executed on a different thread asynchronously which means that exceptions in that new thread will never be handled by your catch block. For example the code below:
public static void main(String[] args) {
int retryCounter = 0;
int maxRetries = 3;
Thread t = new Thread(new Runnable() {
public void run() {
System.out.println("..." + Thread.currentThread());
throw new RuntimeException("Thrown by me!!!");
}});
while (retryCounter < maxRetries) {
try {
Thread.sleep(3000);
System.out.println("***" + Thread.currentThread());
t.start();
break;
} catch (Exception e) {
System.out.println("retrying attempt " + retryCounter);
System.out.println(e);
retryCounter++;
if (retryCounter >= maxRetries) {
throw new RuntimeException("Max retry exceeded");
}
} finally {
System.out.println("in finally");
}
}
}
prints:
***Thread[main,5,main]
in finally
...Thread[Thread-0,5,main]
Exception in thread "Thread-0" java.lang.RuntimeException: Thrown by me!!!
...
Process finished with exit code 0
Bottom line - To detect errors in your threaded tasks, you will need to think of a different approach.
You could consider using Futures (https://docs.oracle.com/javase/8/docs/api/java/util/concurrent/Future.html). In this case your tasks will need to return a status of their execution while the code that launched them waits on Future.get , examines the status and reruns the task as needed.
The answer of #David Soroko is very close, but I think it need little modification. As your code would run under multi-threaded environment, we should take care of Synchronization
If multiple thread tries to enter while loop, it may happen that 2 invocations of Retry Thread would get started, with 2 calls of t.start(). Ideally, we should start only single invocation during single iteration of while loop.
Little modification :
Declare LOCK object in your class level
// Object to ensure thread safety between multiple threads
private final LOCK = new Object();
Use LOCK object with synchronized block in while loop
synchronized(LOCK) { // This will ensure single thread entry at a time
while (retryCounter < maxRetries) {
try {
Thread.sleep(3000);
System.out.println("***" + Thread.currentThread());
t.start();
break;
} catch (Exception e) {
System.out.println("retrying attempt " + retryCounter);
System.out.println(e);
retryCounter++;
if (retryCounter >= maxRetries) {
throw new RuntimeException("Max retry exceeded");
}
} finally {
System.out.println("in finally");
}
}
}
The variables retryCounter and maxRetries are read by multiple threads. This means they are shared resources between threads. We need to ensure thread safety there as well. There is concept of Atomicity and Volatility.
Atomicity : When any thread T1 is doing operation on Variable A, at the same time, if other thread T2 tries to do operation on A, either operation of T1 should get completed fully or operation of T1 should be aborted fully, before starting operation of T2.
Volatility : When multiple threads are accessing single variable A, all threads should read the actual value from memory. In multithreaded environment, depending on processor, threads do optimisation while reading variable resources like A. Optimised value of A may not match actual value of A in memory. This may lead to Race condition or misbehaviour of your business logic. In your case, the Retry operation may run more than Max Retry count in case of misbehaviour.
Make both shared resources to comply Volatile and Atomic properties
final AtomicInteger retryCounter = new AtomicInteger(0);
final AtomicInteger maxRetries = new AtomicInteger(3);
In the below code about the synchronisation between threads, according to the output generated why is the control being transferred to the execution of the new thread despite the lock being acquired for the same object "dt" in the main method ?
public class DemoThread extends Thread {
public DemoThread() {
}
public void run() {
int i=0;
synchronized(this) {
while(++i<=5) {
sum=i;
try{
sleep(1000);
System.out.println("Woke up from sleep");
if(i>=2) this.notify();
}catch(InterruptedException ie) {
ie.printStackTrace();
System.exit(1);
}
}
}
}
private static int sum;
public static void main(String... args) {
DemoThread dt = new DemoThread();
dt.start();
synchronized(dt) {
try{
System.out.println("main here");
dt.wait();
System.out.println("main here again");
System.out.println("sum = " + sum);
}catch(InterruptedException ie){
ie.printStackTrace();
System.exit(1);
}
}
}
}
Output :
main here
Woke up from sleep
Woke up from sleep
Woke up from sleep
Woke up from sleep
Woke up from sleep
main here again
sum = 5
EDIT: I think I was able to find one of the possible flow of the code to explain the output:
1.Main thread enters in the Sync block in the main method.
2.call to the wait is made. Lock released on the dt object
3.New thread enters the while loop as it has the lock on the object dt
4.Thread.Sleep is executed and it doesn't release the lock
5.notify call is made but doesnot wakr the main thread(?)
6.New and the main thread finish the execution
Please correct me if I am wrong
You are close :
1.Main thread enters in the Sync block in the main method.
2.call to the wait is made. Lock released on the dt object
3.New thread enters the while loop as it has the lock on the object dt
4.Thread.Sleep is executed and it doesn't release the lock
5.notify call is made but doesnot wake the main thread(?)
6.New and the main thread finish the execution
Until the step 4, it is correct.
Here is what it happens at the step 5 :
notify() is invoked and the main() thread is so notified.
But it will not have a chance to run again right now.
Why ? Because the DemoThread thread doesn't release the lock.
The notify() method is indeed executed in a loop inside a synchronized statement.
synchronized (this) {
while (++i <= 5) {
sum = i;
try {
sleep(1000);
System.out.println("Woke up from sleep");
if (i >= 2) {
notify();
}
} catch (InterruptedException ie) {
ie.printStackTrace();
System.exit(1);
}
}
And according to Object.notify() javadoc :
The awakened thread will not be able to proceed until the current
thread relinquishes the lock on this object. The awakened thread will
compete in the usual manner with any other threads that might be
actively competing to synchronize on this object; for example, the
awakened thread enjoys no reliable privilege or disadvantage in being
the next thread to lock this object.
So the main() thread could run only as the run() method DemoThread is terminated.
To let the main() thread to run again, you could reverse in the DemonThread run() method, the synchronized statement and the while statement.
You should also make this thread sleep a little bit to let the main() thread to run again.
public void run() {
int i = 0;
while (++i <= 5) {
// let a chance for other threads
try {
Thread.sleep(100);
} catch (InterruptedException e) {
e.printStackTrace();
}
synchronized (this) {
sum = i;
try {
sleep(1000);
System.out.println("Woke up from sleep");
if (i >= 2) {
notify();
}
} catch (InterruptedException ie) {
ie.printStackTrace();
System.exit(1);
}
}
}
}
Now as i >= 2, as previously, other threads are notified but as the thread leaves the lock as it loops on the while and then sleep 100 ms, the main() thread can run again.
Here is the output :
main here
Woke up from sleep
Woke up from sleep
main here again
sum = 2
Woke up from sleep
Woke up from sleep
Woke up from sleep
The synchronized keyword isn't used to control the execution of a thread, it's used to ensure that only one thread can enter a block of code at any one time.
Typically whole methods can be synchronized or code between {}.
You can also synchronize an object that will be shared between two or more threads, typically some data structure that will be updated by the threads and you need to ensure that the state is consistent and not partially updated.
In your example there is no contention on the synchronize, if you extedt the sample to introduce some object and multiple threads trying to write and read from this object you will get a better understanding.
How does Executor.newSingleThreadExecutor() behave if I am frequently scheduling tasks to run that are being cancelled with future.cancel(true);?
Does the single thread spawned by the executor get interrupted (so the future code needs to clear the interrupt), or does the interrupt flag get automatically cleared when the next future starts up.
Does the Executor need to spawn an additional thread on every interrupt to be used by the remaining task queue?
Is there a better way?
Good question, I don't find this documented anywhere, so I would say it is implementation dependent.
For example OpenJDK does reset the interrupted flag before every executed task:
// If pool is stopping, ensure thread is interrupted;
// if not, ensure thread is not interrupted. This
// requires a recheck in second case to deal with
// shutdownNow race while clearing interrupt
if ((runStateAtLeast(ctl.get(), STOP) ||
(Thread.interrupted() &&
runStateAtLeast(ctl.get(), STOP))) &&
!wt.isInterrupted())
wt.interrupt();
Snippet from from OpenJDK jdk8u ThreadPoolExecutor#runWorker source.
The following sample program demonstrates that the interrupt is called on the thread if you call the cancel method with true. You can even see that it is reusing the same thread. The cancel returns a boolean which indicates if the cancellation was successful. The javadoc of this method is also clear enough.
class Task implements Callable<String> {
#Override
public String call() throws Exception {
try {
System.out.println("Thread name = " + Thread.currentThread().getName());
Thread.sleep(Integer.MAX_VALUE);
} catch (InterruptedException e) {
System.out.println("Interrupted");
return "Interruped";
}
return "X";
}
}
public class Testy {
public static void main(String[] args) throws InterruptedException {
ExecutorService executorService =
Executors.newSingleThreadExecutor();
int count = 0;
while (true) {
System.out.println("Iteration " + count++);
Future<String> submit = executorService.submit(new Task());
Thread.sleep(500);
submit.cancel(true);
}
}
}
Output looks like below
Iteration 0
Thread name = pool-1-thread-1
Iteration 1
Interrupted
Thread name = pool-1-thread-1
Iteration 2
Interrupted
This question already has answers here:
Execution order of multiple threads
(4 answers)
Closed 6 years ago.
I have 3 thread which i would like it to print in order but when I run the program it's keep getting result . I don't understand how it couldn't run thread in order. I would like to continue run thread 1 and 2 and 3 respectively. In each thread there is a loop for printing it's multiple times. So I would like to make the main thread to run each thread in order. This is my code.
threadMessage("Starting MessageLoop thread");
long patience =
long startTime = System.currentTimeMillis();
Thread t = new Thread(new MessageLoop());
Thread t2 = new Thread(new MessageLoop2());
Thread t3 = new Thread(new MessageLoop3());
t.setPriority(10);
t2.setPriority(5);
t3.setPriority(1);
t.start();
t2.start();
t3.start();
This is my thread function(3 threads)
private static class MessageLoop
implements Runnable {
public void run() {
try {
for(int i = 0;i<20;i++)
{
Thread.sleep(1000);
// Print a message
threadMessage("A");
}
} catch (InterruptedException e) {
threadMessage("thread interrupted");
}
}
}
private static class MessageLoop2
implements Runnable {
public void run() {
try {
for(int i = 0;i<20;i++)
{ Thread.sleep(1000);
// Print a message
threadMessage("B");
}
} catch (InterruptedException e) {
threadMessage("thread interrupted");
}
}
private static class MessageLoop3
implements Runnable {
public void run() {
String importantInfo = "E";
try {
for(int i = 0;i<20;i++)
{
Thread.sleep(1000);
// Print a message
threadMessage(importantInfo);
}
} catch (InterruptedException e) {
threadMessage("Thread interrupted");
}
}
And this is my code to make it run in order. I want to make my program run in order like this MessageLoop1 and 2 and 3 respectively.
while (t.isAlive()) {
threadMessage("Still waiting...");
t.join(2000);
if (((System.currentTimeMillis() - startTime) > patience)
&& t.isAlive()) {
threadMessage("Tired of waiting!");
t.interrupt();
// Shouldn't be long now
// -- wait indefinitely
t.join();
}
while(t2.isAlive()){
threadMessage("Still waiting...");
t2.join(1000);
if (((System.currentTimeMillis() - startTime) > patience)
&& t2.isAlive()) {
threadMessage("Tired of waiting!");
t2.interrupt();
// Shouldn't be long now
// -- wait indefinitely
t2.join();
}
}
while(t3.isAlive()){
threadMessage("Still waiting...");
t3.join(1000);
if (((System.currentTimeMillis() - startTime) > patience)
&& t3.isAlive()) {
threadMessage("Tired of waiting!");
t3.interrupt();
// Shouldn't be long now
// -- wait indefinitely
t3.join();
}
}
}
But the result is coming like B,A,C. Can anyone explain this situation? And are my code wrong? Thank you!
That's how threads work. You don't get a guarantee at all which thread will finish first - and that's by design.
I assume what you want, is actually what the jdk calls a future and an ExecutorService.
(pseudocode - will have syntax errors)
ExecutorService s = Executors.newCachedThreadPool();
try {
Future f1 = s.submit(new MessageLoop());
Future f2 = s.submit(new MessageLoop2());
Future f3 = s.submit(new MessageLoop3());
f1.await(10, TimeUnit.SECONDS); // waits for the first thread to finish
// first thread finished now
f2.await(10, TimeUnit.SECONDS);
// second thread finished now
// ...
} finally { s.shutdown(); }
very important is to manage the proper shutdown of the ExecutorService, as the executor service will manage a couple of threads that run until you terminate them. if you don't shut it down, then your applicationo will not terminate.
What makes you assume you are controlling order?
The individual MessageLoop implementations are not blocked from executing in any way. So they just will run at the descretion of the thread scheduling.
You would need to introduce a shared ressource that takes the role of lock between the control thread (trying to enforce the order) and the worker threads.
In your current code the control thread just applies a special sequence on collecting the termination of the workers. That may have been executed and completed earlier in time.
If you are interested in a sequential execution and do not want to execute the tasks inline (same thread as your control), then you might just execute the threads in sequence to achieve your goal of sequential execution. (start each thread and wait for termination before starting another).
As you seam to have a restriction on the order of execution you would need some semaphore to coordinate such execution.
I just started out with threading. I wrote a main class that sets up and starts 100 threads, waits 5 seconds and then interrupts them (at least that's what I thought it did):
public static void main(String[] args) {
List<Thread> threads = new ArrayList<Thread>();
for (int i = 0; i < 100; i++) {
Thread t = new Thread(new Walker());
threads.add(t);
}
System.out.println("Starting threads...");
for (Thread thread : threads) {
thread.start();
}
try {
Thread.sleep(5000);
} catch (InterruptedException e) {
// don't do anything
}
System.out.println("Time's up - Terminating threads...");
for (Thread t : threads) {
t.interrupt();
System.out.print(".");
}
for (Thread t : threads) {
try {
t.join(10);
} catch (InterruptedException e) {
// don't do anything
}
}
System.out.println("");
System.out.println("All done.");
}
The threads looked a bit like this:
public class Walker implements Runnable {
public void run() {
for (int i = 0;; i++) {
//do some complicated stuff that takes some time
System.out.println(Thread.currentThread().getName() + ":" + i);
if (Thread.interrupted()) {
break;
}
}
}
}
Now, the output I got was that the main thread began interrupting threads, but some sub threads continued to run a few times (i.e. loop iterations) before terminating, e.g.
Starting threads...
Thread-1:0
Thread-2:0
Thread-1:1
Thread-3:0
[...]
Time's up - Terminating threads...
......Thread-1:60
Thread-1:61
...Thread-1:62
Thread-2:55
..All done.
[output from threads sometimes continued even here - after the join()]
At that time I didn't fully understand that a single thread could be allocated enough processor time to run a few times - I expected at most one additional run before the main thread had the opportunity to interrupt it.
But while I now see that it is absolutely fine for a thread to be executed for some (long) time before the main thread gets a chance to terminate (i.e. interrupt) it, I am still wondering: is there an easy way to interrupt all child threads in a timely manner from the main thread? (Setting a "time to live" through a thread's constructor and then testing inside the Walker class for it is not what I want.)
Also: is it possible for the last print statement to execute and then see some output from individual threads - after all threads were join()ed? (Maybe I have a glitch somewhere else; the actual code is a bit more complex...)
The problem you observe is probably due to how System.out.println works. It is a synchronized method. So a likely explanation is:
when calling System.out.print("."); after t.interrupt();, your main thread acquires the lock to print
before the lock is released, worker threads arrive at System.out.println(Thread.currentThread().getName() + ":" + i); and wait for the lock
when the main thread releases the lock, all the worker threads that were waiting print their progress.
the main thread arrives at System.out.print("."); again and has to wait for the print lock to be available, etc.
Regarding the fact that you see more prints from the worker threads after "All Done" is printed: you only join for 10 ms, so it is possible that it is not enough and a thread is not finished within 10ms of being interrupted. If you just use join() you should not see that any longer.
Example of Worker class that reproduces the behaviour you observe:
class Walker implements Runnable {
public void run() {
for (int i = 0;; i++) {
try {
Thread.sleep(100);
} catch (InterruptedException ex) {
//do not respond to interruption too quickly on purpose
try {
Thread.sleep(1000);
} catch (InterruptedException e) {}
Thread.currentThread().interrupt();
}
System.out.println(Thread.currentThread().getName() + ":" + i);
if (Thread.interrupted()) {
break;
}
}
}
}
It would be easier with ExecutorService, eg
int nThreads = 100;
ExecutorService ex = Executors.newFixedThreadPool(nThreads);
for (int i = 0; i < nThreads; i++) {
ex.execute(new Walker());
}
Thread.sleep(5000);
ex.shutdownNow();
Maybe I have a glitch somewhere else; the actual code is a bit more complex...
Yes it is a glitch, unfortunately isn't a simple set 1 property, java side.
If the code is commercial, complex, than you can allocate a bit more time to write some native libraries, for major Os type. With that help you can easily play with threads as you wanted.
The first times has an overhead for developing and understanding how the threads are woking in native, os side, than just call a function with a few params :)
Not sure, if is helping, the glitch exists.