We Keep Coding

Executorservice wait after tasks are finished

I have this code and what it does is wait for all tasks are complished and only then return values to the widget. WorkerThread is a runnable that needs to finish before next loop.
final ScheduledExecutorService ecs = Executors.newScheduledThreadPool(size2/2);
while(size2>1) {
for (int i = 0; i < size2/2; i++) {
Runnable worker = null;
try {
worker = new WorkerThread(players.take(), players.take() ,area,players);
} catch (InterruptedException interruptedException) {
interruptedException.printStackTrace();
}
ecs.submit(worker);
}
area.append("\n\n next Round");
size2=size2/2;
}
So what I need to do is to have in this case 8 tasks complete, get values that they append to the widget, then wait for example 2 seconds and proceed with the loop now with 4 tasks.

You can use Phaser.
Phaser phaser = new Phaser(1); // Create Phaser instance
final ScheduledExecutorService ecs = Executors.newScheduledThreadPool(size2/2);
while(size2>1) {
for (int i = 0; i < size2/2; i++) {
Runnable worker = null;
try {
worker = new WorkerThread(players.take(), players.take() ,area,players);
} catch (InterruptedException interruptedException) {
interruptedException.printStackTrace();
}
phaser.register(); // Register a party
ecs.submit(worker);
}
phaser.arriveAndAwaitAdvance(); // Wait for completion of all tasks
area.append("\n\n next Round");
size2=size2/2;
}
Here is your consumer.
public class WorkerThread {
public WorkerThread(Phaser phaser) {
this.phaser = phaser;
}
public void run() {
processThings();
phaser.arriveAndDeregister(); // Deregister a party
}
}

How to control ParallelSuite thread count?

I have following code:
#RunWith(ParallelSuite.class)
#Suite.SuiteClasses({Test1.class,
Test2.class,
Test3.class,
Test4.class,
Test5.class,
Test6.class,
Test7.class})
public class ParallelRunner {
}
I ran it and figured out that only 3 test run in parallel. These test anough long - 15+ seconds
Is it OS scheduler issue or junit thread pool limit?
How can I configure thread pool limit ?

Inside ParallelComputer you can find following method:
private static Runner parallelize(Runner runner) {
if (runner instanceof ParentRunner) {
((ParentRunner)runner).setScheduler(new RunnerScheduler() {
private final ExecutorService fService = Executors.newCachedThreadPool();
public void schedule(Runnable childStatement) {
this.fService.submit(childStatement);
}
public void finished() {
try {
this.fService.shutdown();
this.fService.awaitTermination(9223372036854775807L, TimeUnit.NANOSECONDS);
} catch (InterruptedException var2) {
var2.printStackTrace(System.err);
}
}
});
}
return runner;
}
we need to replace this:
private final ExecutorService fService = Executors.newCachedThreadPool();
I just copy pasted ParallelComputer class code and replaced this line with:
private final ExecutorService fService = Executors.newFixedThreadPool(MAX_THREAD_COUNT);

It's backed by a ForkedThreadPool which is work stealing.
static ForkJoinPool setUpForkJoinPool() {
int numThreads;
try {
String configuredNumThreads = System.getProperty("maxParallelTestThreads");
numThreads = Math.max(2, Integer.parseInt(configuredNumThreads));
} catch (Exception ignored) {
Runtime runtime = Runtime.getRuntime();
numThreads = Math.max(2, runtime.availableProcessors());
}
ForkJoinPool.ForkJoinWorkerThreadFactory threadFactory = pool -> {
if (pool.getPoolSize() >= pool.getParallelism()) {
return null;
} else {
ForkJoinWorkerThread thread = ForkJoinPool.defaultForkJoinWorkerThreadFactory.newThread(pool);
thread.setName("JUnit-" + thread.getName());
return thread;
}
};
return new ForkJoinPool(numThreads, threadFactory, null, false);
}
Effectively can set maxParallelTestThreads, else it will back to max of the number of processors, or two threads min.

Tracking Executing Threads

I am trying to figure out how I can track all the threads that my application is spawning. Initially, I thought I had it figured out using a CyclicBarrier, however I am seeing threads executing after my await call.
Below is the working pseudo code:
public class ThreadTesterRunner {
public static void main(String[] args) throws InterruptedException {
final CyclicBarrier cb = new CyclicBarrier(1);
ThreadRunner tr = new ThreadRunner(cb);
Thread t = new Thread(tr, "Thread Runner");
t.start();
boolean process = true;
// wait until all threads process, then print reports
while (process){
if(tr.getIsFinished()){
System.out.println("Print metrics");
process = false;
}
Thread.sleep(1000);
}
}
}
class ThreadRunner implements Runnable {
static int timeOutTime = 2;
private ExecutorService executorService = Executors.newFixedThreadPool(10);
private final CyclicBarrier barrier;
private boolean isFinished=false;
public ThreadRunner(CyclicBarrier cb) {
this.barrier = cb;
}
public void run(){
try {
boolean stillLoop = true; int i = 0;
while (stillLoop){
int size;
Future<Integer> future = null;
try {
future = executorService.submit(new Reader()); // sleeps
size = future.get();
} catch (InterruptedException | ExecutionException ex) {
// handle Errs
}
if(i == 3){
stillLoop = false;
this.barrier.await();
this.isFinished=true;
}
//System.out.println("i = "+i+" Size is: "+size+"\r");
i++;
}
} catch (InterruptedException | BrokenBarrierException e1) {
e1.printStackTrace();
}
}
public boolean getIsFinished(){
return this.isFinished;
}
}
class Reader implements Callable {
private ExecutorService executorService = Executors.newFixedThreadPool(1);
#Override
public Object call() throws Exception {
System.out.println("Reading...");
Thread.sleep(2000);
executorService.submit(new Writer());
return 1000;
}
}
class Writer implements Callable {
#Override
public Void call() throws Exception {
Thread.sleep(4000);
System.out.println("Wrote");
return null;
}
}
Can anyone suggest a way to ONLY print "print metrics" after all threads have run?

It doesn't seem like you're doing anything to coordinate with your Reader and Writer threads, which are the ones you want to wait for. If you pass your synchronization barrier through to those threads so that they can register and signal when they are done, it works just fine.
Here's a version rewritten to do so, using a Phaser instead of a CyclicBarrier. Note that each Reader and Writer registers itself upon construction, and notifies the synchronization barrier when it is done executing:
public class ThreadTesterRunner {
public static void main(String[] args) throws InterruptedException {
final Phaser cb = new Phaser();
ThreadRunner tr = new ThreadRunner(cb);
Thread t = new Thread(tr, "Thread Runner");
t.start();
boolean process = true;
// wait until all threads process, then print reports
while (process){
if(tr.getIsFinished()){
System.out.println("Print metrics");
process = false;
}
//else {
// System.out.println("Waiting: registered=" + cb.getRegisteredParties() + ", arrived=" + cb.getArrivedParties() + ", unarrived=" + cb.getUnarrivedParties());
//}
Thread.sleep(1000);
}
}
}
class ThreadRunner implements Runnable {
static int timeOutTime = 2;
private ExecutorService executorService = Executors.newFixedThreadPool(10);
private final Phaser barrier;
private boolean isFinished=false;
public ThreadRunner(Phaser phaser) {
this.barrier = phaser;
}
public void run(){
try {
boolean stillLoop = true; int i = 0;
while (stillLoop){
int size;
Future<Integer> future = null;
try {
future = executorService.submit(new Reader(this.barrier)); // sleeps
size = future.get();
} catch (InterruptedException | ExecutionException ex) {
// handle Errs
}
if(i == 3){
stillLoop = false;
this.barrier.awaitAdvance(0);
this.isFinished=true;
}
//System.out.println("i = "+i+" Size is: "+size+"\r");
i++;
}
} catch (Exception e1) {
e1.printStackTrace();
}
}
public boolean getIsFinished(){
return this.isFinished;
}
}
class Reader implements Callable {
private Phaser barrier;
private ExecutorService executorService = Executors.newFixedThreadPool(1);
public Reader(Phaser phase) {
phase.register();
this.barrier = phase;
}
#Override
public Object call() throws Exception {
System.out.println("Reading...");
Thread.sleep(2000);
executorService.submit(new Writer(this.barrier));
this.barrier.arrive();
return 1000;
}
}
class Writer implements Callable {
private Phaser barrier;
public Writer(Phaser phase) {
phase.register();
this.barrier = phase;
}
#Override
public Void call() throws Exception {
Thread.sleep(4000);
System.out.println("Wrote");
this.barrier.arrive();
return null;
}
}

From what I can see you aren't waiting for the Writer to finish in the Reader. Is that the problem you are seeing?
You are also accessing isFinished from more than one thread without synchronization (which however, merely may delay the termination of the loop in this situation).
I don't see CyclicBarrier doing anything.
Not sure what you are trying to do, but I'd think about how simpler I can make it. For example, can Reader and Writer be combined into one task? Then, waiting for them to finish would merely be:
executorService.invokeAll(tasks);
System.out.println("Print metrics");
where tasks is a collection of tasks (see also this javadoc)

Text is not getting printed once the Threads are done [duplicate]

This question already has answers here:
How to wait for all threads to finish, using ExecutorService?
(27 answers)
Closed 8 years ago.
Please have a look at the following code.
public class BigFileWholeProcessor {
private static final int NUMBER_OF_THREADS = 2;
public void processFile(String fileName) {
BlockingQueue<String> fileContent = new LinkedBlockingQueue<String>();
BigFileReader bigFileReader = new BigFileReader(fileName, fileContent);
BigFileProcessor bigFileProcessor = new BigFileProcessor(fileContent);
ExecutorService es = Executors.newFixedThreadPool(NUMBER_OF_THREADS);
es.execute(bigFileReader);
es.execute(bigFileProcessor);
es.shutdown();
if(es.isTerminated())
{
System.out.println("Completed Work");
}
}
}
public class BigFileReader implements Runnable {
private final String fileName;
int a = 0;
public static final String SENTINEL = "SENTINEL";
private final BlockingQueue<String> linesRead;
public BigFileReader(String fileName, BlockingQueue<String> linesRead) {
this.fileName = fileName;
this.linesRead = linesRead;
}
#Override
public void run() {
try {
//since it is a sample, I avoid the manage of how many lines you have read
//and that stuff, but it should not be complicated to accomplish
BufferedReader br = new BufferedReader(new FileReader(new File("E:/Amazon HashFile/Hash.txt")));
String str = "";
while((str=br.readLine())!=null)
{
linesRead.put(str);
System.out.println(a);
a++;
}
linesRead.put(SENTINEL);
} catch (Exception ex) {
ex.printStackTrace();
}
System.out.println("Completed");
}
}
public class BigFileProcessor implements Runnable {
private final BlockingQueue<String> linesToProcess;
public BigFileProcessor (BlockingQueue<String> linesToProcess) {
this.linesToProcess = linesToProcess;
}
#Override
public void run() {
String line = "";
try {
while ( (line = linesToProcess.take()) != null) {
//do what you want/need to process this line...
if(line==BigFileReader.SENTINEL)
{
break;
}
String [] pieces = line.split("(...)/g");
}
} catch (InterruptedException e) {
e.printStackTrace();
}
}
}
I want to print the text "completed work" in BigFileWholeProcessor once all the thread work is done. But instead, it is not getting printed. Why is this? How to identify that all the threads are done and need printing?

shutdown() only signal ES to shutdown, you need
awaitTermination(long timeout, TimeUnit unit)
before print message

Use submit() method instead of execute(). The get() method can be used if you want to wait for the thread to finish at any point of time. Read documentation on use of Future object for further details.
ExecutorService es = Executors.newFixedThreadPool(2);
Future<?> f = es.submit(new Thread(new TestRun()));
f.get(); // Wait for result... (i.e similar to `join()` in this case)
es.shutdown(); // Shutdown ExecutorService
System.out.println("Done.");
I have defined a TestRun class implementing Runnable, not shown here. The Future object makes more sense in other scenarios.

How do we know threadPoolExecutor has finished execution

I have a parent thread that sends messages to MQ and it manages a ThreadPoolExecutor for worker threads which listen to MQ and writes message to output file. I manage a threadpool of size 5. So when I run my program, I have 5 files with messages. Everything works fine until here. I now need to merge these 5 files in my parent thread.
How do I know ThreadPoolExecutor finished processing so I can start merging files.
public class ParentThread {
private MessageSender messageSender;
private MessageReciever messageReciever;
private Queue jmsQueue;
private Queue jmsReplyQueue;
ExecutorService exec = Executors.newFixedThreadPool(5);
public void sendMessages() {
System.out.println("Sending");
File xmlFile = new File("c:/filename.txt");
List<String> lines = null;
try {
lines = FileUtils.readLines(xmlFile, null);
} catch (IOException e) {
e.printStackTrace();
}
for (String line : lines){
messageSender.sendMessage(line, this.jmsQueue, this.jmsReplyQueue);
}
int count = 0;
while (count < 5) {
messageSender.sendMessage("STOP", this.jmsQueue, this.jmsReplyQueue);
count++;
}
}
public void listenMessages() {
long finishDate = new Date().getTime();
for (int i = 0; i < 5; i++) {
Worker worker = new Worker(i, this.messageReciever, this.jmsReplyQueue);
exec.execute(worker);
}
exec.shutdown();
if(exec.isTerminated()){ //PROBLEM is HERE. Control Never gets here.
long currenttime = new Date().getTime() - finishDate;
System.out.println("time taken: "+currenttime);
mergeFiles();
}
}
}
This is my worker class
public class Worker implements Runnable {
private boolean stop = false;
private MessageReciever messageReciever;
private Queue jmsReplyQueue;
private int processId;
private int count = 0;
private String message;
private File outputFile;
private FileWriter outputFileWriter;
public Worker(int processId, MessageReciever messageReciever,
Queue jmsReplyQueue) {
this.processId = processId;
this.messageReciever = messageReciever;
this.jmsReplyQueue = jmsReplyQueue;
}
public void run() {
openOutputFile();
listenMessages();
}
private void listenMessages() {
while (!stop) {
String message = messageReciever.receiveMessage(null,this.jmsReplyQueue);
count++;
String s = "message: " + message + " Recieved by: "
+ processId + " Total recieved: " + count;
System.out.println(s);
writeOutputFile(s);
if (StringUtils.isNotEmpty(message) && message.equals("STOP")) {
stop = true;
}
}
}
private void openOutputFile() {
try {
outputFile = new File("C:/mahi/Test", "file." + processId);
outputFileWriter = new FileWriter(outputFile);
} catch (IOException e) {
System.out.println("Exception while opening file");
stop = true;
}
}
private void writeOutputFile(String message) {
try {
outputFileWriter.write(message);
outputFileWriter.flush();
} catch (IOException e) {
System.out.println("Exception while writing to file");
stop = true;
}
}
}
How will I know when the ThreadPool has finished processing so I can do my other clean up work?
Thanks

If you Worker class implements Callable instead of Runnable, then you'd be able to see when your threads complete by using a Future object to see if the Thread has returned some result (e.g. boolean which would tell you whether it has finished execution or not).
Take a look in section "8. Futures and Callables" # website below, it has exactly what you need imo:
http://www.vogella.com/articles/JavaConcurrency/article.html
Edit: So after all of the Futures indicate that their respective Callable's execution is complete, its safe to assume your executor has finished execution and can be shutdown/terminated manually.

Something like this:
exec.shutdown();
// waiting for executors to finish their jobs
while (!exec.awaitTermination(50, TimeUnit.MILLISECONDS));
// perform clean up work

You can use a thread for monitoring ThreadPoolExecutor like that
import java.util.concurrent.ThreadPoolExecutor;
public class MyMonitorThread implements Runnable {
private ThreadPoolExecutor executor;
private int seconds;
private boolean run=true;
public MyMonitorThread(ThreadPoolExecutor executor, int delay)
{
this.executor = executor;
this.seconds=delay;
}
public void shutdown(){
this.run=false;
}
#Override
public void run()
{
while(run){
System.out.println(
String.format("[monitor] [%d/%d] Active: %d, Completed: %d, Task: %d, isShutdown: %s, isTerminated: %s",
this.executor.getPoolSize(),
this.executor.getCorePoolSize(),
this.executor.getActiveCount(),
this.executor.getCompletedTaskCount(),
this.executor.getTaskCount(),
this.executor.isShutdown(),
this.executor.isTerminated()));
try {
Thread.sleep(seconds*1000);
} catch (InterruptedException e) {
e.printStackTrace();
}
}
}
}
And add
MyMonitorThread monitor = new MyMonitorThread(executorPool, 3);
Thread monitorThread = new Thread(monitor);
monitorThread.start();
to your class where ThreadPoolExecutor is located.
It will show your threadpoolexecutors states in every 3 seconds.