We Keep Coding

How to wait until for few callable tasks complete and then execute

I want to execute some processors in two threads.. few of them are independent and could run anytime but few of them have dependency.. whenver order of execution reaches to that processor i want to check whether all previous callable tasks are executed or not? and future should be executed once this current execute.
following is the main thread method
PackageExportGraph executeMultiThread(PackageExportGraph exportGraphInp, PackageExportContext exportContextnInp)
throws WTException {
Map<PackageExportDependencyProcessor, Boolean> processorToParallelExecutionMap = new LinkedHashMap<>();
this.processorQueue = new LinkedBlockingQueue<>();
ExecutorService execService = null;
try {
int threads = 2;// 2
countDownLatch = new CountDownLatch(threads);
execService = ExecutorServiceFactory.getDefault().newExecutorService(threads, true);
boolean isThread1Started = false;
ThreadedDepProcessor thread1 = new ThreadedDepProcessor(
exportGraphInp, countDownLatch,
processorToParallelExecutionMap, processorQueue, exportContextnInp, isThread1Started);
threadList.add(thread1);
thread1.addListener(this);
boolean isThread2Started = false;
ThreadedDepProcessor thread2 = new ThreadedDepProcessor(
exportGraphInp, countDownLatch,
processorToParallelExecutionMap, processorQueue, exportContextnInp, isThread2Started);
threadList.add(thread2);
thread1.addListener(this);
List<Future<LinkedBlockingQueue>> futureList = new ArrayList<>();
for (ThreadedDepProcessor thread : threadList) {
Future f = execService.submit(thread);
System.out.println("f " + f);
futureList.add(f);
}
int currentidx = 0;
for (PackageExportDependencyProcessor processor : origOrderedList) {
if (!processorToParallelExecutionMap.containsKey(processor)) {
System.out.println(" parallel threadStatusMap values 1 - " + threadStatusMap.values());
System.out.println("Adding parallel - " + processor);
if (currentidx > 0) {
while (threadStatusMap.containsValue(false)) {
System.out.println("Waiting");
System.out.println("threadStatusMap values - " + threadStatusMap.values());
Thread.sleep(1000);
}
Thread.sleep(2000);
// execService.awaitTermination(5, TimeUnit.SECONDS);
System.out.println("Size - " + futureList.size());
for (Future f : futureList) {
System.out.println("futureList is done " + f.isDone());
System.out.println("Getting future Object");
if (f.isDone()) {
continue;
}
Object o = f.get(10, TimeUnit.SECONDS);
System.out.println(o);
/*
* Object object = f.get(10, TimeUnit.SECONDS); System.out.println("Obj " + object);
*/
}
processorQueue.put(processor);
Thread.sleep(2000);
}
else {
processorQueue.put(processor);
Thread.sleep(2000);
System.out.println("Size - " + futureList.size());
for (Future f : futureList) {
System.out.println("futureList is done " + f.isDone());
System.out.println("Getting future Object");
if (f.isDone()) {
continue;
}
Object o = f.get(10, TimeUnit.SECONDS);
System.out.println(o);
/*
* Object object = f.get(10, TimeUnit.SECONDS); System.out.println("Obj " + object);
*/
}
// execService.awaitTermination(5, TimeUnit.SECONDS);
while (threadStatusMap.containsValue(false)) {
System.out.println("Waiting");
System.out.println("threadStatusMap values - " + threadStatusMap.values());
Thread.sleep(1000);
}
}
for (ThreadedDepProcessor thread : threadList) {
System.out.println("Finished adding dependents" + thread.finishedAddingDependents.get());
}
}
else {
System.out.println("Adding non-parallel - " + processor);
processorQueue.put(processor);
}
currentidx++;
}
} catch (WTException | RuntimeException exc) {
if (Objects.nonNull(execService)) {
execService.shutdown();
}
throw exc;
} catch (Exception exc) {
throw new WTException(exc);
} finally {
System.out.println("shutting down");
try {
countDownLatch.await();
} catch (InterruptedException e) {
e.printStackTrace();
}
execService.shutdown();
}
return exportGraphInp;
}
and this is callable
#Override
public LinkedBlockingQueue call() throws WTException, InterruptedException {
try {
System.out.println("Started - ");
isThreadStarted = true;
while (!processorQueue.isEmpty()) {
nextEligible = processorQueue.take();
if (Objects.isNull(nextEligible)) {
finishedAddingDependents.set(true);
break;
}
System.out.println("calling addDependentObjects for processor - " + nextEligible);
nextEligible.addDependentObjects(exportGraph, exportContext);
nextEligible = null;
// notifyListeners();
}
} catch (Exception e) {
System.out.println("Error occured " + e);
e.printStackTrace();
return processorQueue;
} finally {
countDownLatch.countDown();
System.out.println("countDownLatch now - " + countDownLatch.getCount());
}
return processorQueue;
}
}
I was trying to check while(future.isDone()) but its going indefinite loop.
i want to check whether thread/callable execution is started or not.
If started then while executing serial processor i want to wait till all existing is executed and then start execution and wait till its execution do not pick next one

What i did is manitained one synchronized collection, which will let us know status of execution of each processor, and based on that we can wait or go ahead

How do I avoid call to same method only if arguments are same, on different threads

private void startProcess(Long id)
throws UnAuthorizedException, UnAuthenticatedException, InvalidRequestException {
......
createComponent(Long id);
......
}
This method createComponent or the startProcess itself should not be called again on separate thread, if the id is same

If your method takes time and you want to block concurrent access only when the ID is the same (or any combination of parameters) then you need to store a lock for every ID (or any combination of parameters).
These locks can be stored in a Map.
If it's just an ID of type Long you can use it directly as the Map's key. If you have multiple parameters, wrap them in a custom class along with the required equals/hashcode methods.
private Map<Long, Lock> locks = new ConcurrentHashMap<>(100);
private void startProcess(Long id) {
Lock lock = locks.computeIfAbsent(id, id2 -> new ReentrantLock());
lock.lock();
try {
System.out.println(System.currentTimeMillis() + " Starting " + id + " on thread " + Thread.currentThread().getName());
try { Thread.sleep(2000); } catch (InterruptedException ex) { }
System.out.println(System.currentTimeMillis() + " Done with " + id + " on thread " + Thread.currentThread().getName());
} finally {
lock.unlock();
locks.remove(lock);
}
}
Called with:
new Thread(() -> startProcess(1L), "A").start();
new Thread(() -> startProcess(1L), "B").start();
new Thread(() -> startProcess(2L), "C").start();
this is one possible output:
1595950202869 Starting 1 on thread A
1595950202879 Starting 2 on thread C
1595950204870 Done with 1 on thread A
1595950204870 Starting 1 on thread B
1595950204879 Done with 2 on thread C
1595950206870 Done with 1 on thread B
[Edit] Sample code for custom class for parameters:
private Map<ProcessParams, Lock> locks = new ConcurrentHashMap<>(100);
private void startProcess(long id, String name) {
Lock lock = locks.computeIfAbsent(new ProcessParams(id, name), id2 -> new ReentrantLock());
lock.lock();
try {
System.out.println(System.currentTimeMillis() + " Starting " + name + " on thread " + Thread.currentThread().getName());
try { Thread.sleep(2000); } catch (InterruptedException ex) { }
System.out.println(System.currentTimeMillis() + " Done with " + name + " on thread " + Thread.currentThread().getName());
} finally {
lock.unlock();
locks.remove(lock);
}
}
public static class ProcessParams {
private final long id;
private final String name;
public ProcessParams(long id, String name) {
this.id = id;
this.name = Objects.requireNonNull(name, "name");
}
#Override
public boolean equals(Object o) {
if (this == o)
return true;
if (!(o instanceof ProcessParams))
return false;
ProcessParams other = (ProcessParams)o;
return id == other.id && name.equals(other.name);
}
#Override
public int hashCode() {
int hashCode = 0x811C9DC5;
hashCode = 0x01000193 * (hashCode ^ Long.hashCode(id));
hashCode = 0x01000193 * (hashCode ^ name.hashCode());
return hashCode;
}
}

Multiple ExecutorCompletionService not working

I've a requirement to start and stop task from java application. I'm trying to use
ExecutorService to create threads and ExecutorCompletionService to check
processing status of thread . Startup and stop is a continious activity so in my
test code I've created a while loop .
public class ProcessController {
String[] processArray = { "Process1", "Process2", "Process3", "Process4", "Process5", "Process6", "Process7" };
private List<String> processList = Arrays.asList(processArray);
public static void main(String[] args ) {
ExecutorService startUpExecutor = Executors.newFixedThreadPool(3);
ExecutorService cleanUpExecutor = Executors.newFixedThreadPool(3);
CompletionService<String> startUpCompletionService = new ExecutorCompletionService<>(startUpExecutor);
CompletionService<String> cleanUpCompletionService = new ExecutorCompletionService<>(cleanUpExecutor);
List<Future<String>> cleanupFutures = new ArrayList<Future<String>>();
List<Future<String>> startupFutures = new ArrayList<Future<String>>();
ProcessController myApp = new ProcessController();
int i = 0;
while (i++ < 3) {
System.out.println("**********Starting Iteration " + i + "************* =====> ");
if (!cleanupFutures.isEmpty()) cleanupFutures.clear();
myApp.processList.forEach(process -> cleanupFutures.add(cleanUpCompletionService.submit(new CleanupTask(process))));
myApp.processList.forEach(process -> startupFutures.add(startUpCompletionService.submit(new StartupTask(process))));
for (Future<String> f : cleanupFutures) {
try {
String result = cleanUpCompletionService.take().get();
System.out.println("Result from Cleanup thread : " + result);
} catch (InterruptedException e) {
e.printStackTrace();
} catch (ExecutionException e) {
e.printStackTrace();
}
}
for (Future<String> f1 : startupFutures) {
try {
String result = startUpCompletionService.take().get();
System.out.println("Result from startup thread : " + result);
} catch (InterruptedException e) {
e.printStackTrace();
} catch (ExecutionException e) {
e.printStackTrace();
}
}
System.out.println("**********Finished Iteration " + i + "************* =====> ");
}
startUpExecutor.shutdown();
cleanUpExecutor.shutdown();
}
}
CleanupTask class
public class CleanupTask implements Callable<String> {
private String task;
public CleanupTask(String task) {
this.task = task;
}
#Override
public String call() throws Exception {
checkIfAnyFinished();
return "finished clean up processing for " + getThreadId();
}
private void checkIfAnyFinished( )
{
System.out.println( getThreadId() + " Checking if task " + this.task + " is finished");
try {
isFinished();
} catch (InterruptedException e) {
e.printStackTrace();
}
}
private void isFinished() throws InterruptedException {
Thread.sleep(1000*4);
}
private String getThreadId()
{
return Thread.currentThread().getName();
}
}
Startup Task class
public class StartupTask implements Callable<String> {
private String processSchedule ;
public StartupTask(String processSchedule) {
this.processSchedule = processSchedule;
}
#Override
public String call() {
scheduleifdue();
return "finished start up up processing for " + getThreadId();
}
private void scheduleifdue()
{
System.out.println(getThreadId() + " Checking " + this.processSchedule + " is due or not");
try {
Thread.sleep(4000);
} catch (InterruptedException e) {
e.printStackTrace();
}
}
private String getThreadId()
{
return Thread.currentThread().getName();
}
}
Above code successfully complete iteration 1 and start 2nd iteration . But it never finish and keeps running.
When I run the same application only with one task( either cleanup or startup) then it run without any issues. I'm not sure what is causing issue.

Stop ScheduledThreadPoolExecutor from innerTask

So I have a Thread wherein ScheduledThreadPoolExecutor is created with periodic Task, so I want to stop my ScheduledThreadPoolExecutor from Task when condition occurs.
After that from Thread wherein ScheduledThreadPoolExecutor is existed to make a notify to another Thread. Perhaps I did something wrong, I cannot to send notify from InnerThread to parent Thread Buyer. After that from Buyer sending another notify to MasterContainer.
How can I do this?
import java.util.Date;
import java.util.concurrent.*;
public class Buyer implements Runnable {
private CommonObj cmnObj;
public Buyer(CommonObj msg) {
this.cmnObj = cmnObj;
}
#Override
public void run() {
String name = Thread.currentThread().getName();
System.out.println(name + " is starting");
ScheduledThreadPoolExecutor sch = (ScheduledThreadPoolExecutor)
Executors.newScheduledThreadPool(1);
sch.setRemoveOnCancelPolicy(true);
FutureRunnable periodicTask = new FutureRunnable() {
#Override
public void run() {
try {
System.out.println("\t periodicTask Execution Time: "
+ ScheduledExample.fmt.format(new Date()));
try {
Thread.sleep(2 * 1000);
synchronized (this) {
System.out.println("\t periodicTask need to close: "
+ ScheduledExample.fmt.format(new Date()));
this.getFuture().cancel(true);
System.out.println("\t periodicTask cancelled: "
+ ScheduledExample.fmt.format(new Date()));
this.notify();
return;
}
} catch (InterruptedException e) {
e.printStackTrace();
}
System.out.println("\t periodicTask End Time: "
+ ScheduledExample.fmt.format(new Date()));
} catch (Exception e) {
}
}
};
Future<?> periodicFuture = sch.scheduleAtFixedRate(periodicTask, 3, 3, TimeUnit.SECONDS);
periodicTask.setFuture(periodicFuture);
synchronized (sch) {
try {
System.out.println(name + " is before wait");
sch.wait();
} catch (InterruptedException e) {
e.printStackTrace();
}
System.out.println(name + " is before notify");
this.notify();
}
System.out.println(name + " is ended");
}
}
abstract class FutureRunnable implements Runnable {
private Future<?> future;
public Future<?> getFuture() {
return future;
}
public void setFuture(Future<?> future) {
this.future = future;
}
}

In your code, your inner task syncronized on periodicTask and outer syncronized on sch, this does not work.
If you want to syncronize inner and outer thread, you should syncronize on the same object, as well as call wait and notify on the same object.

How do we know threadPoolExecutor has finished execution

I have a parent thread that sends messages to MQ and it manages a ThreadPoolExecutor for worker threads which listen to MQ and writes message to output file. I manage a threadpool of size 5. So when I run my program, I have 5 files with messages. Everything works fine until here. I now need to merge these 5 files in my parent thread.
How do I know ThreadPoolExecutor finished processing so I can start merging files.
public class ParentThread {
private MessageSender messageSender;
private MessageReciever messageReciever;
private Queue jmsQueue;
private Queue jmsReplyQueue;
ExecutorService exec = Executors.newFixedThreadPool(5);
public void sendMessages() {
System.out.println("Sending");
File xmlFile = new File("c:/filename.txt");
List<String> lines = null;
try {
lines = FileUtils.readLines(xmlFile, null);
} catch (IOException e) {
e.printStackTrace();
}
for (String line : lines){
messageSender.sendMessage(line, this.jmsQueue, this.jmsReplyQueue);
}
int count = 0;
while (count < 5) {
messageSender.sendMessage("STOP", this.jmsQueue, this.jmsReplyQueue);
count++;
}
}
public void listenMessages() {
long finishDate = new Date().getTime();
for (int i = 0; i < 5; i++) {
Worker worker = new Worker(i, this.messageReciever, this.jmsReplyQueue);
exec.execute(worker);
}
exec.shutdown();
if(exec.isTerminated()){ //PROBLEM is HERE. Control Never gets here.
long currenttime = new Date().getTime() - finishDate;
System.out.println("time taken: "+currenttime);
mergeFiles();
}
}
}
This is my worker class
public class Worker implements Runnable {
private boolean stop = false;
private MessageReciever messageReciever;
private Queue jmsReplyQueue;
private int processId;
private int count = 0;
private String message;
private File outputFile;
private FileWriter outputFileWriter;
public Worker(int processId, MessageReciever messageReciever,
Queue jmsReplyQueue) {
this.processId = processId;
this.messageReciever = messageReciever;
this.jmsReplyQueue = jmsReplyQueue;
}
public void run() {
openOutputFile();
listenMessages();
}
private void listenMessages() {
while (!stop) {
String message = messageReciever.receiveMessage(null,this.jmsReplyQueue);
count++;
String s = "message: " + message + " Recieved by: "
+ processId + " Total recieved: " + count;
System.out.println(s);
writeOutputFile(s);
if (StringUtils.isNotEmpty(message) && message.equals("STOP")) {
stop = true;
}
}
}
private void openOutputFile() {
try {
outputFile = new File("C:/mahi/Test", "file." + processId);
outputFileWriter = new FileWriter(outputFile);
} catch (IOException e) {
System.out.println("Exception while opening file");
stop = true;
}
}
private void writeOutputFile(String message) {
try {
outputFileWriter.write(message);
outputFileWriter.flush();
} catch (IOException e) {
System.out.println("Exception while writing to file");
stop = true;
}
}
}
How will I know when the ThreadPool has finished processing so I can do my other clean up work?
Thanks

If you Worker class implements Callable instead of Runnable, then you'd be able to see when your threads complete by using a Future object to see if the Thread has returned some result (e.g. boolean which would tell you whether it has finished execution or not).
Take a look in section "8. Futures and Callables" # website below, it has exactly what you need imo:
http://www.vogella.com/articles/JavaConcurrency/article.html
Edit: So after all of the Futures indicate that their respective Callable's execution is complete, its safe to assume your executor has finished execution and can be shutdown/terminated manually.

Something like this:
exec.shutdown();
// waiting for executors to finish their jobs
while (!exec.awaitTermination(50, TimeUnit.MILLISECONDS));
// perform clean up work

You can use a thread for monitoring ThreadPoolExecutor like that
import java.util.concurrent.ThreadPoolExecutor;
public class MyMonitorThread implements Runnable {
private ThreadPoolExecutor executor;
private int seconds;
private boolean run=true;
public MyMonitorThread(ThreadPoolExecutor executor, int delay)
{
this.executor = executor;
this.seconds=delay;
}
public void shutdown(){
this.run=false;
}
#Override
public void run()
{
while(run){
System.out.println(
String.format("[monitor] [%d/%d] Active: %d, Completed: %d, Task: %d, isShutdown: %s, isTerminated: %s",
this.executor.getPoolSize(),
this.executor.getCorePoolSize(),
this.executor.getActiveCount(),
this.executor.getCompletedTaskCount(),
this.executor.getTaskCount(),
this.executor.isShutdown(),
this.executor.isTerminated()));
try {
Thread.sleep(seconds*1000);
} catch (InterruptedException e) {
e.printStackTrace();
}
}
}
}
And add
MyMonitorThread monitor = new MyMonitorThread(executorPool, 3);
Thread monitorThread = new Thread(monitor);
monitorThread.start();
to your class where ThreadPoolExecutor is located.
It will show your threadpoolexecutors states in every 3 seconds.