We Keep Coding

How do I make a single-task FILO background thread?

I have a bunch of threads that spawn somewhat arbitrarily. When they are racing each other, only the one that spawned last is relevant. The other threads can be thrown away or stopped. But I am not sure how to do that, so I have implemented a very basic counter that checks whether the thread is the latest spawned thread.
edit: I would like to be able to kill threads that are taking too long (as they are no longer necessary); probably not from within the threads themselves as they are busy doing something else.
This code works, it seems. But it doesn't feel robust. Can someone give me a hint toward a proper way to do this?
class Main {
private static volatile int latestThread = 0;
public static void main(String[] args) {
for (int i = 0; i < 10; i++) {
spawnThread();
}
}
private static void spawnThread() {
latestThread++;
int thisThread = latestThread;
new Thread(() -> {
try {
Thread.sleep(10);
} catch (InterruptedException e) {
e.printStackTrace();
}
if (latestThread == thisThread) {
// only the latest "active" thread is relevant
System.out.println("I am the latest thread! " + thisThread);
}
}).start();
}
}
output:
I am the latest thread! 10
code in replit.com

ThreadPoolExecutor is almost what I need, specifically DiscardOldestPolicy. You can set the queue size to 1, so one thread is running and one thread is in the queue, and the oldest in the queue just gets shunted. Clean!
But it finishes two threads (not only the latest), which is not 100% what I was looking for. Although arguably good enough:
import java.util.concurrent.ArrayBlockingQueue;
import java.util.concurrent.ThreadPoolExecutor;
import java.util.concurrent.TimeUnit;
import java.util.concurrent.atomic.AtomicInteger;
public class DiscardOldest {
private static int threadCounter = 1;
public static void main(String[] args) throws InterruptedException {
int poolSize = 0;
int maxPoolSize = 1;
int queueSize = 1;
long aliveTime = 1000;
ArrayBlockingQueue<Runnable> queue = new ArrayBlockingQueue<>(queueSize);
ThreadPoolExecutor executor = new ThreadPoolExecutor(poolSize, maxPoolSize, aliveTime, TimeUnit.MILLISECONDS, queue, new ThreadPoolExecutor.DiscardOldestPolicy());
for (int i = 0; i < 4; i++) {
spawnThread(executor);
}
}
private static void spawnThread(ThreadPoolExecutor executor) {
final int thisThread = threadCounter++;
System.out.println(thisThread + " spawning");
executor.execute(() -> {
try {
Thread.sleep(100);
System.out.println(thisThread + " finished!");
} catch (InterruptedException e) {
e.printStackTrace();
}
});
}
}
Ouput:
1 spawning
2 spawning
3 spawning
4 spawning
1 finished!
4 finished!

Rather than relaying on an index, a born time could be set. If there's a younger thread (was born later) the thread should terminate its execution.
public class Last {
private static volatile long latestThread = 0L;
/**
* #param args
*/
public static void main(String[] args) {
for (int i = 0; i < 3; i++) {
spawnThread(System.nanoTime(), i);
}
}
private static void spawnThread(long startTime, int index) {
new Thread(() -> {
latestThread = startTime;
long thisThread = startTime;
boolean die = false;
try {
while (!die) {
Thread.sleep(1);
if (thisThread < latestThread) {
System.out.println(
index + ": I am not the latest thread :-(\n\t" + thisThread + "\n\t" + latestThread);
die = true;
} else if (thisThread == latestThread) {
System.out.println(
index + ": Yes! This is the latest thread!\n\t" + thisThread + "\n\t" + latestThread);
Thread.sleep(1);
System.out.println("Bye!");
die = true;
}
}
} catch (InterruptedException e) {
e.printStackTrace();
}
}).start();
}
}
Result:
0: I am not the latest thread :-(
39667589567880
39667602317461
2: Yes! This is the latest thread!
39667602317461
39667602317461
1: I am not the latest thread :-(
39667602257160
39667602317461
Bye!

I did a little research based on comments from everybody (thanks!) and ThreadPoolExecutor is almost what I need, but I want a pool with the total size of 1 (no queue) that kills the active thread once a new thread comes along, which is not allowed in a thread pool and not in line with what a ThreadPool is for. So instead, I came up with a reference to the active thread, and when a new thread comes a long it kills the old one, which seems to do what I want:
import java.util.concurrent.atomic.AtomicInteger;
public class Interrupt {
private static final AtomicInteger CURRENT_THREAD = new AtomicInteger(0);
private static Thread activeThread = new Thread(() -> {});
public static void main(String[] args) throws InterruptedException {
for (int i = 0; i < 4; i++) {
spawnThread();
Thread.sleep(3);
}
}
private static void spawnThread() {
if (activeThread.isAlive()) {
activeThread.interrupt();
}
activeThread = new Thread(() -> {
int thisThread = CURRENT_THREAD.incrementAndGet();
System.out.println(thisThread + " working");
try {
Thread.sleep(1000);
System.out.println(thisThread + " finished!");
} catch (InterruptedException ignored) {}
});
activeThread.start();
}
}
Output:
3 working
2 working
1 working
4 working
4 finished!

How to generate PDF documents from rpt in a multi-threaded approach?

I have a rpt file, using which i will be generating multiple reports in pdf format. Using the Engine class from inet clear reports. The process takes very long as I have nearly 10000 reports to be generated. Can I use the Mutli-thread or some other approach to speed up the process?
Any help of how it can be done would be helpful
My partial code.
//Loops
Engine eng = new Engine(Engine.EXPORT_PDF);
eng.setReportFile(rpt); //rpt is the report name
if (cn.isClosed() || cn == null ) {
cn = ds.getConnection();
}
eng.setConnection(cn);
System.out.println(" After set connection");
eng.setPrompt(data[i], 0);
ReportProperties repprop = eng.getReportProperties();
repprop.setPaperOrient(ReportProperties.DEFAULT_PAPER_ORIENTATION, ReportProperties.PAPER_FANFOLD_US);
eng.execute();
System.out.println(" After excecute");
try {
PDFExportThread pdfExporter = new PDFExportThread(eng, sFileName, sFilePath);
pdfExporter.execute();
} catch (Exception e) {
e.printStackTrace();
}
PDFExportThread execute
public void execute() throws IOException {
FileOutputStream fos = null;
try {
String FileName = sFileName + "_" + (eng.getPageCount() - 1);
File file = new File(sFilePath + FileName + ".pdf");
if (!file.getParentFile().exists()) {
file.getParentFile().mkdirs();
}
if (!file.exists()) {
file.createNewFile();
}
fos = new FileOutputStream(file);
for (int k = 1; k <= eng.getPageCount(); k++) {
fos.write(eng.getPageData(k));
}
fos.flush();
fos.close();
} catch (Exception e) {
e.printStackTrace();
} finally {
if (fos != null) {
fos.close();
fos = null;
}
}
}

This is a very basic code. A ThreadPoolExecutor with a fixed size threads in a pool is the backbone.
Some considerations:
The thread pool size should be equal or less than the DB connection pool size. And, it should be of an optimal number which is reasonable for parallel Engines.
The main thread should wait for sufficient time before killing all threads. I have put 1 hour as the wait time, but that's just an example.
You'll need to have proper Exception handling.
From the API doc, I saw stopAll and shutdown methods from the Engine class. So, I'm invoking that as soon as our work is done. That's again, just an example.
Hope this helps.
import java.io.File;
import java.io.FileOutputStream;
import java.io.IOException;
import java.sql.Connection;
import java.util.concurrent.Executors;
import java.util.concurrent.ThreadPoolExecutor;
import java.util.concurrent.TimeUnit;
public class RunEngine {
public static void main(String[] args) throws Exception {
final String rpt = "/tmp/rpt/input/rpt-1.rpt";
final String sFilePath = "/tmp/rpt/output/";
final String sFileName = "pdfreport";
final Object[] data = new Object[10];
ThreadPoolExecutor executor = (ThreadPoolExecutor) Executors.newFixedThreadPool(10);
for (int i = 0; i < data.length; i++) {
PDFExporterRunnable runnable = new PDFExporterRunnable(rpt, data[i], sFilePath, sFileName, i);
executor.execute(runnable);
}
executor.shutdown();
executor.awaitTermination(1L, TimeUnit.HOURS);
Engine.stopAll();
Engine.shutdown();
}
private static class PDFExporterRunnable implements Runnable {
private final String rpt;
private final Object data;
private final String sFilePath;
private final String sFileName;
private final int runIndex;
public PDFExporterRunnable(String rpt, Object data, String sFilePath,
String sFileName, int runIndex) {
this.rpt = rpt;
this.data = data;
this.sFilePath = sFilePath;
this.sFileName = sFileName;
this.runIndex = runIndex;
}
#Override
public void run() {
// Loops
Engine eng = new Engine(Engine.EXPORT_PDF);
eng.setReportFile(rpt); // rpt is the report name
Connection cn = null;
/*
* DB connection related code. Check and use.
*/
//if (cn.isClosed() || cn == null) {
//cn = ds.getConnection();
//}
eng.setConnection(cn);
System.out.println(" After set connection");
eng.setPrompt(data, 0);
ReportProperties repprop = eng.getReportProperties();
repprop.setPaperOrient(ReportProperties.DEFAULT_PAPER_ORIENTATION,
ReportProperties.PAPER_FANFOLD_US);
eng.execute();
System.out.println(" After excecute");
FileOutputStream fos = null;
try {
String FileName = sFileName + "_" + runIndex;
File file = new File(sFilePath + FileName + ".pdf");
if (!file.getParentFile().exists()) {
file.getParentFile().mkdirs();
}
if (!file.exists()) {
file.createNewFile();
}
fos = new FileOutputStream(file);
for (int k = 1; k <= eng.getPageCount(); k++) {
fos.write(eng.getPageData(k));
}
fos.flush();
fos.close();
} catch (Exception e) {
e.printStackTrace();
} finally {
if (fos != null) {
try {
fos.close();
} catch (IOException e) {
e.printStackTrace();
}
fos = null;
}
}
}
}
/*
* Dummy classes to avoid compilation errors.
*/
private static class ReportProperties {
public static final String PAPER_FANFOLD_US = null;
public static final String DEFAULT_PAPER_ORIENTATION = null;
public void setPaperOrient(String defaultPaperOrientation, String paperFanfoldUs) {
}
}
private static class Engine {
public static final int EXPORT_PDF = 1;
public Engine(int exportType) {
}
public static void shutdown() {
}
public static void stopAll() {
}
public void setPrompt(Object singleData, int i) {
}
public byte[] getPageData(int k) {
return null;
}
public int getPageCount() {
return 0;
}
public void execute() {
}
public ReportProperties getReportProperties() {
return null;
}
public void setConnection(Connection cn) {
}
public void setReportFile(String reportFile) {
}
}
}

I will offer this "answer" as a possible quick & dirty solution to get you started on a parallelization effort.
One way or another you're going to build a render farm.
I don't think there is a trivial way to do this in java; I would love to have someone post an answer that show how to parallelize your example in just a few lines of code. But until that happens this will hopefully help you make some progress.
You're going to have limited scaling in the same JVM instance.
But... let's see how far you get with that and see if it helps enough.
Design challenge #1: restarting.
You will probably want a place to keep the status for each of your reports e.g. "units of work".
You want this in case you need to re-start everything (maybe your server crashes) and you don't want to re-run all of the reports thus far.
Lots of ways you can do this; database, check to see if a "completed" file exists in your report folder (not sufficient for the *.pdf to exist, as that may be incomplete... for xyz_200.pdf you could maybe make an empty xyz_200.done or xyz_200.err file to help with re-running any problem children... and by the time you code up that file manipulation/checking/initialization logic, seems like it may have been easier to add a column to your database which holds the list of work to-be-done).
Design consideration #2: maximizing throughput (avoiding overload).
You don't want to saturate you system and run one thousand reports in parallel.
Maybe 10.
Maybe 100.
Probably not 5,000.
You will need to do some sizing research and see what gets you near 80 to 90% system utilization.
Design consideration #3: scaling across multiple servers
Overly complex, outside the scope of a Stack Exchange answer.
You'd have to spin up JVM's on multiple systems that are running something like the workers below, and a report-manager that can pull work items from a shared "queue" structure, again a database table is probably easier here than doing something file-based (or a network feed).
Sample Code
Caution: None of this code is well tested, it almost certainly has an abundance of typos, logic errors and poor design. Use at your own risk.
So anyway... I do want to give you the basic idea of a rudimentary task runner.
Replace your "// Loops" example in the question with code like the following:
main loop (original code example)
This is more or less doing what your example code did, modified to push most of the work into ReportWorker (new class, see below). Lots of stuff seems to be packed into your original question's example of "// Loop", so I'm not trying to reverse engineer that.
fwiw, it was unclear to me where "rpt" and "data[i]" are coming from so I hacked up some test data.
public class Main {
public static boolean complete( String data ) {
return false; // for testing nothing is complete.
}
public static void main(String args[] ) {
String data[] = new String[] {
"A",
"B",
"C",
"D",
"E" };
String rpt = "xyz";
// Loop
ReportManager reportMgr = new ReportManager(); // a new helper class (see below), it assigns/monitors work.
long startTime = System.currentTimeMillis();
for( int i = 0; i < data.length; ++i ) {
// complete is something you should write that knows if a report "unit of work"
// finished successfully.
if( !complete( data[i] ) ) {
reportMgr.assignWork( rpt, data[i] ); // so... where did values for your "rpt" variable come from?
}
}
reportMgr.waitForWorkToFinish(); // out of new work to assign, let's wait until everything in-flight complete.
long endTime = System.currentTimeMillis();
System.out.println("Done. Elapsed time = " + (endTime - startTime)/1000 +" seconds.");
}
}
ReportManager
This class is not thread safe, just have your original loop keep calling assignWork() until you're out of reports to assign then keep calling it until all work is done, e.g. waitForWorkToFinish(), as shown above. (fwiw, I don't think you could say any of the classes here are especially thread safe).
public class ReportManager {
public int polling_delay = 500; // wait 0.5 seconds for testing.
//public int polling_delay = 60 * 1000; // wait 1 minute.
// not high throughput millions of reports / second, we'll run at a slower tempo.
public int nWorkers = 3; // just 3 for testing.
public int assignedCnt = 0;
public ReportWorker workers[];
public ReportManager() {
// initialize our manager.
workers = new ReportWorker[ nWorkers ];
for( int i = 0; i < nWorkers; ++i ) {
workers[i] = new ReportWorker( i );
System.out.println("Created worker #"+i);
}
}
private ReportWorker handleWorkerError( int i ) {
// something went wrong, update our "report" status as one of the reports failed.
System.out.println("handlerWokerError(): failure in "+workers[i]+", resetting worker.");
workers[i].teardown();
workers[i] = new ReportWorker( i ); // just replace everything.
return workers[i]; // the new worker will, incidentally, be avaialble.
}
private ReportWorker handleWorkerComplete( int i ) {
// this unit of work was completed, update our "report" status tracker as success.
System.out.println("handleWorkerComplete(): success in "+workers[i]+", resetting worker.");
workers[i].teardown();
workers[i] = new ReportWorker( i ); // just replace everything.
return workers[i]; // the new worker will, incidentally, be avaialble.
}
private int activeWorkerCount() {
int activeCnt = 0;
for( int i = 0; i < nWorkers; ++i ) {
ReportWorker worker = workers[i];
System.out.println("activeWorkerCount() i="+i+", checking worker="+worker);
if( worker.hasError() ) {
worker = handleWorkerError( i );
}
if( worker.isComplete() ) {
worker = handleWorkerComplete( i );
}
if( worker.isInitialized() || worker.isRunning() ) {
++activeCnt;
}
}
System.out.println("activeWorkerCount() activeCnt="+activeCnt);
return activeCnt;
}
private ReportWorker getAvailableWorker() {
// check each worker to see if anybody recently completed...
// This (rather lazily) creates completely new ReportWorker instances.
// You might want to try pooling (salvaging and reinitializing them)
// to see if that helps your performance.
System.out.println("\n-----");
ReportWorker firstAvailable = null;
for( int i = 0; i < nWorkers; ++i ) {
ReportWorker worker = workers[i];
System.out.println("getAvailableWorker(): i="+i+" worker="+worker);
if( worker.hasError() ) {
worker = handleWorkerError( i );
}
if( worker.isComplete() ) {
worker = handleWorkerComplete( i );
}
if( worker.isAvailable() && firstAvailable==null ) {
System.out.println("Apparently worker "+worker+" is 'available'");
firstAvailable = worker;
System.out.println("getAvailableWorker(): i="+i+" now firstAvailable = "+firstAvailable);
}
}
return firstAvailable; // May (or may not) be null.
}
public void assignWork( String rpt, String data ) {
ReportWorker worker = getAvailableWorker();
while( worker == null ) {
System.out.println("assignWork: No workers available, sleeping for "+polling_delay);
try { Thread.sleep( polling_delay ); }
catch( InterruptedException e ) { System.out.println("assignWork: sleep interrupted, ignoring exception "+e); }
// any workers avaialble now?
worker = getAvailableWorker();
}
++assignedCnt;
worker.initialize( rpt, data ); // or whatever else you need.
System.out.println("assignment #"+assignedCnt+" given to "+worker);
Thread t = new Thread( worker );
t.start( ); // that is pretty much it, let it go.
}
public void waitForWorkToFinish() {
int active = activeWorkerCount();
while( active >= 1 ) {
System.out.println("waitForWorkToFinish(): #active workers="+active+", waiting...");
// wait a minute....
try { Thread.sleep( polling_delay ); }
catch( InterruptedException e ) { System.out.println("assignWork: sleep interrupted, ignoring exception "+e); }
active = activeWorkerCount();
}
}
}
ReportWorker
public class ReportWorker implements Runnable {
int test_delay = 10*1000; //sleep for 10 seconds.
// (actual code would be generating PDF output)
public enum StatusCodes { UNINITIALIZED,
INITIALIZED,
RUNNING,
COMPLETE,
ERROR };
int id = -1;
StatusCodes status = StatusCodes.UNINITIALIZED;
boolean initialized = false;
public String rpt = "";
public String data = "";
//Engine eng;
//PDFExportThread pdfExporter;
//DataSource_type cn;
public boolean isInitialized() { return initialized; }
public boolean isAvailable() { return status == StatusCodes.UNINITIALIZED; }
public boolean isRunning() { return status == StatusCodes.RUNNING; }
public boolean isComplete() { return status == StatusCodes.COMPLETE; }
public boolean hasError() { return status == StatusCodes.ERROR; }
public ReportWorker( int id ) {
this.id = id;
}
public String toString( ) {
return "ReportWorker."+id+"("+status+")/"+rpt+"/"+data;
}
// the example code doesn't make clear if there is a relationship between rpt & data[i].
public void initialize( String rpt, String data /* data[i] in original code */ ) {
try {
this.rpt = rpt;
this.data = data;
/* uncomment this part where you have the various classes availble.
* I have it commented out for testing.
cn = ds.getConnection();
Engine eng = new Engine(Engine.EXPORT_PDF);
eng.setReportFile(rpt); //rpt is the report name
eng.setConnection(cn);
eng.setPrompt(data, 0);
ReportProperties repprop = eng.getReportProperties();
repprop.setPaperOrient(ReportProperties.DEFAULT_PAPER_ORIENTATION, ReportProperties.PAPER_FANFOLD_US);
*/
status = StatusCodes.INITIALIZED;
initialized = true; // want this true even if we're running.
} catch( Exception e ) {
status = StatusCodes.ERROR;
throw new RuntimeException("initialze(rpt="+rpt+", data="+data+")", e);
}
}
public void run() {
status = StatusCodes.RUNNING;
System.out.println("run().BEGIN: "+this);
try {
// delay for testing.
try { Thread.sleep( test_delay ); }
catch( InterruptedException e ) { System.out.println(this+".run(): test interrupted, ignoring "+e); }
/* uncomment this part where you have the various classes availble.
* I have it commented out for testing.
eng.execute();
PDFExportThread pdfExporter = new PDFExportThread(eng, sFileName, sFilePath);
pdfExporter.execute();
*/
status = StatusCodes.COMPLETE;
System.out.println("run().END: "+this);
} catch( Exception e ) {
System.out.println("run().ERROR: "+this);
status = StatusCodes.ERROR;
throw new RuntimeException("run(rpt="+rpt+", data="+data+")", e);
}
}
public void teardown() {
if( ! isInitialized() || isRunning() ) {
System.out.println("Warning: ReportWorker.teardown() called but I am uninitailzied or running.");
// should never happen, fatal enough to throw an exception?
}
/* commented out for testing.
try { cn.close(); }
catch( Exception e ) { System.out.println("Warning: ReportWorker.teardown() ignoring error on connection close: "+e); }
cn = null;
*/
// any need to close things on eng?
// any need to close things on pdfExporter?
}
}

FutureTask.get method is blocking forever, although I am setting it's value from another thread

I am trying to signal between two threads using the below FutureResult class which extends FutureTask class. When run the script, it prints the following result.
SENDING: 0
SENT: 0
POLL: FutureResult#513431
SIGNALLED: FutureResult#513431
Then the program hang up forever. I expect FutureResult instance should return the value from it's blocking get method. Then print the result in the console. But FutureResult.get is blocking forever.
import java.util.concurrent.*;
/**
* Created by someone on 20/08/2015.
*/
final public class FutureResult<T> extends FutureTask<T> {
private static final Object SS = "SS";
public FutureResult() {
super(() -> null);
}
public void signal(final T value) {
set(value);
}
public void signalError(final Throwable throwable) {
setException(throwable);
}
public static void main(String... args) throws Exception {
final ArrayBlockingQueue<FutureResult> queue = new ArrayBlockingQueue<>(1000000);
new Thread(() -> {
while (true) {
try {
final FutureResult poll = queue.take();
System.out.println("POLL: " + poll);
if (poll != null) {
poll.signal(SS);
System.out.println("SIGNALLED: " + poll);
}
} catch (Exception e) {
e.printStackTrace();
}
}
}).start();
new Thread(() -> {
for (int i = 0; i < 1; i++) {
final FutureResult<Object> result = new FutureResult<>();
System.out.println("SENDING: " + i);
queue.offer(new FutureResult());
try {
System.out.println("SENT: " + i);
result.get();
System.out.println("GOT : " + i);
} catch (InterruptedException e) {
e.printStackTrace();
} catch (ExecutionException e) {
e.printStackTrace();
}
}
}).start();
}
}

This is the problem:
queue.offer(new FutureResult());
You're setting the value on one FutureResult, but that's not the one you're waiting for. Just change that line to:
queue.offer(result);
and it works fine.

Looks like the confusion is in the use of FutureTask. FutureTask is designed as a Runnable; running it is necessary.
Honestly, based on the code, it looks like the custom code is implementing something similar to FutureTask. If the intent here is to learn to use FutureTask, then create a FutureTask instance with a "run" method, and then execute that run method. On completion of the run method, the FutureTask.get() will complete.

Using CountDownLatch & Object.wait inside recursive block hangs

Problem: While trying to retrieve values inside a recursive block in a phased manner, the execution gets hung.
Description: CountDownLatch & Object.wait are used to achieve the phased manner access of value inside the recursive block. But, the program hangs with following output:
2 < 16
3 < 16
4 < 16
5 < 16
Current total: 5
Inside of wait
Inside of wait
Program:
import java.util.concurrent.*;
public class RecursiveTotalFinder {
private static CountDownLatch latch1;
private static CountDownLatch latch2;
private static CountDownLatch latch3;
public static void main(String... args) {
latch1 = new CountDownLatch(1);
latch2 = new CountDownLatch(1);
latch3 = new CountDownLatch(1);
//Create object
TotalFinder tf = new TotalFinder(latch1,latch2,latch3);
//Start the thread
tf.start();
//Wait for results from TotalFinder
try {
latch1.await();
} catch(InterruptedException ie) {
ie.printStackTrace();
}
//Print the result after 5th iteration
System.out.println("Current total: "+tf.getCurrentTotal());
tf.releaseWaitLock();
tf.resetWaitLock();
//Wait for results again
try {
latch2.await();
} catch(InterruptedException ie) {
ie.printStackTrace();
}
//Print the result after 10th iteration
System.out.println("Current total: "+tf.getCurrentTotal());
tf.releaseWaitLock();
tf.resetWaitLock();
//Wait for results again
try {
latch3.await();
} catch(InterruptedException ie) {
ie.printStackTrace();
}
//Print the result after 15th iteration
System.out.println("Current total: "+tf.getCurrentTotal());
tf.releaseWaitLock();
tf.resetWaitLock();
}
}
class TotalFinder extends Thread{
CountDownLatch tfLatch1;
CountDownLatch tfLatch2;
CountDownLatch tfLatch3;
private static int count = 1;
private static final class Lock { }
private final Object lock = new Lock();
private boolean gotSignalFromMaster = false;
public TotalFinder(CountDownLatch latch1, CountDownLatch latch2,
CountDownLatch latch3) {
tfLatch1 = latch1;
tfLatch2 = latch2;
tfLatch3 = latch3;
}
public void run() {
findTotal(16);
}
//Find total
synchronized void findTotal(int cnt) {
if(count%5==0) {
if(count==5)
tfLatch1.countDown();
if(count==10)
tfLatch2.countDown();
if(count==15)
tfLatch3.countDown();
//Sleep for sometime
try {
Thread.sleep(3000);
} catch(InterruptedException ie) {
ie.printStackTrace();
}
//Wait till current total is printed
synchronized(lock) {
while(gotSignalFromMaster==false) {
try {
System.out.println(" Inside of wait");
lock.wait();
} catch(InterruptedException ie) {
ie.printStackTrace();
}
}
System.out.println("Came outside of wait");
}
}
count +=1;
if(count < cnt) {
System.out.println(count +" < "+cnt);
findTotal(cnt);
}
}
//Return the count value
public int getCurrentTotal() {
return count;
}
//Release lock
public void releaseWaitLock() {
//Sleep for sometime
try {
Thread.sleep(5000);
} catch(InterruptedException ie) {
ie.printStackTrace();
}
synchronized(lock) {
gotSignalFromMaster=true;
lock.notifyAll();
}
}
//Reset wait lock
public void resetWaitLock() {
gotSignalFromMaster = false;
}
}
Analysis:
In my initial analysis it looks like the wait is happening recursively eventhough notifyAll is invoked from the main program.
Help:
Why free lock using notfiyAll after a CountDownLatch didn't take effect? Need someone's help in understanding what exactly is happening in this program.

The main message about wait and notify that I got from JCIP was that I'd probably use them wrongly, so better to avoid using them directly unless strictly necessary. As such, I think that you should reconsider the use of these methods.
In this case, I think that you can do it more elegantly using SynchronousQueue. Perhaps something like this might work:
import java.util.concurrent.*;
public class RecursiveTotalFinder {
public static void main(String... args) throws InterruptedException {
SynchronousQueue<Integer> syncQueue = new SynchronousQueue<>();
//Create object
TotalFinder tf = new TotalFinder(syncQueue, 5);
//Start the thread
tf.start();
for (int i = 0; i < 3; ++i) {
System.out.println("Current total: " + syncQueue.take());
}
}
}
class TotalFinder extends Thread{
private final SynchronousQueue<Integer> syncQueue;
private final int syncEvery;
private int count;
public TotalFinder(SynchronousQueue<Integer> syncQueue,
int syncEvery) {
this.syncQueue = syncQueue;
this.syncEvery = syncEvery;
}
public void run() {
try {
findTotal(16);
} catch (InterruptedException e) {
Thread.currentThread().interrupt();
throw new RuntimeException(e);
}
}
//Find total
void findTotal(int cnt) throws InterruptedException {
if((count > 0) && (count%syncEvery==0)) {
syncQueue.put(count);
}
count +=1;
if(count < cnt) {
System.out.println(count +" < "+cnt);
findTotal(cnt);
}
}
}
As to why your original approach doesn't work, it's because the main thread sets gotSignalFromMaster to true and then immediately back to false, and this happens before the other thread is able to check its value. If you stick a bit of a sleep into the resetWaitLock, it proceeds beyond the point where it currently hangs; however, it then hangs at the end instead of terminating.
Note that having to use Thread.sleep to wait for another thread to change some state is a poor approach - not least because it makes your program really slow. Using synchronization utilities leads to faster and much easier-to-reason-about program.

How do we know threadPoolExecutor has finished execution

I have a parent thread that sends messages to MQ and it manages a ThreadPoolExecutor for worker threads which listen to MQ and writes message to output file. I manage a threadpool of size 5. So when I run my program, I have 5 files with messages. Everything works fine until here. I now need to merge these 5 files in my parent thread.
How do I know ThreadPoolExecutor finished processing so I can start merging files.
public class ParentThread {
private MessageSender messageSender;
private MessageReciever messageReciever;
private Queue jmsQueue;
private Queue jmsReplyQueue;
ExecutorService exec = Executors.newFixedThreadPool(5);
public void sendMessages() {
System.out.println("Sending");
File xmlFile = new File("c:/filename.txt");
List<String> lines = null;
try {
lines = FileUtils.readLines(xmlFile, null);
} catch (IOException e) {
e.printStackTrace();
}
for (String line : lines){
messageSender.sendMessage(line, this.jmsQueue, this.jmsReplyQueue);
}
int count = 0;
while (count < 5) {
messageSender.sendMessage("STOP", this.jmsQueue, this.jmsReplyQueue);
count++;
}
}
public void listenMessages() {
long finishDate = new Date().getTime();
for (int i = 0; i < 5; i++) {
Worker worker = new Worker(i, this.messageReciever, this.jmsReplyQueue);
exec.execute(worker);
}
exec.shutdown();
if(exec.isTerminated()){ //PROBLEM is HERE. Control Never gets here.
long currenttime = new Date().getTime() - finishDate;
System.out.println("time taken: "+currenttime);
mergeFiles();
}
}
}
This is my worker class
public class Worker implements Runnable {
private boolean stop = false;
private MessageReciever messageReciever;
private Queue jmsReplyQueue;
private int processId;
private int count = 0;
private String message;
private File outputFile;
private FileWriter outputFileWriter;
public Worker(int processId, MessageReciever messageReciever,
Queue jmsReplyQueue) {
this.processId = processId;
this.messageReciever = messageReciever;
this.jmsReplyQueue = jmsReplyQueue;
}
public void run() {
openOutputFile();
listenMessages();
}
private void listenMessages() {
while (!stop) {
String message = messageReciever.receiveMessage(null,this.jmsReplyQueue);
count++;
String s = "message: " + message + " Recieved by: "
+ processId + " Total recieved: " + count;
System.out.println(s);
writeOutputFile(s);
if (StringUtils.isNotEmpty(message) && message.equals("STOP")) {
stop = true;
}
}
}
private void openOutputFile() {
try {
outputFile = new File("C:/mahi/Test", "file." + processId);
outputFileWriter = new FileWriter(outputFile);
} catch (IOException e) {
System.out.println("Exception while opening file");
stop = true;
}
}
private void writeOutputFile(String message) {
try {
outputFileWriter.write(message);
outputFileWriter.flush();
} catch (IOException e) {
System.out.println("Exception while writing to file");
stop = true;
}
}
}
How will I know when the ThreadPool has finished processing so I can do my other clean up work?
Thanks

If you Worker class implements Callable instead of Runnable, then you'd be able to see when your threads complete by using a Future object to see if the Thread has returned some result (e.g. boolean which would tell you whether it has finished execution or not).
Take a look in section "8. Futures and Callables" # website below, it has exactly what you need imo:
http://www.vogella.com/articles/JavaConcurrency/article.html
Edit: So after all of the Futures indicate that their respective Callable's execution is complete, its safe to assume your executor has finished execution and can be shutdown/terminated manually.

Something like this:
exec.shutdown();
// waiting for executors to finish their jobs
while (!exec.awaitTermination(50, TimeUnit.MILLISECONDS));
// perform clean up work

You can use a thread for monitoring ThreadPoolExecutor like that
import java.util.concurrent.ThreadPoolExecutor;
public class MyMonitorThread implements Runnable {
private ThreadPoolExecutor executor;
private int seconds;
private boolean run=true;
public MyMonitorThread(ThreadPoolExecutor executor, int delay)
{
this.executor = executor;
this.seconds=delay;
}
public void shutdown(){
this.run=false;
}
#Override
public void run()
{
while(run){
System.out.println(
String.format("[monitor] [%d/%d] Active: %d, Completed: %d, Task: %d, isShutdown: %s, isTerminated: %s",
this.executor.getPoolSize(),
this.executor.getCorePoolSize(),
this.executor.getActiveCount(),
this.executor.getCompletedTaskCount(),
this.executor.getTaskCount(),
this.executor.isShutdown(),
this.executor.isTerminated()));
try {
Thread.sleep(seconds*1000);
} catch (InterruptedException e) {
e.printStackTrace();
}
}
}
}
And add
MyMonitorThread monitor = new MyMonitorThread(executorPool, 3);
Thread monitorThread = new Thread(monitor);
monitorThread.start();
to your class where ThreadPoolExecutor is located.
It will show your threadpoolexecutors states in every 3 seconds.