This question already has answers here:
How to wait for all threads to finish, using ExecutorService?
(27 answers)
Closed 8 years ago.
Please have a look at the following code.
public class BigFileWholeProcessor {
private static final int NUMBER_OF_THREADS = 2;
public void processFile(String fileName) {
BlockingQueue<String> fileContent = new LinkedBlockingQueue<String>();
BigFileReader bigFileReader = new BigFileReader(fileName, fileContent);
BigFileProcessor bigFileProcessor = new BigFileProcessor(fileContent);
ExecutorService es = Executors.newFixedThreadPool(NUMBER_OF_THREADS);
es.execute(bigFileReader);
es.execute(bigFileProcessor);
es.shutdown();
if(es.isTerminated())
{
System.out.println("Completed Work");
}
}
}
public class BigFileReader implements Runnable {
private final String fileName;
int a = 0;
public static final String SENTINEL = "SENTINEL";
private final BlockingQueue<String> linesRead;
public BigFileReader(String fileName, BlockingQueue<String> linesRead) {
this.fileName = fileName;
this.linesRead = linesRead;
}
#Override
public void run() {
try {
//since it is a sample, I avoid the manage of how many lines you have read
//and that stuff, but it should not be complicated to accomplish
BufferedReader br = new BufferedReader(new FileReader(new File("E:/Amazon HashFile/Hash.txt")));
String str = "";
while((str=br.readLine())!=null)
{
linesRead.put(str);
System.out.println(a);
a++;
}
linesRead.put(SENTINEL);
} catch (Exception ex) {
ex.printStackTrace();
}
System.out.println("Completed");
}
}
public class BigFileProcessor implements Runnable {
private final BlockingQueue<String> linesToProcess;
public BigFileProcessor (BlockingQueue<String> linesToProcess) {
this.linesToProcess = linesToProcess;
}
#Override
public void run() {
String line = "";
try {
while ( (line = linesToProcess.take()) != null) {
//do what you want/need to process this line...
if(line==BigFileReader.SENTINEL)
{
break;
}
String [] pieces = line.split("(...)/g");
}
} catch (InterruptedException e) {
e.printStackTrace();
}
}
}
I want to print the text "completed work" in BigFileWholeProcessor once all the thread work is done. But instead, it is not getting printed. Why is this? How to identify that all the threads are done and need printing?
shutdown() only signal ES to shutdown, you need
awaitTermination(long timeout, TimeUnit unit)
before print message
Use submit() method instead of execute(). The get() method can be used if you want to wait for the thread to finish at any point of time. Read documentation on use of Future object for further details.
ExecutorService es = Executors.newFixedThreadPool(2);
Future<?> f = es.submit(new Thread(new TestRun()));
f.get(); // Wait for result... (i.e similar to `join()` in this case)
es.shutdown(); // Shutdown ExecutorService
System.out.println("Done.");
I have defined a TestRun class implementing Runnable, not shown here. The Future object makes more sense in other scenarios.
Related
I am trying to process few million records from a text file (i.e. reading the file sequentially using one thread, and trying to process the retrieved lines using multiple threads). A method call after 'queue.take();' is only executing for the number of times equal to initial capacity allocated to BlockingQueue (100 in this example), and then the process doesn't pickup anymore records.
Could you please help in debugging or identify the issue?
Main Method:
############
final int threadCount = 10;
BlockingQueue<String> queue = new ArrayBlockingQueue<String>(100);
ExecutorService service = Executors.newFixedThreadPool(threadCount);
for (int i = 0; i < (threadCount - 1); i++) {
service.submit(new EvaluateLine(queue));
}
service.submit(new ProcessFile(queue)).get();
service.shutdownNow();
service.awaitTermination(365, TimeUnit.DAYS);
EvaluateLine:
#############
private final BlockingQueue<String> queue;
public EvaluateLine(BlockingQueue<String> queue){
this.queue = queue;
}
#Override
public void run() {
String line;
while(true) {
try {
line = queue.take();
SyncOutput.Process(line);
} catch (InterruptedException ex) {
break;
}
}
}
ProcessFile:
############
private final BlockingQueue<String> queue;
public ProcessFile(BlockingQueue<String> queue) {
this.queue = queue;
}
#Override
public void run() {
Path path = Paths.get("C:\\Desktop\\testdata.txt");
BufferedReader br = null;
try {
br =Files.newBufferedReader(path, StandardCharsets.UTF_8);
for (String line; (line = br.readLine()) != null; ) {
queue.put(line);
}
} catch(IOException e){
e.printStackTrace();
} catch(InterruptedException e){
e.printStackTrace();
} finally{
try {
if (br != null) br.close();
}catch(IOException e){
e.printStackTrace();
}
}
Edit:
SyncOutput
##########
class SyncOutput{
public static void ProcessExpression(String inputLine) {
evalExpression(inputLine);
}
public static double evalExpression(String s){
Expression e = new ExpressionBuilder(s)
.build();
return e.evaluate();
}
}
The following code segment(using Expression evaluation library: exp4j) is blocking the multi-thread execution, not sure why. But I have replaced this code block with a different library(parsii) and now everything looks fine.
public static void ProcessExpression(String inputLine) {
evalExpression(inputLine);
}
public static double evalExpression(String s){
Expression e = new ExpressionBuilder(s)
.build();
return e.evaluate();
}
I am trying to figure out how I can track all the threads that my application is spawning. Initially, I thought I had it figured out using a CyclicBarrier, however I am seeing threads executing after my await call.
Below is the working pseudo code:
public class ThreadTesterRunner {
public static void main(String[] args) throws InterruptedException {
final CyclicBarrier cb = new CyclicBarrier(1);
ThreadRunner tr = new ThreadRunner(cb);
Thread t = new Thread(tr, "Thread Runner");
t.start();
boolean process = true;
// wait until all threads process, then print reports
while (process){
if(tr.getIsFinished()){
System.out.println("Print metrics");
process = false;
}
Thread.sleep(1000);
}
}
}
class ThreadRunner implements Runnable {
static int timeOutTime = 2;
private ExecutorService executorService = Executors.newFixedThreadPool(10);
private final CyclicBarrier barrier;
private boolean isFinished=false;
public ThreadRunner(CyclicBarrier cb) {
this.barrier = cb;
}
public void run(){
try {
boolean stillLoop = true; int i = 0;
while (stillLoop){
int size;
Future<Integer> future = null;
try {
future = executorService.submit(new Reader()); // sleeps
size = future.get();
} catch (InterruptedException | ExecutionException ex) {
// handle Errs
}
if(i == 3){
stillLoop = false;
this.barrier.await();
this.isFinished=true;
}
//System.out.println("i = "+i+" Size is: "+size+"\r");
i++;
}
} catch (InterruptedException | BrokenBarrierException e1) {
e1.printStackTrace();
}
}
public boolean getIsFinished(){
return this.isFinished;
}
}
class Reader implements Callable {
private ExecutorService executorService = Executors.newFixedThreadPool(1);
#Override
public Object call() throws Exception {
System.out.println("Reading...");
Thread.sleep(2000);
executorService.submit(new Writer());
return 1000;
}
}
class Writer implements Callable {
#Override
public Void call() throws Exception {
Thread.sleep(4000);
System.out.println("Wrote");
return null;
}
}
Can anyone suggest a way to ONLY print "print metrics" after all threads have run?
It doesn't seem like you're doing anything to coordinate with your Reader and Writer threads, which are the ones you want to wait for. If you pass your synchronization barrier through to those threads so that they can register and signal when they are done, it works just fine.
Here's a version rewritten to do so, using a Phaser instead of a CyclicBarrier. Note that each Reader and Writer registers itself upon construction, and notifies the synchronization barrier when it is done executing:
public class ThreadTesterRunner {
public static void main(String[] args) throws InterruptedException {
final Phaser cb = new Phaser();
ThreadRunner tr = new ThreadRunner(cb);
Thread t = new Thread(tr, "Thread Runner");
t.start();
boolean process = true;
// wait until all threads process, then print reports
while (process){
if(tr.getIsFinished()){
System.out.println("Print metrics");
process = false;
}
//else {
// System.out.println("Waiting: registered=" + cb.getRegisteredParties() + ", arrived=" + cb.getArrivedParties() + ", unarrived=" + cb.getUnarrivedParties());
//}
Thread.sleep(1000);
}
}
}
class ThreadRunner implements Runnable {
static int timeOutTime = 2;
private ExecutorService executorService = Executors.newFixedThreadPool(10);
private final Phaser barrier;
private boolean isFinished=false;
public ThreadRunner(Phaser phaser) {
this.barrier = phaser;
}
public void run(){
try {
boolean stillLoop = true; int i = 0;
while (stillLoop){
int size;
Future<Integer> future = null;
try {
future = executorService.submit(new Reader(this.barrier)); // sleeps
size = future.get();
} catch (InterruptedException | ExecutionException ex) {
// handle Errs
}
if(i == 3){
stillLoop = false;
this.barrier.awaitAdvance(0);
this.isFinished=true;
}
//System.out.println("i = "+i+" Size is: "+size+"\r");
i++;
}
} catch (Exception e1) {
e1.printStackTrace();
}
}
public boolean getIsFinished(){
return this.isFinished;
}
}
class Reader implements Callable {
private Phaser barrier;
private ExecutorService executorService = Executors.newFixedThreadPool(1);
public Reader(Phaser phase) {
phase.register();
this.barrier = phase;
}
#Override
public Object call() throws Exception {
System.out.println("Reading...");
Thread.sleep(2000);
executorService.submit(new Writer(this.barrier));
this.barrier.arrive();
return 1000;
}
}
class Writer implements Callable {
private Phaser barrier;
public Writer(Phaser phase) {
phase.register();
this.barrier = phase;
}
#Override
public Void call() throws Exception {
Thread.sleep(4000);
System.out.println("Wrote");
this.barrier.arrive();
return null;
}
}
From what I can see you aren't waiting for the Writer to finish in the Reader. Is that the problem you are seeing?
You are also accessing isFinished from more than one thread without synchronization (which however, merely may delay the termination of the loop in this situation).
I don't see CyclicBarrier doing anything.
Not sure what you are trying to do, but I'd think about how simpler I can make it. For example, can Reader and Writer be combined into one task? Then, waiting for them to finish would merely be:
executorService.invokeAll(tasks);
System.out.println("Print metrics");
where tasks is a collection of tasks (see also this javadoc)
I want the threads to wait until last one uses br and notifies the other threads. But it gets stuck entering first wait(),what am i missing?
public class CrawlerThread implements Runnable {
private BufferedReader br;
private FileHandler fileHandler;
private File sourceFile;
private String skillString;
private Map<String, String> urlData = new HashMap<String, String>();
private String urlFirst = Initializer.urlFirst;
public static Integer threadCount = 0;
public CrawlerThread(BufferedReader br, FileHandler fileHandler,
File sourceFile, String skillString, Map<String, String> urlData) {
this.br = br;
this.fileHandler = fileHandler;
this.sourceFile = sourceFile;
this.skillString = skillString;
this.urlData.putAll(urlData);
new Thread(this).start();
}
#Override
public void run() {
System.out.println("!!!!");
String companyName;
String searchString;
SearchObject searchObject = new SearchObject();
try {String c;
while ((c=br.readLine())!=null && c.equalsIgnoreCase("Company Name")) {
try {
if ((companyName = br.readLine().trim()) != null) {
if (threadCount == (Initializer.MAX_THREAD - 1)) {
synchronized(br){
System.out.println("++");
br.close();
br.notifyAll();}
} else
try {
System.out.println("**" + threadCount);
synchronized (br) {
synchronized (threadCount) {
threadCount++;
}
br.wait();
}
} catch (InterruptedException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
To use wait/notify, both threads should acquire the shared lock and both check for a condition and modify if necessary, if you are certain there will be only 1 thread waiting the notify() is OK, if not then use notifyAll(), basically the waiting thread should be like like:
Sample waiting thread condition:
synchronized(lock){
while(!condition){
lock.wait();
}
}
Sample notifier thread:
synchronized(lock){
condition=true;
lock.notifyAll();
}
You could also use a CountDownLatch:
final CountDownLatch latch=new CountDownLatch(1);
Waiting thread:
public void waitForCondition(){
latch.await();
}
Notifier thread:
public void notifyWaitingTreads(){
latch.countDown();
}
I have a parent thread that sends messages to MQ and it manages a ThreadPoolExecutor for worker threads which listen to MQ and writes message to output file. I manage a threadpool of size 5. So when I run my program, I have 5 files with messages. Everything works fine until here. I now need to merge these 5 files in my parent thread.
How do I know ThreadPoolExecutor finished processing so I can start merging files.
public class ParentThread {
private MessageSender messageSender;
private MessageReciever messageReciever;
private Queue jmsQueue;
private Queue jmsReplyQueue;
ExecutorService exec = Executors.newFixedThreadPool(5);
public void sendMessages() {
System.out.println("Sending");
File xmlFile = new File("c:/filename.txt");
List<String> lines = null;
try {
lines = FileUtils.readLines(xmlFile, null);
} catch (IOException e) {
e.printStackTrace();
}
for (String line : lines){
messageSender.sendMessage(line, this.jmsQueue, this.jmsReplyQueue);
}
int count = 0;
while (count < 5) {
messageSender.sendMessage("STOP", this.jmsQueue, this.jmsReplyQueue);
count++;
}
}
public void listenMessages() {
long finishDate = new Date().getTime();
for (int i = 0; i < 5; i++) {
Worker worker = new Worker(i, this.messageReciever, this.jmsReplyQueue);
exec.execute(worker);
}
exec.shutdown();
if(exec.isTerminated()){ //PROBLEM is HERE. Control Never gets here.
long currenttime = new Date().getTime() - finishDate;
System.out.println("time taken: "+currenttime);
mergeFiles();
}
}
}
This is my worker class
public class Worker implements Runnable {
private boolean stop = false;
private MessageReciever messageReciever;
private Queue jmsReplyQueue;
private int processId;
private int count = 0;
private String message;
private File outputFile;
private FileWriter outputFileWriter;
public Worker(int processId, MessageReciever messageReciever,
Queue jmsReplyQueue) {
this.processId = processId;
this.messageReciever = messageReciever;
this.jmsReplyQueue = jmsReplyQueue;
}
public void run() {
openOutputFile();
listenMessages();
}
private void listenMessages() {
while (!stop) {
String message = messageReciever.receiveMessage(null,this.jmsReplyQueue);
count++;
String s = "message: " + message + " Recieved by: "
+ processId + " Total recieved: " + count;
System.out.println(s);
writeOutputFile(s);
if (StringUtils.isNotEmpty(message) && message.equals("STOP")) {
stop = true;
}
}
}
private void openOutputFile() {
try {
outputFile = new File("C:/mahi/Test", "file." + processId);
outputFileWriter = new FileWriter(outputFile);
} catch (IOException e) {
System.out.println("Exception while opening file");
stop = true;
}
}
private void writeOutputFile(String message) {
try {
outputFileWriter.write(message);
outputFileWriter.flush();
} catch (IOException e) {
System.out.println("Exception while writing to file");
stop = true;
}
}
}
How will I know when the ThreadPool has finished processing so I can do my other clean up work?
Thanks
If you Worker class implements Callable instead of Runnable, then you'd be able to see when your threads complete by using a Future object to see if the Thread has returned some result (e.g. boolean which would tell you whether it has finished execution or not).
Take a look in section "8. Futures and Callables" # website below, it has exactly what you need imo:
http://www.vogella.com/articles/JavaConcurrency/article.html
Edit: So after all of the Futures indicate that their respective Callable's execution is complete, its safe to assume your executor has finished execution and can be shutdown/terminated manually.
Something like this:
exec.shutdown();
// waiting for executors to finish their jobs
while (!exec.awaitTermination(50, TimeUnit.MILLISECONDS));
// perform clean up work
You can use a thread for monitoring ThreadPoolExecutor like that
import java.util.concurrent.ThreadPoolExecutor;
public class MyMonitorThread implements Runnable {
private ThreadPoolExecutor executor;
private int seconds;
private boolean run=true;
public MyMonitorThread(ThreadPoolExecutor executor, int delay)
{
this.executor = executor;
this.seconds=delay;
}
public void shutdown(){
this.run=false;
}
#Override
public void run()
{
while(run){
System.out.println(
String.format("[monitor] [%d/%d] Active: %d, Completed: %d, Task: %d, isShutdown: %s, isTerminated: %s",
this.executor.getPoolSize(),
this.executor.getCorePoolSize(),
this.executor.getActiveCount(),
this.executor.getCompletedTaskCount(),
this.executor.getTaskCount(),
this.executor.isShutdown(),
this.executor.isTerminated()));
try {
Thread.sleep(seconds*1000);
} catch (InterruptedException e) {
e.printStackTrace();
}
}
}
}
And add
MyMonitorThread monitor = new MyMonitorThread(executorPool, 3);
Thread monitorThread = new Thread(monitor);
monitorThread.start();
to your class where ThreadPoolExecutor is located.
It will show your threadpoolexecutors states in every 3 seconds.
I am trying to create a continuous thread where a server recieves/sends messages from a client however when I try to check for a next element it gets stuck:
public void run()
{
try
{
try
{
ArrayList<Socket> connections = parent.getConnections();
in = new Scanner(socket.getInputStream());
while(true)
{
if(in.hasNextLine()) // Gets stuck here
{
String message = in.nextLine();
System.out.println("Client said " + message);
}
}
}
finally
{
socket.close();
}
}
catch(Exception e)
{
e.printStackTrace();
}
How do I make the loop not get stuck at the specified point
Assuming you want to be able to deal with 'lines', I'd probably start with something like this:
public class SocketReader implements Runnable {
private final InputStream stream;
private final Queue<String> destination;
private volatile boolean active = true;
private SocketReader(InputStream stream, Queue<String> destination) {
this.stream = stream;
this.destination = destination;
}
public static SocketReader getReader(Socket toRead, Queue<String> destination) throws IOException {
return new SocketReader(toRead.getInputStream(), destination);
}
public void shutdown() {
active = false;
}
public void run() {
while(active) {
if (stream.hasNextLine() && active) {
final String line = stream.nextLine;
destination.add(line);
}
}
try {
stream.close();
} catch (IOException e) {
// Log somewhere
}
}
}
Drop this into its own thread (or as part of a thread or executor pool, really), and you've made the rest of your application non-blocking with regards to this code. EXPECT this to block while waiting for updates from stream.hasNextLine(). You can even supply a BlockingQueue if you don't wish to actively poll a queue, but are handling updates in some other fashion.
You can then do something like this for output:
public class QueuedPrinter implements Runnable {
private final Queue<String> input;
private final PrintStream destination;
private volatile boolean active;
public QueuedPrinter(Queue<String> input, PrintStream destination) {
this.input = input;
this.destination = destination;
}
public void shutdown() {
active = false;
}
public void run() {
while(active) {
final String line = input.poll();
if (line != null && active) {
destination.println(line);
}
}
}
}
Please note that I haven't tested this, and you may have to adjust things slightly for other Checked exceptions. You probably need to put in additional error-checking code (null-handling comes to mind). Also, this isn't completely threadsafe, but is likely to be 'good enough' for most uses.