We Keep Coding

Thread synchronization dont work unless add print statement

I am making a program that checks if a string is contained in a tree of directories and text files and I use producer-consumer pattern. Unfortunately my consumer thread doesn't want to stop unless I add a print statement. I tried everything - synchronization, making fields volatile but still can't find the problem.
public class Producer
extends Thread
{
private volatile Storage store;
private volatile Reader read;
Producer(Storage store, Reader read){
this.read = read;
this.store = store;
}
public void run()
{
while (!read.isEmpty()) {
String FileName = read.returnAllPaths().peek().getFileName().toString();
String item = null;
try {
item = read.returnAllPaths().take().toString();
} catch (InterruptedException e1) {
e1.printStackTrace();
}
File currentFile = new File(item);
try (BufferedReader reader = new BufferedReader(new FileReader(currentFile))) {
String line;
while ((line = reader.readLine()) != null) {
FileAndLine current = new FileAndLine(FileName, line);
store.fillStore(current);
}
} catch (IOException e) {
e.printStackTrace();
}
}
store.setEndOfPaths(true);
}
}
public class Consumer
extends Thread
{
private volatile Storage store;
private String clue;
public Consumer(Storage store, String clue){
this.store = store;
this.clue = clue;
}
public void run()
{
FileAndLine currentLine;
while(!store.isEndOfPaths() || !store.isEmpty()){
currentLine = store.depleteStore();
System.out.println("q");
if(currentLine.line.contains(clue))
System.out.println(currentLine.FileName + ": " + currentLine.line);
}
}
}
public class Storage {
private BlockingQueue<FileAndLine> Store;
private boolean full;
private volatile boolean endOfPaths;
public Storage(){
Store = new LinkedBlockingQueue<FileAndLine>();
full = false;
}
private boolean isFull(){
return full;
}
public synchronized BlockingQueue<FileAndLine> getStore(){
return this.Store;
}
public synchronized boolean isEmpty(){
return Store.isEmpty();
}
public synchronized void setEndOfPaths(boolean set){
endOfPaths = set;
}
public synchronized boolean isEndOfPaths(){
return endOfPaths;
}
public synchronized void fillStore(FileAndLine line){
while(isFull()){
try {
wait();
} catch (InterruptedException e) {
e.printStackTrace();
}
}
Store.add(line);
full = false;
notifyAll();
if(Store.size() == 1000){
full = true;
}
}
public synchronized FileAndLine depleteStore(){
FileAndLine line;
if(endOfPaths == true && Store.isEmpty())
{
return new FileAndLine("", "");
}
while(Store.isEmpty())
{
try {
wait();
} catch (InterruptedException e) {
e.printStackTrace();
}
}
line = new FileAndLine(Store.remove());
if(Store.size() < 1000){
full = false;
notifyAll();
}
return line;
}
}

When you do
private volatile Reader read;
This means the Reader read reference is volatile. This means when you read this field, it's access is volatile. However, it doesn't mean the object referenced is thread safe. The test you have is
while (!read.isEmpty()) {
and it is this test which has to be thread safe.
Note: when you write to the console, you are using a synchronized block indirectly as all the method of PrintStream is synchronized and this has both a read and write barrier.
I am making a program that checks if a string is contained in a tree of directories and text files
String text = "looking-for";
Files.walk(Paths.get("mydir"))
.parallel()
.filter(p -> p.toFile().isFile())
.forEach(p -> {
try {
if (Files.lines(p)
.anyMatch(l -> l.contains(text)) {
System.out.println("file " + p + " contains " + text);
});
} catch (IOException e) {
e.printStackTrace();
}
});
You don't need manage all the handling of dividing up the work to multiple threads and co-ordinating them, esp when you have a task which is data processing.

Tracking Executing Threads

I am trying to figure out how I can track all the threads that my application is spawning. Initially, I thought I had it figured out using a CyclicBarrier, however I am seeing threads executing after my await call.
Below is the working pseudo code:
public class ThreadTesterRunner {
public static void main(String[] args) throws InterruptedException {
final CyclicBarrier cb = new CyclicBarrier(1);
ThreadRunner tr = new ThreadRunner(cb);
Thread t = new Thread(tr, "Thread Runner");
t.start();
boolean process = true;
// wait until all threads process, then print reports
while (process){
if(tr.getIsFinished()){
System.out.println("Print metrics");
process = false;
}
Thread.sleep(1000);
}
}
}
class ThreadRunner implements Runnable {
static int timeOutTime = 2;
private ExecutorService executorService = Executors.newFixedThreadPool(10);
private final CyclicBarrier barrier;
private boolean isFinished=false;
public ThreadRunner(CyclicBarrier cb) {
this.barrier = cb;
}
public void run(){
try {
boolean stillLoop = true; int i = 0;
while (stillLoop){
int size;
Future<Integer> future = null;
try {
future = executorService.submit(new Reader()); // sleeps
size = future.get();
} catch (InterruptedException | ExecutionException ex) {
// handle Errs
}
if(i == 3){
stillLoop = false;
this.barrier.await();
this.isFinished=true;
}
//System.out.println("i = "+i+" Size is: "+size+"\r");
i++;
}
} catch (InterruptedException | BrokenBarrierException e1) {
e1.printStackTrace();
}
}
public boolean getIsFinished(){
return this.isFinished;
}
}
class Reader implements Callable {
private ExecutorService executorService = Executors.newFixedThreadPool(1);
#Override
public Object call() throws Exception {
System.out.println("Reading...");
Thread.sleep(2000);
executorService.submit(new Writer());
return 1000;
}
}
class Writer implements Callable {
#Override
public Void call() throws Exception {
Thread.sleep(4000);
System.out.println("Wrote");
return null;
}
}
Can anyone suggest a way to ONLY print "print metrics" after all threads have run?

It doesn't seem like you're doing anything to coordinate with your Reader and Writer threads, which are the ones you want to wait for. If you pass your synchronization barrier through to those threads so that they can register and signal when they are done, it works just fine.
Here's a version rewritten to do so, using a Phaser instead of a CyclicBarrier. Note that each Reader and Writer registers itself upon construction, and notifies the synchronization barrier when it is done executing:
public class ThreadTesterRunner {
public static void main(String[] args) throws InterruptedException {
final Phaser cb = new Phaser();
ThreadRunner tr = new ThreadRunner(cb);
Thread t = new Thread(tr, "Thread Runner");
t.start();
boolean process = true;
// wait until all threads process, then print reports
while (process){
if(tr.getIsFinished()){
System.out.println("Print metrics");
process = false;
}
//else {
// System.out.println("Waiting: registered=" + cb.getRegisteredParties() + ", arrived=" + cb.getArrivedParties() + ", unarrived=" + cb.getUnarrivedParties());
//}
Thread.sleep(1000);
}
}
}
class ThreadRunner implements Runnable {
static int timeOutTime = 2;
private ExecutorService executorService = Executors.newFixedThreadPool(10);
private final Phaser barrier;
private boolean isFinished=false;
public ThreadRunner(Phaser phaser) {
this.barrier = phaser;
}
public void run(){
try {
boolean stillLoop = true; int i = 0;
while (stillLoop){
int size;
Future<Integer> future = null;
try {
future = executorService.submit(new Reader(this.barrier)); // sleeps
size = future.get();
} catch (InterruptedException | ExecutionException ex) {
// handle Errs
}
if(i == 3){
stillLoop = false;
this.barrier.awaitAdvance(0);
this.isFinished=true;
}
//System.out.println("i = "+i+" Size is: "+size+"\r");
i++;
}
} catch (Exception e1) {
e1.printStackTrace();
}
}
public boolean getIsFinished(){
return this.isFinished;
}
}
class Reader implements Callable {
private Phaser barrier;
private ExecutorService executorService = Executors.newFixedThreadPool(1);
public Reader(Phaser phase) {
phase.register();
this.barrier = phase;
}
#Override
public Object call() throws Exception {
System.out.println("Reading...");
Thread.sleep(2000);
executorService.submit(new Writer(this.barrier));
this.barrier.arrive();
return 1000;
}
}
class Writer implements Callable {
private Phaser barrier;
public Writer(Phaser phase) {
phase.register();
this.barrier = phase;
}
#Override
public Void call() throws Exception {
Thread.sleep(4000);
System.out.println("Wrote");
this.barrier.arrive();
return null;
}
}

From what I can see you aren't waiting for the Writer to finish in the Reader. Is that the problem you are seeing?
You are also accessing isFinished from more than one thread without synchronization (which however, merely may delay the termination of the loop in this situation).
I don't see CyclicBarrier doing anything.
Not sure what you are trying to do, but I'd think about how simpler I can make it. For example, can Reader and Writer be combined into one task? Then, waiting for them to finish would merely be:
executorService.invokeAll(tasks);
System.out.println("Print metrics");
where tasks is a collection of tasks (see also this javadoc)

Text is not getting printed once the Threads are done [duplicate]

This question already has answers here:
How to wait for all threads to finish, using ExecutorService?
(27 answers)
Closed 8 years ago.
Please have a look at the following code.
public class BigFileWholeProcessor {
private static final int NUMBER_OF_THREADS = 2;
public void processFile(String fileName) {
BlockingQueue<String> fileContent = new LinkedBlockingQueue<String>();
BigFileReader bigFileReader = new BigFileReader(fileName, fileContent);
BigFileProcessor bigFileProcessor = new BigFileProcessor(fileContent);
ExecutorService es = Executors.newFixedThreadPool(NUMBER_OF_THREADS);
es.execute(bigFileReader);
es.execute(bigFileProcessor);
es.shutdown();
if(es.isTerminated())
{
System.out.println("Completed Work");
}
}
}
public class BigFileReader implements Runnable {
private final String fileName;
int a = 0;
public static final String SENTINEL = "SENTINEL";
private final BlockingQueue<String> linesRead;
public BigFileReader(String fileName, BlockingQueue<String> linesRead) {
this.fileName = fileName;
this.linesRead = linesRead;
}
#Override
public void run() {
try {
//since it is a sample, I avoid the manage of how many lines you have read
//and that stuff, but it should not be complicated to accomplish
BufferedReader br = new BufferedReader(new FileReader(new File("E:/Amazon HashFile/Hash.txt")));
String str = "";
while((str=br.readLine())!=null)
{
linesRead.put(str);
System.out.println(a);
a++;
}
linesRead.put(SENTINEL);
} catch (Exception ex) {
ex.printStackTrace();
}
System.out.println("Completed");
}
}
public class BigFileProcessor implements Runnable {
private final BlockingQueue<String> linesToProcess;
public BigFileProcessor (BlockingQueue<String> linesToProcess) {
this.linesToProcess = linesToProcess;
}
#Override
public void run() {
String line = "";
try {
while ( (line = linesToProcess.take()) != null) {
//do what you want/need to process this line...
if(line==BigFileReader.SENTINEL)
{
break;
}
String [] pieces = line.split("(...)/g");
}
} catch (InterruptedException e) {
e.printStackTrace();
}
}
}
I want to print the text "completed work" in BigFileWholeProcessor once all the thread work is done. But instead, it is not getting printed. Why is this? How to identify that all the threads are done and need printing?

shutdown() only signal ES to shutdown, you need
awaitTermination(long timeout, TimeUnit unit)
before print message

Use submit() method instead of execute(). The get() method can be used if you want to wait for the thread to finish at any point of time. Read documentation on use of Future object for further details.
ExecutorService es = Executors.newFixedThreadPool(2);
Future<?> f = es.submit(new Thread(new TestRun()));
f.get(); // Wait for result... (i.e similar to `join()` in this case)
es.shutdown(); // Shutdown ExecutorService
System.out.println("Done.");
I have defined a TestRun class implementing Runnable, not shown here. The Future object makes more sense in other scenarios.

How do we know threadPoolExecutor has finished execution

I have a parent thread that sends messages to MQ and it manages a ThreadPoolExecutor for worker threads which listen to MQ and writes message to output file. I manage a threadpool of size 5. So when I run my program, I have 5 files with messages. Everything works fine until here. I now need to merge these 5 files in my parent thread.
How do I know ThreadPoolExecutor finished processing so I can start merging files.
public class ParentThread {
private MessageSender messageSender;
private MessageReciever messageReciever;
private Queue jmsQueue;
private Queue jmsReplyQueue;
ExecutorService exec = Executors.newFixedThreadPool(5);
public void sendMessages() {
System.out.println("Sending");
File xmlFile = new File("c:/filename.txt");
List<String> lines = null;
try {
lines = FileUtils.readLines(xmlFile, null);
} catch (IOException e) {
e.printStackTrace();
}
for (String line : lines){
messageSender.sendMessage(line, this.jmsQueue, this.jmsReplyQueue);
}
int count = 0;
while (count < 5) {
messageSender.sendMessage("STOP", this.jmsQueue, this.jmsReplyQueue);
count++;
}
}
public void listenMessages() {
long finishDate = new Date().getTime();
for (int i = 0; i < 5; i++) {
Worker worker = new Worker(i, this.messageReciever, this.jmsReplyQueue);
exec.execute(worker);
}
exec.shutdown();
if(exec.isTerminated()){ //PROBLEM is HERE. Control Never gets here.
long currenttime = new Date().getTime() - finishDate;
System.out.println("time taken: "+currenttime);
mergeFiles();
}
}
}
This is my worker class
public class Worker implements Runnable {
private boolean stop = false;
private MessageReciever messageReciever;
private Queue jmsReplyQueue;
private int processId;
private int count = 0;
private String message;
private File outputFile;
private FileWriter outputFileWriter;
public Worker(int processId, MessageReciever messageReciever,
Queue jmsReplyQueue) {
this.processId = processId;
this.messageReciever = messageReciever;
this.jmsReplyQueue = jmsReplyQueue;
}
public void run() {
openOutputFile();
listenMessages();
}
private void listenMessages() {
while (!stop) {
String message = messageReciever.receiveMessage(null,this.jmsReplyQueue);
count++;
String s = "message: " + message + " Recieved by: "
+ processId + " Total recieved: " + count;
System.out.println(s);
writeOutputFile(s);
if (StringUtils.isNotEmpty(message) && message.equals("STOP")) {
stop = true;
}
}
}
private void openOutputFile() {
try {
outputFile = new File("C:/mahi/Test", "file." + processId);
outputFileWriter = new FileWriter(outputFile);
} catch (IOException e) {
System.out.println("Exception while opening file");
stop = true;
}
}
private void writeOutputFile(String message) {
try {
outputFileWriter.write(message);
outputFileWriter.flush();
} catch (IOException e) {
System.out.println("Exception while writing to file");
stop = true;
}
}
}
How will I know when the ThreadPool has finished processing so I can do my other clean up work?
Thanks

If you Worker class implements Callable instead of Runnable, then you'd be able to see when your threads complete by using a Future object to see if the Thread has returned some result (e.g. boolean which would tell you whether it has finished execution or not).
Take a look in section "8. Futures and Callables" # website below, it has exactly what you need imo:
http://www.vogella.com/articles/JavaConcurrency/article.html
Edit: So after all of the Futures indicate that their respective Callable's execution is complete, its safe to assume your executor has finished execution and can be shutdown/terminated manually.

Something like this:
exec.shutdown();
// waiting for executors to finish their jobs
while (!exec.awaitTermination(50, TimeUnit.MILLISECONDS));
// perform clean up work

You can use a thread for monitoring ThreadPoolExecutor like that
import java.util.concurrent.ThreadPoolExecutor;
public class MyMonitorThread implements Runnable {
private ThreadPoolExecutor executor;
private int seconds;
private boolean run=true;
public MyMonitorThread(ThreadPoolExecutor executor, int delay)
{
this.executor = executor;
this.seconds=delay;
}
public void shutdown(){
this.run=false;
}
#Override
public void run()
{
while(run){
System.out.println(
String.format("[monitor] [%d/%d] Active: %d, Completed: %d, Task: %d, isShutdown: %s, isTerminated: %s",
this.executor.getPoolSize(),
this.executor.getCorePoolSize(),
this.executor.getActiveCount(),
this.executor.getCompletedTaskCount(),
this.executor.getTaskCount(),
this.executor.isShutdown(),
this.executor.isTerminated()));
try {
Thread.sleep(seconds*1000);
} catch (InterruptedException e) {
e.printStackTrace();
}
}
}
}
And add
MyMonitorThread monitor = new MyMonitorThread(executorPool, 3);
Thread monitorThread = new Thread(monitor);
monitorThread.start();
to your class where ThreadPoolExecutor is located.
It will show your threadpoolexecutors states in every 3 seconds.

How to bypass scanner check next element (if that makes sense)

I am trying to create a continuous thread where a server recieves/sends messages from a client however when I try to check for a next element it gets stuck:
public void run()
{
try
{
try
{
ArrayList<Socket> connections = parent.getConnections();
in = new Scanner(socket.getInputStream());
while(true)
{
if(in.hasNextLine()) // Gets stuck here
{
String message = in.nextLine();
System.out.println("Client said " + message);
}
}
}
finally
{
socket.close();
}
}
catch(Exception e)
{
e.printStackTrace();
}
How do I make the loop not get stuck at the specified point

Assuming you want to be able to deal with 'lines', I'd probably start with something like this:
public class SocketReader implements Runnable {
private final InputStream stream;
private final Queue<String> destination;
private volatile boolean active = true;
private SocketReader(InputStream stream, Queue<String> destination) {
this.stream = stream;
this.destination = destination;
}
public static SocketReader getReader(Socket toRead, Queue<String> destination) throws IOException {
return new SocketReader(toRead.getInputStream(), destination);
}
public void shutdown() {
active = false;
}
public void run() {
while(active) {
if (stream.hasNextLine() && active) {
final String line = stream.nextLine;
destination.add(line);
}
}
try {
stream.close();
} catch (IOException e) {
// Log somewhere
}
}
}
Drop this into its own thread (or as part of a thread or executor pool, really), and you've made the rest of your application non-blocking with regards to this code. EXPECT this to block while waiting for updates from stream.hasNextLine(). You can even supply a BlockingQueue if you don't wish to actively poll a queue, but are handling updates in some other fashion.
You can then do something like this for output:
public class QueuedPrinter implements Runnable {
private final Queue<String> input;
private final PrintStream destination;
private volatile boolean active;
public QueuedPrinter(Queue<String> input, PrintStream destination) {
this.input = input;
this.destination = destination;
}
public void shutdown() {
active = false;
}
public void run() {
while(active) {
final String line = input.poll();
if (line != null && active) {
destination.println(line);
}
}
}
}
Please note that I haven't tested this, and you may have to adjust things slightly for other Checked exceptions. You probably need to put in additional error-checking code (null-handling comes to mind). Also, this isn't completely threadsafe, but is likely to be 'good enough' for most uses.