We Keep Coding

Producer Consumer in Java using threads never terminates

I have a Producer-Consumer problem to implement in Java, where I want the producer thread to run for a specific amount of time e.g. 1 day, putting objects in a BlockingQueue -specifically tweets, streamed from Twitter Streaming API via Twitter4j- and the consumer thread to consume these objects from the queue and write them to file. I've used the PC logic from Read the 30Million user id's one by one from the big file, where producer is the FileTask and consumer is the CPUTask (check first answer; my approach uses the same iterations/try-catch blocks with it). Of course I adapted the implementations accordingly.
My main function is:
public static void main(String[] args) {
....
final int threadCount = 2;
// BlockingQueue with a capacity of 200
BlockingQueue<Tweet> tweets = new ArrayBlockingQueue<>(200);
// create thread pool with given size
ExecutorService service = Executors.newFixedThreadPool(threadCount);
Future<?> f = service.submit(new GathererTask(tweets));
try {
f.get(1,TimeUnit.MINUTES); // Give specific time to the GathererTask
} catch (InterruptedException | ExecutionException | TimeoutException e) {
f.cancel(true); // Stop the Gatherer
}
try {
service.submit(new FileTask(tweets)).get(); // Wait til FileTask completes
} catch (InterruptedException | ExecutionException e) {
e.printStackTrace();
}
service.shutdownNow();
try {
service.awaitTermination(7, TimeUnit.DAYS);
} catch (InterruptedException e) {
e.printStackTrace();
}
}
Now, the problem is that, although it does stream the tweets and writes them to file, it never terminates and never gets to the f.cancel(true) part. What should I change for it to work properly? Also, could you explain in your answer what went wrong here with the thread logic, so I learn from my mistake? Thank you in advance.
These are the run() functions of my PC classes:
Producer:
#Override
public void run() {
StatusListener listener = new StatusListener(){
public void onStatus(Status status) {
try {
tweets.put(new Tweet(status.getText(),status.getCreatedAt(),status.getUser().getName(),status.getHashtagEntities()));
} catch (InterruptedException e) {
e.printStackTrace();
Thread.currentTread.interrupt(); // Also tried this command
}
}
public void onException(Exception ex) {
ex.printStackTrace();
}
};
twitterStream.addListener(listener);
... // More Twitter4j commands
}
Consumer:
public void run() {
Tweet tweet;
try(PrintWriter out = new PrintWriter(new BufferedWriter(new FileWriter("out.csv", true)))) {
while(true) {
try {
// block if the queue is empty
tweet = tweets.take();
writeTweetToFile(tweet,out);
} catch (InterruptedException ex) {
break; // GathererTask has completed
}
}
// poll() returns null if the queue is empty
while((tweet = tweets.poll()) != null) {
writeTweetToFile(tweet,out);
}
} catch (IOException e) {
e.printStackTrace();
}
}

You should check if your Thread classes are handling the InterruptedException, if not, they will wait forever. This might help.

Ping a server without freezing the Thread

I tried to use multiple threads, sadly no luck:
public synchronized boolean pingServer(final String ip, final short port) {
final boolean[] returnbol = new boolean[1];
Thread tt = new Thread(new Runnable() {
#Override
public void run() {
try {
Socket s = new Socket(ip, port);
s.close();
returnbol[0] = true;
} catch (IOException e) {
returnbol[0] = false;
}
}
});
tt.start();
try {
tt.join();
} catch (InterruptedException e) {
tt.stop();
}
tt.stop();
return returnbol[0];
}
The main thread still Freezes for some reason.
Is there a "lagless" way to ping a server?

What exactly did you want to got in
try {
tt.join();
} catch (InterruptedException e) {
tt.stop();
}
block?
Here you joined to parallel thread and waits till this thread will ends (got ping result).
You have next options:
Wait till ping ends
Don't wait... and don't got result
Use some concurrency classes like Future<> to got result (but you will block thread at moment you ask result if it not retrieved yet)
Or you can use some 'callback' function/interface to threw result from inner 'ping' thread.

You will need to remove the following lines from your code.
The tt.join() will force the main thread to wait for tt to finish.
try {
tt.join();
} catch (InterruptedException e) {
tt.stop();
}
tt.stop();
Use a Future instead to get the result for later use

CompletionService - submit() does not block to ensure all threads are created

I am submitting Callables to an ExecutorCompletionService and it seems like the submit() method does not block code while submitting Callables. Here is the code that I have:
ExecutorService executor = Executors.newFixedThreadPool(30);
BlockingQueue<Future<Data>> completionQueue = new LinkedBlockingQueue();
ExecutorCompletionService<Data> completionService = new ExecutorCompletionService<Data>(executor, completionQueue);
while(receivingPackets) {
Callable<Data> splitPacketCallable = new SplitPacket(packetString);
completionService.submit(splitPacketCallable);
try {
// Allow submit to finish
TimeUnit.MILLISECONDS.sleep(50);
} catch (InterruptedException ex) {
System.out.println("Something went wrong with sleeping");
}
try {
Future<Data> dataFuture = completionService.poll();
if (dataFuture != null) {
Data data = dataFuture.get();
fileWriter.writeLine(data.toString());
}
} catch (InterruptedException ex) {
System.out.println("Error from poll: " + ex.toString());
} catch (ExecutionException ex) {
System.out.println("Error from get: " + ex.toString());
}
}
// Finish any remaining threads
while (!completionQueue.isEmpty()) {
try {
Future<Data> dataFuture = completionService.take();
Data data = dataFuture.get();
fileWriter.writeLine(data.toString());
} catch (InterruptedException ex) {
System.out.println("Error from take: " + ex.toString());
} catch (ExecutionException ex) {
System.out.println("Error from get: " + ex.toString());
}
}
fileWriter.close();
executor.shutdown();
A few things to note:
Data is a class that stores data in a special format. SplitPacket is a class that implements Callable that takes in a String that has arrived and splits it into chunks to be saved in Data. fileWriter and its method writeLine is a Runnable Class that will asynchronously write to a single file from multiple threads.
If I play with the sleep in the for loop, I start getting inconstant results in my output file. If I sleep for 50 ms every time I submit a Callable, everything works perfectly. However, if I submit with a low value (say 0-5 ms), I start getting dropped threads in the output. To me, this implies that the submit() method of ExecutorCompletionService does not block. However, because blocking a submitted callable seems vital, I also assume I am just implementing this wrong.
In my case, I don't know how many packets will be coming in so I need to be able to continuously add Callables to the Executor. I have tried this with a for loop instead of a while loop so that I can send a given number of packets and see if they get printed on the other end, and I can only get them to go through if I have a delay after submit.
Is there a way to fix this without adding a hack-y delay?

If you look at the source of ExecutorCompletionService you will see that the Futures are being added to completionQueue after the task is marked as done.
private class QueueingFuture extends FutureTask<Void> {
QueueingFuture(RunnableFuture<V> task) {
super(task, null);
this.task = task;
}
protected void done() { completionQueue.add(task); }
private final Future<V> task;
}
You may have an empty queue but still running tasks.
The simplest thing you can do is just count the tasks.
int count = 0;
while(receivingPackets) {
...
completionService.submit(splitPacketCallable);
++count;
...
try {
Future<Data> dataFuture = completionService.poll();
if (dataFuture != null) {
--count;
...
}
...
}
// Finish any remaining threads
while (count-- > 0) {
...
}

java: read and write on a file at the same time ,concurrency on file [duplicate]

I have read many a posts where-in they speak about reading and writing into the file NOT simultaneously using JavaME. I have a special use case scenarios where-in my log file (maybe full file or just portion of the file) is uploaded to the server on regular basis. This must continue without hampering the current logging of the application in this same file.
The code sample is a under:
boolean writing = true;
boolean reading = true;
void main() {
new Thread("THREAD-FILE-READ") {
public void run() {
InputStream instream = getFileInStream();
if (null != instream) {
while (reading) {
try {
try {
synchronized(READ_LOCK) {
READ_LOCK.wait();
}
} catch (InterruptedException ex) {
ex.printStackTrace();
}
if (writtenCharsLen > 0) {
byte[] bytes = new byte[writtenCharsLen];
instream.read(bytes, 0, writtenCharsLen);
System.out.println("Read="+new String(bytes));
bytes = null;
writtenCharsLen = 0;
}
} catch (IOException ioe) {
ioe.printStackTrace();
}
}
}
closeStream(instream);
}
}.start();
new Thread("THREAD-FILE-WRITE") {
public void run() {
OutputStream outstream = getFileOutStream();
if (null != outstream) {
while (writing) {
try {
byte[] str = randomString();
if (null != str) {
writtenCharsLen = str.length;
System.out.println("Write=" + new String(str));
outstream.write(str);
str = null;
}
} catch (IOException ex) {
ex.printStackTrace();
} finally {
notifyReadStream();
}
try {
synchronized(WRITE_LOCK) {
WRITE_LOCK.wait();
}
} catch (InterruptedException ex) {
ex.printStackTrace();
}
}
}
closeStream(outstream );
}
}.start();
}
void notifyReadStream() {
try {
synchronized (READ_LOCK) {
READ_LOCK.notify();
}
} catch (Exception e) {
e.printStackTrace();
}
}
void notifyWriteStream() {
try {
synchronized (WRITE_LOCK) {
WRITE_LOCK.notify();
}
} catch (Exception e) {
e.printStackTrace();
}
}
In the above code I will replace sop-read and sop-write with proper calls to network IO methods.
PS: Since this piece of code will run of multiple files and multitude of devices i need the modification as compressed as possible to keep my runtime heap as low as possible. Also this piece of code will run till the application life cycle hence closing and opening the file in middle is out of consideration.
Present Undesired Result:
The read and write threads are showing running sop's for read and write. The read thread is reading from the position the writing thread has written. I am not facing any exception in this code but the result is undesired. I have also tried synchronizing read and write streams but that is throwing IllegalMonitorStateException
Expected Result:
Reading of the stream must be triggered after writing into the stream is completed, also the read thread must be able to read from any position in the file.
Any help / pointers is useful?
EDIT: I was able to synchronize the read and the write streams using different monitors but i still feel, i could have done better using single monitor. Will try it sometime later.

I will attack this problem:
Present Undesired Result: The read and write threads are showing running sop's for read and write. The read thread is reading from the position the writing thread has written. I am not facing any exception in this code but the result is undesired. I have also tried synchronizing read and write streams but that is throwing IllegalMonitorStateException.
If you have synchronized the access using monitors i.e. the reader calls someObject.wait() and the writer calls someObject.notify(), remember that you have to wrap these calls in a synchronized block on someObject:
synchronized(someObject) {
someObject.wait();
}
synchronized(someObject) {
someObject.notify();
}
This is the cause for IllegalMonitorStateException.

Your first problem is that you are setting writtenCharsLen before you write the data. If your read thread sees it being non-zero before the write thread actually writes it, you have a problem. Move writtenCharsLen = str.length after the write.
Another problem I see in your sample is that the threads never yield control. They will be CPU hogs.

Java invoke thread inside it's run()

I wrote a thread class that checks the socket connection to the server by sending a small string every one second.
begin() method executes the thread.
After connection is lost, the thread tries to connect again.
My question is if it's ok to re-run by begin() the thread inside the run() method like I did (see below).
public void begin() {
Check = new Thread(this);
Check.start();
}
#Override
public void run() {
Thread thisThread = Thread.currentThread();
while (Check==thisThread) {
try {
oos.writeObject("a");
// oos.flush();
synchronized (this) {
while (pleaseWait) {
try {
System.out.println("waiting");
wait();
System.out.println("not waiting");
}
catch (Exception e) {
System.err.println("Thread is interrupted: "+e.getMessage());
}
}
}
sleep(1000);
} catch (Exception ex) {
v = new Visual("The connection is lost. The system will try to reconnect now.");
this.end();
try {
Server=ClientLogin.checkingServers(); //returns the reachable server string address
socket = new Socket(Server, ServerPort);
System.out.println("Connected: " + socket);
oos = new ObjectOutputStream(socket.getOutputStream());
begin();
v = new Visual("The system is reconnected.");
}
catch(UnknownHostException uhe){
System.out.println("Host unknown: " + uhe.getMessage());
v = new Visual("The system has failed to reconnected.");
}
catch (IOException ioe) {
System.out.println("The system cannot connect to servers: " + ioe.getMessage());
v = new Visual("The system has failed to reconnected.");
}
catch (Exception e) {
System.out.println("The system has failed to reconnect: " + e.getMessage());
v = new Visual("The system has failed to reconnected.");
}
}
}
}
public void end() {
Check = null;
}

I don't know any reason why that wouldn't work, but it looks kinda messy. You may have to declare Check as volatile to ensure that the loop always reads the current value, for those times when the new thread overwrites it.
IMHO a better approach would be a separate "supervisor" thread which is responsible for starting one of these threads, and then uses Thread.join() to wait for it to die, at which point it can start it up again.
In this way your main thread's logic can concentrate on exactly what it's supposed to do, without needing to have any "self awareness".

First, the code is not thread safe. The "Check" field is written by one thread but read by another, but it is not synchronised. There is no guarantee that the new started thread is going to see the updated value of "Check", i.e. the new thread will get the old thread's reference when checking "Check==thisThread" and do the wrong thing,
This particular problem can be fixed by making "Check" field volatile. It makes sure when it is updated, every thread will see the new value.
It is not "wrong" to call "begin()" in the run() method. However I wouldn't recommend it because you created a recursive call here effectively. There is a good chance you will get it wrong and fall into infinite loop. Try the simple design below. It uses a while loop instead of recursion.
package com.thinkinginobjects;
public class HeathChecker {
public void run() {
while (true) {
boolean success = checkHeath();
if (!success) {
//log and re-establish connection
} else {
Thread.sleep(1000);
}
}
}
private boolean checkHeath() {
try {
oos.writeObject("a");
return true;
} catch (Exception ex) {
return false;
}
}
}

it is ok, however why do you need to start a thread every time? Isn't it better to use Timer and TimerTask?
http://docs.oracle.com/javase/6/docs/api/java/util/TimerTask.html
http://docs.oracle.com/javase/6/docs/api/java/util/Timer.html