I have a Producer-Consumer problem to implement in Java, where I want the producer thread to run for a specific amount of time e.g. 1 day, putting objects in a BlockingQueue -specifically tweets, streamed from Twitter Streaming API via Twitter4j- and the consumer thread to consume these objects from the queue and write them to file. I've used the PC logic from Read the 30Million user id's one by one from the big file, where producer is the FileTask and consumer is the CPUTask (check first answer; my approach uses the same iterations/try-catch blocks with it). Of course I adapted the implementations accordingly.
My main function is:
public static void main(String[] args) {
....
final int threadCount = 2;
// BlockingQueue with a capacity of 200
BlockingQueue<Tweet> tweets = new ArrayBlockingQueue<>(200);
// create thread pool with given size
ExecutorService service = Executors.newFixedThreadPool(threadCount);
Future<?> f = service.submit(new GathererTask(tweets));
try {
f.get(1,TimeUnit.MINUTES); // Give specific time to the GathererTask
} catch (InterruptedException | ExecutionException | TimeoutException e) {
f.cancel(true); // Stop the Gatherer
}
try {
service.submit(new FileTask(tweets)).get(); // Wait til FileTask completes
} catch (InterruptedException | ExecutionException e) {
e.printStackTrace();
}
service.shutdownNow();
try {
service.awaitTermination(7, TimeUnit.DAYS);
} catch (InterruptedException e) {
e.printStackTrace();
}
}
Now, the problem is that, although it does stream the tweets and writes them to file, it never terminates and never gets to the f.cancel(true) part. What should I change for it to work properly? Also, could you explain in your answer what went wrong here with the thread logic, so I learn from my mistake? Thank you in advance.
These are the run() functions of my PC classes:
Producer:
#Override
public void run() {
StatusListener listener = new StatusListener(){
public void onStatus(Status status) {
try {
tweets.put(new Tweet(status.getText(),status.getCreatedAt(),status.getUser().getName(),status.getHashtagEntities()));
} catch (InterruptedException e) {
e.printStackTrace();
Thread.currentTread.interrupt(); // Also tried this command
}
}
public void onException(Exception ex) {
ex.printStackTrace();
}
};
twitterStream.addListener(listener);
... // More Twitter4j commands
}
Consumer:
public void run() {
Tweet tweet;
try(PrintWriter out = new PrintWriter(new BufferedWriter(new FileWriter("out.csv", true)))) {
while(true) {
try {
// block if the queue is empty
tweet = tweets.take();
writeTweetToFile(tweet,out);
} catch (InterruptedException ex) {
break; // GathererTask has completed
}
}
// poll() returns null if the queue is empty
while((tweet = tweets.poll()) != null) {
writeTweetToFile(tweet,out);
}
} catch (IOException e) {
e.printStackTrace();
}
}
You should check if your Thread classes are handling the InterruptedException, if not, they will wait forever. This might help.
Related
I'm learning gRPC with Java and as an example I've defined three request types (cuboid, sphere and cylinder) and a single response type (String) where I put message about the calculated volume of a specific geometry. I followed this example, which use blocking stub on the client side and the program runs correctly. However, I want to try the asynchronous approach, so here is my client side code, written with newFutureStub and ListenableFuture:
public static void main(String[] args) {
ManagedChannel channel = ManagedChannelBuilder
.forAddress("localhost",8080)
.usePlaintext()
.build();
GeometryServiceGrpc.GeometryServiceFutureStub stub = GeometryServiceGrpc.newFutureStub(channel);
ListenableFuture<Response> cuboidResp = stub.calcCuboidVol(CuboidVolumeRequest.newBuilder()
.setLength(2)
.setWidth(3)
.setHeight(4)
.build());
cuboidResp.addListener(() -> {
try {
System.out.println(cuboidResp.get().getResponse());
} catch (InterruptedException e) {
e.printStackTrace();
} catch (ExecutionException e) {
e.printStackTrace();
}
}, command -> {
command.run();
});
ListenableFuture<Response> sphereResp = stub.calcSphereVol(SphereVolumeRequest.newBuilder()
.setRadius(2)
.build());
sphereResp.addListener(() -> {
try {
System.out.println(sphereResp.get().getResponse());
} catch (InterruptedException e) {
e.printStackTrace();
} catch (ExecutionException e) {
e.printStackTrace();
}
}, command -> {
command.run();
});
ListenableFuture<Response> cylinderResp = stub.calcCylinderVol(CylinderVolumeRequest.newBuilder()
.setRadius(2)
.setHeight(3)
.build());
cylinderResp.addListener(() -> {
try {
System.out.println(cylinderResp.get().getResponse());
} catch (InterruptedException e) {
e.printStackTrace();
} catch (ExecutionException e) {
e.printStackTrace();
}
}, command -> {
command.run();
});
channel.shutdown();
}
Since we must use ListenableFuture I attach its method addListener() to each return instance of a specific geometry method (the Response message has a single string field called response). Basically, the desired result is just to print the calculations once they're ready and that's why a System.out.println is called in the runnable part. Although the program gets executed successfully, nothing is printed. But if check a specific geometry, then all results appear to the console. Example:
if(!cylinderResp.isDone()) {
try {
cylinderResp.get().getResponse();
} catch (InterruptedException e) {
e.printStackTrace();
} catch (ExecutionException e) {
e.printStackTrace();
}
}
Does this means that the program terminates before the asynchronous part is completed? And is the executable part (commnad) written correctly (I searched and found out that this part can be executed by a ExecutorService if we want the code to be run on different thread(s) - not a requirement in my case)?
Does this means that the program terminates before the asynchronous part is completed? Yes, that's right, you should wait it execute complete by call cylinderResp.get() or channel.awaitTermination(5, TimeUnit.SECONDS);
And is the executable part (commnad) written correctly, If you want run task in specify thread pool you should add a ExecutorService, or it will executed by default thread pool.
For completely asynchronous you should use StreamObserver.
For more detail of how to use gRPC in Java you can reference my code helloworlde/grpc-java-sample
I'm using the ReadData class from https://github.com/iota-community/java-iota-workshop/blob/master/src/main/java/com/iota/ReadData.java to retrieve a message from the Iota Tangle (essentially a distributed Network) via a hash value (the bundlehash).
That's my method:
private String readMessageFromHash(String BundleHash) {
final String[] s = new String[]{""};
Thread thread = new Thread(new Runnable() {
#Override
public void run() {
try {
s[0] = ReadData.getTMessage(BundleHash);
} catch (Exception e) {
e.printStackTrace();
}
}
});
thread.start();
try {
thread.join();
} catch (InterruptedException e) {
e.printStackTrace();
}
return s[0];
}
I need the return value in my next line of code but without multithreading my program crashes.
With mutlithreading it sometimes works, but most of time it doesn't work (returns an empty String).
I tried using:
thread.start();
try {
while(s[0].length < 1){}
thread.join();
} catch (InterruptedException e) {
e.printStackTrace();
}
return s[0];
but it just loops infinitely.
I think the issue is my program not waiting long enough for a response from the network.
I am trying to use Java's ExecutorService to send out concurrent REST requests which make various logs of system information (coming from a controller), however am running into a bug. About half of my requests successfully make it to the target, but the other half appear as if they are sent, but are not found on the server they were sent to. I think I may have a flaw in the logic of setting up my ExecutorService. The function log() shown below can be called from a REST call to the controller, and is supposed to create a new thread which sends out a separate HTTP request, and continue with the main thread so as not to wait for the network I/O. After much searching, I believe I have the ExecutorService shutdown properly an wait for the thread to complete. Can anybody see some type of error in the logic of my thread creation, as multiple requests from the controller can continue to come in?
//Controller
//code
#RequestMapping(value="/log", method= RequestMethod.GET)
public String log()
{
genomicsLogger.log(Severity.err, Category.LOG, "This is a log from the reporting manager!");
return "Hopefully logged";
}
//ClassB
public String log(String trns , String user, Severity severity, Category category, String msg) {
trnsField = trns;
userField = user;
...
...
...
ExecutorService executor = Executors.newFixedThreadPool(1);
Runnable task = () -> {
try {
System.out.println("Started thread: " + Thread.currentThread().getName());
restService.consumeRest(true, instance.getUri().toString(), LOG_URI, list, log, HttpMethod.POST, new HttpHeaders(), String.class);
System.out.println("SENT REST REQUEST");
} catch (URISyntaxException e) {
e.printStackTrace();
} catch (KeyStoreException e) {
e.printStackTrace();
} catch (IOException e) {
e.printStackTrace();
} catch (CertificateException e) {
e.printStackTrace();
} catch (NoSuchAlgorithmException e) {
e.printStackTrace();
} catch (UnrecoverableKeyException e) {
e.printStackTrace();
} catch (KeyManagementException e) {
e.printStackTrace();
}
}
};
executor.submit(task);
try {
System.out.println("attempt to shutdown executor");
executor.shutdown();
executor.awaitTermination(5, TimeUnit.SECONDS);
}
catch (InterruptedException e) {
System.err.println("tasks interrupted");
}
finally {
if (!executor.isTerminated()) {
System.err.println("cancel non-finished tasks");
}
executor.shutdownNow();
System.out.println("shutdown finished");
}
return "";
}
You are creating executor service with each log and killing it.
This is not the way to use it, it is meant to be reused, make it e.g. a field in this class and set some number of threads that you are willing to use for it (probably higher than 1). And don't do shutdown on it until you are really sure it won't be used (e.g. during application shutdown).
I have a thread which i wanted to run always until the JVM is stopped. What is the best way to do that ?
public void run() {
String event = sc.nextLine();
try {
queue.put(event); // thread will block here
} catch (InterruptedException e) {
e.printStackTrace();
}
}
Just adding an infinite loop should do the trick
public void run() {
while(true){
String event = sc.nextLine();
try {
queue.put(event); // thread will block here
} catch (InterruptedException e) {
e.printStackTrace();
}
}
}
while (true) { runBody(); }
Add exception handling if necessary.
I tried to use multiple threads, sadly no luck:
public synchronized boolean pingServer(final String ip, final short port) {
final boolean[] returnbol = new boolean[1];
Thread tt = new Thread(new Runnable() {
#Override
public void run() {
try {
Socket s = new Socket(ip, port);
s.close();
returnbol[0] = true;
} catch (IOException e) {
returnbol[0] = false;
}
}
});
tt.start();
try {
tt.join();
} catch (InterruptedException e) {
tt.stop();
}
tt.stop();
return returnbol[0];
}
The main thread still Freezes for some reason.
Is there a "lagless" way to ping a server?
What exactly did you want to got in
try {
tt.join();
} catch (InterruptedException e) {
tt.stop();
}
block?
Here you joined to parallel thread and waits till this thread will ends (got ping result).
You have next options:
Wait till ping ends
Don't wait... and don't got result
Use some concurrency classes like Future<> to got result (but you will block thread at moment you ask result if it not retrieved yet)
Or you can use some 'callback' function/interface to threw result from inner 'ping' thread.
You will need to remove the following lines from your code.
The tt.join() will force the main thread to wait for tt to finish.
try {
tt.join();
} catch (InterruptedException e) {
tt.stop();
}
tt.stop();
Use a Future instead to get the result for later use