We're using Spring Cloud AWS to interact with SQS. We use the #SqsListener annotation to pull messages off our queues. We have deletionPolicy = NEVER, which means we manually acknowledge all messages we pick off.
Our problem is that the SimpleMessageListenerContainer (which handles the processing of messages from a queue) waits for all worker threads to finish before picking further messages off the queue.
In other words, what we're seeing is this:
Pull 10 messages off a queue.
Start 10 threads to do work.
One of the threads doing work gets blocked on a slow IO call.
The application is now blocked from fetching more messages off the queue, and is as such blocked from doing more work at all, until that slow call finishes.
We can see the code in SimpleMessageListenerContainer.AsynchronousMessageListener which is responsible for this
#Override
public void run() {
while (isQueueRunning()) {
try {
ReceiveMessageResult receiveMessageResult = getAmazonSqs().receiveMessage(this.queueAttributes.getReceiveMessageRequest());
CountDownLatch messageBatchLatch = new CountDownLatch(receiveMessageResult.getMessages().size());
for (Message message : receiveMessageResult.getMessages()) {
if (isQueueRunning()) {
MessageExecutor messageExecutor = new MessageExecutor(this.logicalQueueName, message, this.queueAttributes);
getTaskExecutor().execute(new SignalExecutingRunnable(messageBatchLatch, messageExecutor));
} else {
messageBatchLatch.countDown();
}
}
try {
messageBatchLatch.await();
} catch (InterruptedException e) {
Thread.currentThread().interrupt();
}
} catch (Exception e) {
getLogger().warn("An Exception occurred while polling queue '{}'. The failing operation will be " +
"retried in {} milliseconds", this.logicalQueueName, getBackOffTime(), e);
try {
//noinspection BusyWait
Thread.sleep(getBackOffTime());
} catch (InterruptedException ie) {
Thread.currentThread().interrupt();
}
}
}
}
Ideally, we'd like for the message listener to continually pick messages off the queue for processing.
We can't seem to implement our own MessageListenerContainer since the AbstractMessageListenerContainer is package local.
Is there any way around this behavior?
What is holding the message polling thread is the messageBatchLatch.await() statement. It seems that just removing the latch would do it. Something like:
#Override
public void run() {
while (isQueueRunning()) {
try {
ReceiveMessageResult receiveMessageResult = getAmazonSqs().receiveMessage(this.queueAttributes.getReceiveMessageRequest());
for (Message message : receiveMessageResult.getMessages()) {
if (isQueueRunning()) {
MessageExecutor messageExecutor = new MessageExecutor(this.logicalQueueName, message, this.queueAttributes);
getTaskExecutor().execute(new SignalExecutingRunnable(messageExecutor));
}
}
} catch (Exception e) {
getLogger().warn("An Exception occurred while polling queue '{}'. The failing operation will be " +
"retried in {} milliseconds", this.logicalQueueName, getBackOffTime(), e);
try {
//noinspection BusyWait
Thread.sleep(getBackOffTime());
} catch (InterruptedException ie) {
Thread.currentThread().interrupt();
}
}
}
}
This will work if your TaskExecutor implementation:
- Has a fixed-size thread pool
- Blocks when execute function gets called and no threads are available.
This is how most of implementations work but it is worth checking yours.
Related
In the below code snipped when dispose() is called, then the emitter thread is interrupted (InterruptedException is thrown out of sleep method).
Observable<Integer> obs = Observable.create(emitter -> {
for (int i = 0; i < 10; i++) {
if (emitter.isDisposed()) {
System.out.println("> exiting.");
emitter.onComplete();
return;
}
emitter.onNext(i);
System.out.println("> calculation = " + i);
try {
Thread.sleep(1000);
} catch (InterruptedException e) {
e.printStackTrace();
}
}
emitter.onComplete();
});
Disposable disposable = obs
.subscribeOn(Schedulers.computation())
.subscribe(System.out::println);
try {
Thread.sleep(2000);
} catch (InterruptedException e) {
e.printStackTrace();
}
disposable.dispose();
From debugging session I see that the interrupt origins from FutureTask which is cancelled during disposal. In there, the thread that is calling dispose() is checked against runner thread, and if it does not match, the emitter is interrupted. The thread is different since I used computation Scheduler.
Is there any way to make dispose not interrupt such emitter or is it how this actually should always be handled? An issue I see with this approach is when I would have an interruptible operation (simulated here by sleep) that I would want to complete normally before calling onComplete().
Please refer to What's different in 2.0 - Error handling.
One important design requirement for 2.x is that no Throwable errors should be swallowed. This means errors that can't be emitted because the downstream's lifecycle already reached its terminal state or the downstream cancelled a sequence which was about to emit an error.
So you can either wrap everything inside a try/catch and properly emit the error:
Observable<Integer> obs = Observable.create(emitter -> {
try {
// ...
} catch (InterruptedException ex) {
// check if the interrupt is due to cancellation
// if so, no need to signal the InterruptedException
if (!disposable.isDisposed()) {
observer.onError(ex);
}
}
});
or setup a global error consumer to ignore it:
RxJavaPlugins.setErrorHandler(e -> {
// ..
if (e instanceof InterruptedException) {
// fine, some blocking code was interrupted by a dispose call
return;
}
// ...
Log.warning("Undeliverable exception received, not sure what to do", e);
});
I have a simple Kafka consumer in Java. I'm trying to catch an exception if Kafka broker is not available. I need it to interrupt the thread.
I have code like this:
KafkaConsumer<String, String> kafkaConsumer = new KafkaConsumer<>(kafkaConsumerProperties());
kafkaConsumer.subscribe(Arrays.asList(topic));
try {
ConsumerRecords<String, String> records = kafkaConsumer.poll(500);
// records handling
} catch(Exception e) {
System.out.println(e.getMessage());
}finally {
kafkaConsumer.close();
}
If Kafka server is down I don't catch any exceptions but following messages are displayed in log:
18/03/28 13:33:39 WARN clients.NetworkClient: [Consumer clientId=consumer-3, groupId=JAVA] Connection to node -1 could not be established. Broker may not be available.
18/03/28 13:33:40 WARN clients.NetworkClient: [Consumer clientId=consumer-1, groupId=JAVA] Connection to node -1 could not be established. Broker may not be available.
Is there a way to handle it in my thread?
Something like this can work :
Runtime.getRuntime().addShutdownHook(new Thread() {
public void run() {
System.out.println("Starting exit...");
consumer.wakeup(); 1
try {
mainThread.join();
} catch (InterruptedException e) {
e.printStackTrace();
}
}
});
You do consumer.wakeup() to interrupt current consumer's operation.
The mainThread.join() is put in there to ensure that main thread actually finishes and is not shut down in middle of processing after wakeup. Please remember that shutdownHook is responsible for handling interrupts as well, not only ordinary program shutdown.
I've solved the problem somewhat clumsy, but it works for me.
public boolean testSocket(String serversList) {
String[] sockets = serversList.split(",");
int unactive = 0;
for (int i = 0; i < sockets.length; i++) {
try {
String[] socket = sockets[i].split(":");
(new Socket(socket[0], Integer.valueOf(socket[1]))).close();
} catch (IOException e) {
unactive++;
}
}
if (unactive < sockets.length) return true;
return false;
}
Recently, I find some BlockingOperationException in my netty4 project.
Some people said that when using the sync() method of start netty's ServerBootstrap can cause dead lock, because sync() will invoke await() method, and there is a method called 'checkDeadLock' in await().
But I don't think so. ServerBootstrap use the EventLoopGroup called boosGroup, and Channel use the workerGroup to operation IO, I don't think they will influence each other, they have different EventExecutor.
And in my practice, Deadlock exception doesn't appear in the Netty startup process, most of which occurs after the Channel of the await writeAndFlush.
Analysis source code, checkDeadLock, BlockingOperationException exception thrown is when the current thread and executor thread of execution is the same.
My project code is blow:
private void channelWrite(T message) {
boolean success = true;
boolean sent = true;
int timeout = 60;
try {
ChannelFuture cf = cxt.write(message);
cxt.flush();
if (sent) {
success = cf.await(timeout);
}
if (cf.isSuccess()) {
logger.debug("send success.");
}
Throwable cause = cf.cause();
if (cause != null) {
this.fireError(new PushException(cause));
}
} catch (LostConnectException e) {
this.fireError(new PushException(e));
} catch (Exception e) {
this.fireError(new PushException(e));
} catch (Throwable e) {
this.fireError(new PushException("Failed to send message“, e));
}
if (!success) {
this.fireError(new PushException("Failed to send message"));
}
}
I know Netty officials advise not to use sync() or await() method, but I want to know what situation will causes deadlocks in process and the current thread and executor thread of execution is the same.
I change my project code.
private void pushMessage0(T message) {
try {
ChannelFuture cf = cxt.writeAndFlush(message);
cf.addListener(new ChannelFutureListener() {
#Override
public void operationComplete(ChannelFuture future) throws PushException {
if (future.isSuccess()) {
logger.debug("send success.");
} else {
throw new PushException("Failed to send message.");
}
Throwable cause = future.cause();
if (cause != null) {
throw new PushException(cause);
}
}
});
} catch (LostConnectException e) {
this.fireError(new PushException(e));
} catch (Exception e) {
this.fireError(new PushException(e));
} catch (Throwable e) {
this.fireError(new PushException(e));
}
}
But I face a new problem, I can't get the pushException from the ChannelHandlerListener.
BlockingOperationException will be throw by netty if you call sync*or await* on a Future in the same thread that the EventExecutor is using and to which the Future is tied to. This is usually the EventLoop that is used by the Channel itself.
Can not call await in IO thread is understandable. However, there are 2 points.
1. If you call below code in channel handler, no exception will be reported, because the the most of the time the check of isDone in await returns true, since you are in IO thread, and IO thread is writing data synchronously. the data has been written when await is called.
ChannelPromise p = ctx.writeAndFlush(msg);
p.await()
If add a handler in different EventExecutorGroup, this check is not necessary, since that executor is newly created and is not the same one with the channel's IO executor.
I am trying to use Java's ExecutorService to send out concurrent REST requests which make various logs of system information (coming from a controller), however am running into a bug. About half of my requests successfully make it to the target, but the other half appear as if they are sent, but are not found on the server they were sent to. I think I may have a flaw in the logic of setting up my ExecutorService. The function log() shown below can be called from a REST call to the controller, and is supposed to create a new thread which sends out a separate HTTP request, and continue with the main thread so as not to wait for the network I/O. After much searching, I believe I have the ExecutorService shutdown properly an wait for the thread to complete. Can anybody see some type of error in the logic of my thread creation, as multiple requests from the controller can continue to come in?
//Controller
//code
#RequestMapping(value="/log", method= RequestMethod.GET)
public String log()
{
genomicsLogger.log(Severity.err, Category.LOG, "This is a log from the reporting manager!");
return "Hopefully logged";
}
//ClassB
public String log(String trns , String user, Severity severity, Category category, String msg) {
trnsField = trns;
userField = user;
...
...
...
ExecutorService executor = Executors.newFixedThreadPool(1);
Runnable task = () -> {
try {
System.out.println("Started thread: " + Thread.currentThread().getName());
restService.consumeRest(true, instance.getUri().toString(), LOG_URI, list, log, HttpMethod.POST, new HttpHeaders(), String.class);
System.out.println("SENT REST REQUEST");
} catch (URISyntaxException e) {
e.printStackTrace();
} catch (KeyStoreException e) {
e.printStackTrace();
} catch (IOException e) {
e.printStackTrace();
} catch (CertificateException e) {
e.printStackTrace();
} catch (NoSuchAlgorithmException e) {
e.printStackTrace();
} catch (UnrecoverableKeyException e) {
e.printStackTrace();
} catch (KeyManagementException e) {
e.printStackTrace();
}
}
};
executor.submit(task);
try {
System.out.println("attempt to shutdown executor");
executor.shutdown();
executor.awaitTermination(5, TimeUnit.SECONDS);
}
catch (InterruptedException e) {
System.err.println("tasks interrupted");
}
finally {
if (!executor.isTerminated()) {
System.err.println("cancel non-finished tasks");
}
executor.shutdownNow();
System.out.println("shutdown finished");
}
return "";
}
You are creating executor service with each log and killing it.
This is not the way to use it, it is meant to be reused, make it e.g. a field in this class and set some number of threads that you are willing to use for it (probably higher than 1). And don't do shutdown on it until you are really sure it won't be used (e.g. during application shutdown).
I have a Producer-Consumer problem to implement in Java, where I want the producer thread to run for a specific amount of time e.g. 1 day, putting objects in a BlockingQueue -specifically tweets, streamed from Twitter Streaming API via Twitter4j- and the consumer thread to consume these objects from the queue and write them to file. I've used the PC logic from Read the 30Million user id's one by one from the big file, where producer is the FileTask and consumer is the CPUTask (check first answer; my approach uses the same iterations/try-catch blocks with it). Of course I adapted the implementations accordingly.
My main function is:
public static void main(String[] args) {
....
final int threadCount = 2;
// BlockingQueue with a capacity of 200
BlockingQueue<Tweet> tweets = new ArrayBlockingQueue<>(200);
// create thread pool with given size
ExecutorService service = Executors.newFixedThreadPool(threadCount);
Future<?> f = service.submit(new GathererTask(tweets));
try {
f.get(1,TimeUnit.MINUTES); // Give specific time to the GathererTask
} catch (InterruptedException | ExecutionException | TimeoutException e) {
f.cancel(true); // Stop the Gatherer
}
try {
service.submit(new FileTask(tweets)).get(); // Wait til FileTask completes
} catch (InterruptedException | ExecutionException e) {
e.printStackTrace();
}
service.shutdownNow();
try {
service.awaitTermination(7, TimeUnit.DAYS);
} catch (InterruptedException e) {
e.printStackTrace();
}
}
Now, the problem is that, although it does stream the tweets and writes them to file, it never terminates and never gets to the f.cancel(true) part. What should I change for it to work properly? Also, could you explain in your answer what went wrong here with the thread logic, so I learn from my mistake? Thank you in advance.
These are the run() functions of my PC classes:
Producer:
#Override
public void run() {
StatusListener listener = new StatusListener(){
public void onStatus(Status status) {
try {
tweets.put(new Tweet(status.getText(),status.getCreatedAt(),status.getUser().getName(),status.getHashtagEntities()));
} catch (InterruptedException e) {
e.printStackTrace();
Thread.currentTread.interrupt(); // Also tried this command
}
}
public void onException(Exception ex) {
ex.printStackTrace();
}
};
twitterStream.addListener(listener);
... // More Twitter4j commands
}
Consumer:
public void run() {
Tweet tweet;
try(PrintWriter out = new PrintWriter(new BufferedWriter(new FileWriter("out.csv", true)))) {
while(true) {
try {
// block if the queue is empty
tweet = tweets.take();
writeTweetToFile(tweet,out);
} catch (InterruptedException ex) {
break; // GathererTask has completed
}
}
// poll() returns null if the queue is empty
while((tweet = tweets.poll()) != null) {
writeTweetToFile(tweet,out);
}
} catch (IOException e) {
e.printStackTrace();
}
}
You should check if your Thread classes are handling the InterruptedException, if not, they will wait forever. This might help.