Working with Java 11 and Spring 2.1.6.RELEASE.
Im expierencing an issue where if I send a few records to the topic that this kafka consumer consumes from, everything works as planned. However If I produce A lot of records (a hundred or so) then the executor queues the processing but never actually does the processing. Am I using the executor wrong? I dont think its a kafka issue. Is there a way to query the executor to debug this?
#Configuration
public class ExecutorServiceConfig {
#Bean
public ExecutorService createExecutorService() {
return Executors.newFixedThreadPool(10);
}
}
#KafkaListener(topics = "${kafka.consumer.topic.name}",
groupId = "${spring.kafka.consumer.group-id}")
public void consume(PayrollDto message) {
log.info("Consumed message for processing:" + message); // this log is hit for all records
executor.execute(new ConsumerExecutor(message));
}
private class ConsumerExecutor implements Runnable {
PayrollDto message;
public ConsumerExecutor(PayrollDto message) {
this.message = message;
}
#Override
public void run() {
log.info("Beginning processing for payroll:" + this.message); // this log is hit for only some records
processPayrollList(this.message);
log.info("Finished processing for payroll:" + this.message);
}
}
It looks like you are using pure Java SE ExecutorService classes rather than Spring-specific TaskExecutor classes.
There is not enough information to diagnose this properly. (You haven't provide any clear evidence that the tasks have been "forgotten". Your reported evidence is that they are not executed. The "forgotten tasks" is only one of a number of possible explanations.)
The only explanations that I can think of are:
Your processPayrollList method is not terminating in some circumstances. It could be deadlocking, going into an infinite loop, waiting forever on some external service and so on.
If enough (i.e. 10) tasks failed to terminate, then you would run out of threads in the pool, and no more tasks would be processed. That is consistent with your evidence.
Something in your application is replacing executor with a different ExecutorService object.
Something in your application is removing tasks from the queue without executing them.
A build or deployment "process" issue; e.g. the code you are running is different to the code you are looking at. (It happens.)
An unreported bug in the Java 11 class library.
Of these, (1) is the most likely (IMO). Explanations (2) and (3) involve application code that I assume you would have mentioned in the question. I would treat (5) as implausible ... unless you can provide some clear evidence in the form of a minimal reproducible example.
Am I using the executor wrong?
It doesn't look like it from the code you have shown us.
Is there a way to query the executor to debug this?
You could take a thread stack dump (e.g. using the jstack command) and look at the status of the threads in the pool.
You could also cast executor to ThreadPoolExecutor and use that API to look at the queue length, the number of active threads and so on.
Note that this is not due to the ExecutorService being shut down. If that happened, you would get RejectedExecutionException in calls to execute.
Related
Some background
We are running a fairly simple application that handles subscriptions and are running into the limits of the external service. The solution is that we are introducing a queue and throttle the consumers of this queue to optimize the throughput.
For this we are using a Quarkus (2.7.5.Final) implementation and using quarkus-smallrye-reactive-messaging-rabbitmq connector provided by quarkus.io
Simplified implementation
rabbitmq-host=localhost
rabbitmq-port=5672
rabbitmq-username=guest
rabbitmq-password=guest
mp.messaging.incoming.subscriptions-in.connector=smallrye-rabbitmq
mp.messaging.incoming.subscriptions-in.queue.name=subscriptions
#Incoming("subscriptions-in")
public CompletionStage<Void> consume(Message<JsonObject> message) {
try {
Thread.sleep(1000);
return message.ack();
} catch (Exception e) {
return message.nack(e);
}
}
The problem
This only uses one worker thread and therefore the jobs are handles 1 by 1, ideally this application picks up as many jobs as there are worker threads available (in parallel), how can I make this work?
I tried
#Incoming("subscriptions-in")
#Blocking
Didn't change anything
#Incoming("subscriptions-in")
#NonBlocking
Didn't change anything
#Incoming("subscriptions-in")
#Blocking(ordered = false)
This made it split of into different worker threads, but ?detached? the job from the queue, so none of the messages got ack'd or nack'd
#Incoming("subscriptions-in-1")
..
#Incoming("subscriptions-in-2")
..
#Incoming("subscriptions-in-3")
These different channels seem to all work on the same worker thread (which is picked on startup)
The only way I currently see is to slim down the application and run one consumer thread each and just run 50 in parallel in kubernetes. This feels wrong and I can't believe there is no way to multithread at least some of the consuming.
Question
I am hopeful that I am missing a simple solution or am missing the concept of this RabbitMQ connector.
Is there anyway to get the #Incoming consumption to run in parallel?
Or is there a way in this Java implementation to increase the prefetch count? If so I can multithread them myself
I've been all over the internet and the Java docs regarding this one; I can't seem to figure out what it is about do while loops I'm not understanding. Here's the background: I have some message handler code that takes some JSON formatted data from a REST endpoint, parses it into a runnable task, then adds this task to a linked blocking queue for processing by the worker thread. Meanwhile, on the worker thread, I have this do while loop to process the message tasks:
do {
PublicTask currentTask = pubMsgQ.poll();
currentTask.run();
} while(pubMsgQ.size() > 0);
pubMsgQ is a LinkedBlockingQueue<PublicTask> (PublicTask implements the Runnable interface). I can't see any problems with this loop (obviously, or else I wouldn't be here), but this is how it behaves during execution: Upon entering the do block, pubMsgQ is polled and returns the runnable task as expected. The task is then run successfully with expected results, but then we get to the while statement. Now, according to the Java docs, poll() should return and remove the head of the queue, so I should expect that pubMsgQ.size() will return 0, right? Wrong I guess, because somehow the while statement passes and the program enters the do block again; of course this time pubMsgQ.poll() returns null (as I would have expected it should) and the program crashes with NullPointerException. What? Please explain like I'm five...
EDIT:
I decided to leave my original post as is above; because I think I actually explain the undesired behavior of that specific piece of the code quite succinctly (the loop is being executed twice while I'm fairly certain there is no way the loop should be executing twice). However, I realize that probably doesn't give enough context for that loop's existence and purpose in the first place, so here is the complete breakdown for what I am actually trying to accomplish with this code as I am sure there is a better way to implement this altogether anyways.
What this loop is actually a part of is a message handler class which implements the MessageHandler interface belonging to my Client Endpoint class [correction from my previous post; I had said the messages coming in were JSON formatted strings from a REST endpoint. This is technically not true: they are JSON formatted strings being received through a web socket connection. Note that while I am using the Spring framework, this is not a STOMP client; I am only using the built-in javax WebSocketContainer as this is more lightweight and easier for me to implement]. When a new message comes in onMessage() is called, which passes the JSON string to the MessageHandler; so here is the code for the entire MessageHandler class:
public class MessageHandler implements com.innotech.gofish.AutoBrokerClient.MessageHandler {
private LinkedBlockingQueue<PublicTask> pubMsgQ = new LinkedBlockingQueue<PublicTask>();
private LinkedBlockingQueue<AuthenticatedTask> authMsgQ = new LinkedBlockingQueue<AuthenticatedTask>();
private MessageLooper workerThread;
private CyclicBarrier latch = new CyclicBarrier(2);
private boolean running = false;
private final boolean authenticated;
public MessageHandler(boolean authenticated) {
this.authenticated = authenticated;
}
#Override
public void handleMessage(String msg) {
try {
//Create new Task and submit it to the message queue:
if(authenticated) {
AuthenticatedTask msgTsk = new AuthenticatedTask(msg);
authMsgQ.put(msgTsk);
} else {
PublicTask msgTsk = new PublicTask(msg);
pubMsgQ.put(msgTsk);
}
//Check status of worker thread:
if(!running) {
workerThread = new MessageLooper();
running = true;
workerThread.start();
} else if(running && !workerThread.active) {
latch.await();
latch.reset();
}
} catch(InterruptedException | BrokenBarrierException e) {
e.printStackTrace();
}
}
private class MessageLooper extends Thread {
boolean active = false;
public MessageLooper() {
}
#Override
public synchronized void run() {
while(running) {
active = true;
if(authenticated) {
do {
AuthenticatedTask currentTask = authMsgQ.poll();
currentTask.run();
if(GoFishApplication.halt) {
GoFishApplication.reset();
}
} while(authMsgQ.size() > 0);
} else {
do {
PublicTask currentTask = pubMsgQ.poll();
currentTask.run();
} while(pubMsgQ.size() > 0);
}
try {
active = false;
latch.await();
} catch (InterruptedException | BrokenBarrierException e) {
e.printStackTrace();
}
}
}
}
}
You may probably see where I'm going with this...what this Gerry-rigged code is trying to do is act as a facsimile for the Looper class provided by the Android Development Kit. The actual desired behavior is as messages are received, the handleMessage() method adds the messages to the queue for processing and the messages are processed on the worker thread separately as long as there are messages to process. If there are no more messages to process, the worker thread waits until it is notified by the handler that more messages have been received; at which point it resumes processing those messages until the queue is once again empty. Rinse and repeat until the user stops the program.
Of course, the closest thing the JDK provides to this is the ThreadPoolExecutor (which I know is probably the actual proper way to implement this); but for the life of me I couldn't figure out how to for this exact case. Finally, as a quick aside so I can be sure to explain everything fully, The reason why there are two queues (and a public and authenticated handler) is because there are two web socket connections. One is an authenticated channel for sending/receiving private messages; the other is un-authenticated and used only to send/receive public messages. There should be no interference, however, given that the authenticated status is final and set at construction; and each Client Endpoint is passed it's own Handler which is instantiated at the time of server connection.
You appear to have a number of concurrency / threading bugs in your code.
Assumptions:
It looks like there could be multiple MessageHandler objects, each with its own pair of queues and (supposedly) at most one MessageLooper thread. It also looks as if a given MessageHandler could be used by multiple request worker threads.
If that is the case, then one problem is that MessageHandler is not thread-safe. Specifically, the handleMessage is accessing and updating fields of the MessageHandler instance without doing any synchronization.
Some of the fields are initialized during object creation and then never changed. They are probably OK. (But you should declare them as final to be sure!) But some of the variables are supposed to change during operation, and they must be handled correctly.
One section that rings particular alarm bells is this:
if (!running) {
workerThread = new MessageLooper();
running = true;
workerThread.start();
} else if (running && !workerThread.active) {
latch.await();
latch.reset();
}
Since this is not synchronized, and the variables are not volatile:
There are race conditions if two threads call this code simultaneously; e.g. between testing running and assigning true to it.
If one thread sets running to true, there are no guarantees that a second thread will see the new value.
The net result is that you could potentially get two or more MessageLooper threads for a given set of queues. That breaks your assumptions in the MessageLooper code.
Looking at the MessageLooper code, I see that you have declared the run method as synchronized. Unfortunately, that doesn't help. The problem is that the run method will be synchronizing on this ... which is the specific instance of MessageLooper. And it will acquire the lock once and release it once. On short, the synchronized is wrong.
(For Java synchronized methods and synchronized blocks to work properly, 1) the threads involved need to synchronize on the same object (i.e. the same primitive lock), and 2) all read and write operations on the state guarded by the lock need to be done while holding the lock. This applies to use of Lock objects as well.)
So ...
There is no synchronization between a MessageLooper thread and any other threads that are adding to or removing from the queues.
There are no guarantees that the MessageLooper thread will notice changes to the running flag.
As I previously noted, you could have two or more MessageLooper polling the same pair of queues.
In short, there are lots of possible explanations for strange behavior in the code in the Question. This includes the specific problem you noticed with the queue size.
Writing correct multi-threaded code is difficult. This is why you should be using an ExecutorService rather than attempting to roll your own code.
But it you do need to roll your own concurrency code, I recommend buying and reading "Java: Concurrency in Practice" by Brian Goetz et al. It is still the only good textbook on this topic ...
My goal is to run multiple objects concurrently without creating new Thread due to scalability issues. One of the usage would be running a keep-alive Socket connection.
while (true) {
final Socket socket = serverSocket.accept();
final Thread thread = new Thread(new SessionHandler(socket)).start();
// this will become a problem when there are 1000 threads.
// I am looking for alternative to mimic the `start()` of Thread without creating new Thread for each SessionHandler object.
}
For brevity, I will use Printer anology.
What I've tried:
Use CompletableFuture, after checking, it use ForkJoinPool which is a thread pool.
What I think would work:
Actor model. Honestly, the concept is new to me today and I am still figuring out how to run an Object method without blocking the main thread.
main/java/SlowPrinter.java
public class SlowPrinter {
private static final Logger logger = LoggerFactory.getLogger(SlowPrinter.class);
void print(String message) {
try {
Thread.sleep(100);
} catch (InterruptedException ignored) {
}
logger.debug(message);
}
}
main/java/NeverEndingPrinter.java
public class NeverEndingPrinter implements Runnable {
private final SlowPrinter printer;
public NeverEndingPrinter(SlowPrinter printer) {
this.printer = printer;
}
#Override
public void run() {
while (true) {
printer.print(Thread.currentThread().getName());
}
}
}
test/java/NeverEndingPrinterTest.java
#Test
void withThread() {
SlowPrinter slowPrinter = new SlowPrinter();
NeverEndingPrinter neverEndingPrinter = new NeverEndingPrinter(slowPrinter);
Thread thread1 = new Thread(neverEndingPrinter);
Thread thread2 = new Thread(neverEndingPrinter);
thread1.start();
thread2.start();
try {
Thread.sleep(1000);
} catch (InterruptedException ignored) {
}
}
Currently, creating a new Thread is the only solution I know of. However, this became issue when there are 1000 of threads.
The solution that many developers in the past have come up with is the ThreadPool. It avoids the overhead of creating many threads by reusing the same limited set of threads.
It however requires that you split up your work in small parts and you have to link the small parts step by step to execute a flow of work that you would otherwise do in a single method on a separate thread. So that's what has resulted in the CompletableFuture.
The Actor model is a more fancy modelling technique to assign the separate steps in a flow, but they will again be executed on a limited number of threads, usually just 1 or 2 per actor.
For a very nice theoretical explanation of what problems are solved this way, see https://en.wikipedia.org/wiki/Staged_event-driven_architecture
If I look back at your original question, your problem is that you want to receive keep-alive messages from multiple sources, and don't want to use a separate thread for each source.
If you use blocking IO like while (socket.getInputStream().read() != -1) {}, you will always need a thread per connection, because that implementation will sleep the thread while waiting for data, so the thread cannot do anything else in the mean time.
Instead, you really should look into NIO. You would only need 1 selector and 1 thread where you continuously check the selector for incoming messages from any source (without blocking the thread), and use something like a HashMap to keep track of which source is still sending messages.
See also Java socket server without using threads
The NIO API is very low-level, BTW, so using a framework like Netty might be easier to get started.
You're looking for a ScheduledExecutorService.
Create an initial ScheduledExecutorService with a fixed appropriate number of threads, e.g. Executors.newScheduledThreadPool(5) for 5 threads, and then you can schedule a recurring task with e.g. service.scheduleAtFixedRate(task, initialDelay, delayPeriod, timeUnit).
Of course, this will use threads internally, but it doesn't have the problem of thousands of threads that you're concerned about.
I am using Apache Camel to connect to various endpoints, including JMS topics, and write to a database. Sometimes the database connection fails (for whatever reason, database issue, network blip, etc) and the messages from the topic subscriber start backing up. At a certain point, there are so many messages backed up waiting to be written to the database that the application throws an out of memory error. So far I understand all that.
The problem I have is the following: When the application is frantically trying to garbage collect before eventually giving up and accepting that it is out of memory, the application stops working, but is still alive. This means that the topic subscriber is still seen as active by the JMS provider, but not reading anything off the topic, so the provider starts queueing up the messages. Eventually the provider falls over also when the maximum depth runs out.
How can I configure my application to either disconnect when reaching a certain heap usage, or kill itself completely much much faster when running out of memory? I believe there are some JVM parameters that allow the application to kill itself much quicker when running out of memory, but I am wondering if that is the best solution or whether there is another way?
First of all I think you should use a JDBC connection pool that is capable of refreshing failed connections. So you do not run into the described scenario in the first place. At least not if the DB/network issue is short lived.
Next I'd protect the message broker by applying producer flow control (at least thats how it is called in ActiveMQ). I.e. prevent message producers from submitting more messages if a certain memory threshold has been breached. If the thresholds are set correctly, then that will prevent your message broker from falling over.
As for your original question: I'd use JMX to monitor the VM. If some metric, e.g. memory, breaches a threshold then you can suspend or shut down the route or the whole Camel context via the MBeans Camel exposes.
You can control (start/stop and suspend/resume) Camel routes using the Camel context methods .stop(), .start(), .suspend() and .resume().
You can spin a separate thread that monitors the current VM memory and stops the required route when a certain condition is met.
new Thread() {
#Override
public void run() {
while(true) {
long free = Runtime.getRuntime().freeMemory();
boolean routeRunning = camelContext.isRouteStarted("yourRoute");
if (free < threshold && routeRunning) {
camelContext.stopRoute("yourRoute");
} else if (free > threshold && !routeRunning) {
camelContext.startRoute("yourRoute");
}
// Check every 10 seconds
Thread.sleep(10000);
}
}
}
As commented in the other answer, relying on this is not particularly robust, but at least a little more robust than getting an OutOfMemoryException. Note that you need to .stop() the route, .suspend() does not deallocate resources, which means the connection with the queue provider is still open and the service looks like it is open for business.
You can also stop the route as part of the error handling of the route itself (this is possibly more robust but would require manual intervention to restart the route once the error is cleared, or a scheduled route that periodically checks if the error condition still exists and restart the route if it is gone). The thing to keep in mind is that you cannot stop a route from the same thread that is servicing the route at the time so you need to spin a separate thread that does the stopping. For example:
route("sample").from("jms://myqueue")
// Handle SQL Exceptions by shutting down the route
.onException(SQLException.class)
.process(new Processor() {
// This processor spawns a new thread that stops the current route
Thread stop;
#Override
public void process(final Exchange exchange) throws Exception {
if (stop == null) {
stop = new Thread() {
#Override
public void run() {
try {
// Stop the current route
exchange.getContext().stopRoute("sample");
} catch (Exception e) {}
}
};
}
// start the thread in background
stop.start();
}
})
.end()
// Standard route processors go here
.to(...);
I have a pipeline of tasks to be done on files, each different type of task runs inside a different executor service. After initilizing each executor service I start the first task, this is guaranteed to not finish until finished processing all files, as it processes a folder either no more work is required or its submits a callable task to service2. So when the shutdown() call on first task is sucessful all files will now be being processed in task2 or a another task further down the pipleline, and so on. When we can shutdown the final service then we have finished.
Loader loader = Loader.getInstanceOf();
List<ExecutorService> services = new ArrayList<ExecutorService>();
ExecutorService es = Executors.newSingleThreadExecutor();
//Init Services
services.add(es);
services.add(task1.getService());
services.add(task2.getService());
services.add(task3.getService());
services.add(task4.getService());
//Start Loading Files
es.submit(loader);
int count = 0;
for (ExecutorService service : services)
{
service.shutdown();
count++;
//Now wait for all submitted tasks to complete, for upto one day per task
service.awaitTermination(10, TimeUnit.DAYS);
MainWindow.logger.severe("Shutdown Task:" + count);
}
public class AnalyserService
{
protected String threadGroup;
public AnalyserService(String threadGroup)
{
this.threadGroup=threadGroup;
}
protected ExecutorService executorService;
protected CompletionService completionService;
protected void initExecutorService()
{
int workerSize = Runtime.getRuntime().availableProcessors();
executorService
= Executors.newFixedThreadPool(workerSize, new SongKongThreadFactory(threadGroup));
}
public ExecutorService getService()
{
if (executorService == null || executorService.isShutdown())
{
initExecutorService();
}
return executorService;
}
}
So this is all working fine Except Ive got my cpu load logic incorrect. Every service uses a pool equal to the number of cpus the computer has. So if computer has 4 cpus and we have 5 services then we could have 20 threads all trying to work at the same time overloading the cpus. I think I should in this case only have 4 threads at a time.
If I limited each service to use one thread then Id only have 5 threads runningat same time, but this still isnt right because
Will no longer be right if have more services or more cpus
Is inefficient, as the pipleline kicks of most of the work will be done by task1 , if I limit it to one cpu it will be slower than neccessary, conversly later on most of the threads will be done by later tasks and task1 will have nothing to do.
I think what I need is for all tasks to share one executor service, and set its poolsize equal to the number of cput the computer has. But then how am I going to identify when the service has finished ?
Im using Java 7, so is there anything in new in Java 7 that may help, currently just using Java 5 concurrency features
The core of your problem is: "[...] overloading the cpus."
If this is the problem, just schedule the priority of your application correctly. By the way, you are more likely to increase IO load than to increase CPU load; a lot of different threads is actually a good thing :-)
However, your question is: " But then how am I going to identify when the service has finished ? "
Very simple answer: submit() instead of invokeAll() and check the isDone() method of the Future object you receive.
http://docs.oracle.com/javase/1.5.0/docs/api/java/util/concurrent/ExecutorService.html#submit(java.util.concurrent.Callable)