I am trying to implement a custom inbound channel adapter in spring integration to consume messages from apache kafka. Based on spring integration examples, I found that I need to create a class that implements MessageSource interface and implement receive() method that would return consumed Message from kafka. But based on consumer example in kafka, the message iterator in KafkaStream is backed by a BlockingQueue. So if no messages are in the queue, the thread will be blocked.
So what is the best way to implement a receive() method as this method can potentially block until there is something to consume.. ?
In more general sense, How do we implement a custom inbound channel for streaming message sources that blocks until there is something ready to consume..?
The receive() method can block (as long as the underlying operation responds properly to an interrupted thread), and from an inbound-channel-adapter perspective, depending on the expectations of the underlying source, it might be preferable to use a fixed-delay trigger. For example, "long polling" can simulate event-driven behavior when a very small delay value is provided.
We have a similar situation in our JMS polling MessageSource implementation. There, the underlying behavior is handled by one of the JmsTemplate's receive() methods. The JmsTemplate itself allows configuration of a timeout value. That means, as an example, you may choose to block for 5-seconds max but then have a very short delay trigger between each blocking receive call. Alternatively, you can specify an indefinite receive timeout. The decision ultimately depends on the expectations of the underlying resource, message throughput, etc.
Also, I wanted to let you know that we are exploring Kafka adapters ourselves. Perhaps you would like to collaborate on this within the spring-integration-extensions repository?
Regards,
Mark
Related
What is the elegant way of halting consumption of messages when an exception happens in the consumer or listener,so the messages can be re-queued. The listener process is consuming messages from the queue and calling a different API. Now if the API is not available, we don't want to consume messages from the queue. Is there any way to stop consuming messages from the queue for a finite time and come back up again when the API is available.
any sample code snippet of how it can be done also will help.
When asking questions like this, it's generally best to show a snippet of your configuration so we can answer appropriately, depending on how you are using the framework.
You can simply call stop() (and start()) on the listener container bean.
If you are using #RabbitListener, the containers are not beans, but are available via the RabbitListenerEndpointRegistry bean.
Calling stop() on the registry stops all the containers.
Or, you can call registry.getListenerContainer(id).stop(), where id is the value of the #RabbitListener 's id property.
Can I make concurrent calls using Spring JMSTemplate?
I want to make 4 external service calls in parallel and am exploring using Spring's JMSTemplate to perform these calls in parallel and wait for the execution to complete.
The other option that I am looking at is to use ExecutorService.
Is there any advantage using one over the other?
JMSTemplate is thread-safe, so making parallel calls to it is not a problem.
Messaging services are usually fast enough for most tasks and can receive your messages with minimal latency, so adding an ExecutorService doesn't seem as the first thing you usually need. What you really need is to correctly configure your JMS connections pool and give it enough open connections (four in your case) so it could handle your parallel requests with no blocking.
You only need ExecutorService in case you don't care about guaranteed delivery and your program needs extremely high speed that your messaging service cannot deliver, which is highly unlikely.
As for receiving replies from your external service, you need to use JMS Request/Reply pattern (you can find examples in this article). Happily, as you're using Spring, you could make Spring Integration do lots of work for you. You need to configure outbound-gateway to send messages and inbound-gateway to receive responses. Since version 2.2 you can also use reply-listener to simplify things on your client side. All these components are covered in the official documentation (with examples as well).
So need to talk to more than two JMS queues (send and or receive) parallel using asynchronous methods. Best option is usng #Asynch at method level
This example contains RestTemplate , But in your case create JmsTemplate beans.
Prerequisites:- Please create proper JMS Beans to connect to the queue. Proper use of this will help to invoke two queues paralleley. Its works for sure because already I have implemented. I just giving the skeleton due to Copyright issues.
More details: Spring Boot + Spring Asynch
https://spring.io/guides/gs/async-method/
Step1: Create a Service Class where JMS Queue
#EnableAsynch
public class JMSApplication {
#Autowired
JmsService jmsService;
public void invokeMe(){
// Start the clock
long start = System.currentTimeMillis();
// Kick of multiple, asynchronous lookups
Future<Object> queue1= jmsService.findqueue1();
Future<Object> queue2= jmsService.findqueue2();
// Wait until they are all done
while (!(queue1.isDone() && queue2.isDone())) {
Thread.sleep(10); //10-millisecond pause between each check
}
// Print results, including elapsed time
System.out.println("Elapsed time: " + (System.currentTimeMillis() - start));
System.out.println(queue1.get());
System.out.println(queue2.get());
}
}
Step2: Write the Service Method which will contain the business logic
for Jms
#Service
public Class JmsService{
#Asynch
public Object findqueue1(){
//Logic to invoke the JMS queue
}
#Asynch
public Object findqueue2(){
//Logic to invoke the JMS queue
}
}
I'm working with RabbitMQ and I'm confused using fanout exchange and convertSendAndReceive (or sendAndReceive) method of RabbitTemplate class.
For example, I have 2 consumers for durable queues QUEUE-01 and QUEUE-02 that are binded to durable fanout exchange FANOUT-01. And 1 publisher to FANOUT-01. I understand what happens when message is being published with convertAndSend (or send) method, message is going to be copied to each queue and will be processed by each consumer. But I'm not sure what will happen if I will call sendAndReceive method? Which consumer will I get a reply from? Is there any specific behaviour? I could not find any documentation on this.
The method sendAndReceive in the RabbitTemplate is used for when you would like to use RPC style messaging. There is an excellent tutorial here.
sendAndReceive() is not appropriate for fanout messaging; it is indeterminate as to which reply will win (the first one, generally). If you want to handle multiple replies and aggregate them you will need to use discrete send and receive calls (or listener container for the replies) and do the aggregation yourself.
Consider using Spring Integration for such situations. It has built-in components for aggregating messages.
If I have a rest service then, I know for sure that each request is treated by a separate thread and the threads can run in parallel.
What happens if I have a rest(http) service as inbound channel in spring integration. Will each request still be treated in parallel or the requests will be placed in queues... and it will be more like single threaded
Normal channels (DirectChannel) use the same execution thread as the object that put something into the channel (they are basically a way of abstracting a method call), so they are multi threaded.
From the docs:
In addition to being the simplest point-to-point channel option, one
of its most important features is that it enables a single thread to
perform the operations on "both sides" of the channel. For example, if
a handler is subscribed to a DirectChannel, then sending a Message to
that channel will trigger invocation of that handler's
handleMessage(Message) method directly in the sender's thread, before
the send() method invocation can return.
Edit
You have a very good point in your question. When you set a Queue element in a channel, spring automatically converts it to a QueueChannel (documentation), and as far as I can remember only one thread will be able to consume from the queue at at time. If you want "real" queue semantics (several producer and consumer threads) you can use an ExecutorChannel
When using Rest (http), the threading is managed by the servlet container; containers support multiple concurrent requests but setting the concurrency is done in the container, not Spring Integration.
With default Direct channels, the container threads will invoke the Spring Integration flow concurrently on the container threads.
I'm trying to write a multithreaded implementation for JMS message processing from a queue.
I've tried with DefaultMessageListenerContainer and SimpleMessageListenerContainer classes.
The problem I have is that it seems like just a single instance of the MessageListener class gets ever instantiated, no matter how I configure it. This forces me to unnecessarily write stateless or thread-safe MessageListener implementations, since I have the ListenerContainer configured to use multiple threads (concurrentConsumers=8).
Is there an obvious solution to this that I'm overlooking?
This is by design. The MessageListener is a dependency that you inject into Spring - it has no way of instantiating new ones.
This forces me to unnecessarily write stateless or thread-safe messageListener implementations
You make that sound like a bad thing. Making your MessageListener is a very good idea, Spring just removes the temptation to do otherwise.
Maybe this answer is too late, but it may benefit others who're searching for it. In short, the answer is using CommonsPoolTargetSource and ProxyFactoryBean.
Check out this link for details: http://forum.springsource.org/showthread.php?34595-MDB-vs-MDP-concurrency
If you want to do something similar for topic, check this: https://stackoverflow.com/a/12668538/266103
Configuring "concurrentConsumers" is just enough to process messages concurrently. This doesn't mean you will have "n" instances of MessageListenerContainer. The MessageListenerContainer may span "tasks" internally to process the messages. Optionally, you may have to configure your logging accordingly to see the information associated with the underlying tasks/threads.
See "Tuning JMS message consumption in Spring" for more details.