I have a stream of incoming data that is sent to RabbitMQ as individual messages.
I want to send these to a service that requires a batch of messages. I need to send the request to the service when I either have a batch of 1000 messages or when 5 seconds have expired. Is this possible using SimpleMessageListenerContainer?
The SimpleMessageListenerContainer supports transactions, however this won't help with the 5 second timeout. I did look at the method doReceiveAndExecute(BlockingQueueConsumer consumer) and "receiveTimeout", but as this variable is inside the transaction loop I could end up waiting 5 seconds per message (1000*5 sec= 83 min).
I currently have a channel aware listener that batches the messages into a bulk processor that will manage my timeouts and queue length. The SimpleMessageListenerContainer is set to manual ack. However as the listener returns before the message has actually been sent to the service I occasionally have issues when I do come to ack the message as the channel has been closed.
I have thought about writing my own ListenerContainer that sends the whole BlockingQueueConsumer to the Listener. Is this the only solution or has anyone managed to do something similar already?
You can use a ChannelAwareMessageListener, set acknowledgeMode=MANUAL; accumulate the deliveries in the listener; start a timer (scheduled task) to execute in +5 seconds and keep a reference to the channel. When a new delivery arrives, cancel the task, add the new delivery to the collection.
When 1000 deliveries arrive (or the scheduled task fires); invoke your service; then use channel.basicAck() (multiple) to ack the processed messages.
You'll have some race conditions to deal with but it should be pretty easy. Perhaps another queue of batches would be easiest with some other thread waiting for batches to arrive in that queue.
EDIT
As of 2.2, the SimpleMessageListenerContainer supports delivering batches of messages natively - see Batched Messages.
Starting with version 2.2, the SimpleMessageListeneContainer can be use to create batches on the consumer side (where the producer sent discrete messages).
Set the container property consumerBatchEnabled to enable this feature. deBatchingEnabled must also be true so that the container is responsible for processing batches of both types. Implement BatchMessageListener or ChannelAwareBatchMessageListener when consumerBatchEnabled is true. See #RabbitListener with Batching for information about using this feature with #RabbitListener.
Related
I have 2 #JmsListener instances on 1 queue, and I want to take a fixed number of messages from the queue and then hold the rest in pending for some time for bulk processing. I have added the condition to check the number of pending message, but due to 2 listeners it is failing. Also, I have to add this condition only inside #JmsListener.
Please suggest how to add the logic of taking fixed messages from queue and holding the rest in pending for achieving throttling.
I don't believe you will be able to use Spring's #JmsListener to do what you want because you simply don't have the control of the consumer which you need to fetch multiple messages and then process them all at once. A listener only gets one message at time and it is invoked as messages arrive so you have no control over when and how you fetch the messages in contrast to a normal JMS MessageConsumer which you can use to manually invoke receive() as many times as you like.
Also, ActiveMQ will do its best to treat each consumer fairly and therefore distribute the same amount of messages to each. Generally speaking, it is bad for one consumer to get all (or most) the messages as it can starve the other consumers and waste resources. That said, you could potentially use consumer priority if you really needed some consumers to get more messages than others.
I want 100 messages to be delivered together to a consumer through activemq, but at the same time producer will be producing messages one at a time.
Reason I want this is because I don't want to handle the overhead of processing each message individually on delivery, instead we want to do bulk processing on delivery.
Is it possible to achieve this through ActiveMQ or should i write my own modifications to achieve this.
ActiveMQ is a JMS 1.1 client / broker implementation therefore there is no API to deliver messages in bulk, the async listener dispatches them one at a time. The client does prefetch more than one message though so the overhead of processing them using async listeners is quite low.
You could achieve your goal by placing every message into a buffer and only doing your processing when the buffer contains N messages. To make it work, you'd want to use an acknowledgement mode such as CLIENT_ACKNOWLEDGE that allows you to not acknowledge the messages that are sitting in the buffer until they are processed; that way if your client crashed with some messages in its memory, they would be re-delivered when the client comes back up.
My JMS consumer produces any number of messages on a JMS queue during the day. As soon as a
message arrives it goes to message listener. If in between I need some other message comes, it goes to another message listener does not wait for first one?
As per my understanding here I need to create two consumer(assume i want to process 2 message concurrently) each having its own session. Both consumer can use the same message listener. Right?
I am not sure if I can achieve it with single consumer but can I with multiple listeners?
something like this Single queue: concurrent message processing with multiple consumers
Per the JMS documentation #bgth cites, multiple MessageListeners in a single session will not provide concurrency:
"The session used to create the message consumer serializes the execution of all message listeners registered with the session. At any time, only one of the session’s message listeners is running"
For concurrency, you need multiple sessions and multiple consumers in separate threads. You can reuse the same MessageListener in this case, but it must be threadsafe.
After reading some document of JMS, I totally puzzled by the phrase synchronous and asynchronouns.
See this page: http://docs.oracle.com/cd/E19798-01/821-1841/bncdq/index.html
Synchronous
You use the receive method to consume a message synchronously.
You can use this method at any time after you call the start method:
connection.start();
Message m = consumer.receive();
connection.start();
Message m = consumer.receive(1000); // time out after a second
To consume a message asynchronously, you use a message listener, described in the next section.
Asynchronous
JMS Message Listeners
A message listener is an object that acts as an asynchronous event handler for messages. This object implements the MessageListener interface, which contains one method, onMessage. In the onMessage method, you define the actions to be taken when a message arrives.
You register the message listener with a specific MessageConsumer by using the setMessageListener method. For example, if you define a class named Listener that implements the MessageListener interface, you can register the message listener as follows:
Listener myListener = new Listener();
consumer.setMessageListener(myListener);
I have two questions:
As what I understood, the nature of JMS is asynchronous. Producer publishes messages to the queue/topic, it doesn't need to wait consumer. This is asynchronous behaviour. How can it be "synchronous"?
If the "mesageListener" is asynchronous, but in my test with spring-jms, I found it always running in a thread. That means, if I write Thread.sleep(2000) in onMessage, it have to be wait 2 seconds before processing next message. Is it "asynchronous"?
If you understand it better like this, consumer.receive() uses a pull model: you read from a queue and are blocked waiting for this message until it comes, or some timeout has elapsed.
Using a listener uses a push model: you register a listener and, when a message comes in, the listener is called, in a separate thread.
Everything is done in a thread in Java, and the listener call is no exception. Whether the listener message handling prevents the processing of other messages in the queue depends on how many threads are dedicated to message processing. If you configure Spring to use a pool of 5 threads to process messages asynchronously, then 5 listeners will be able to process messages in parallel.
Like I understand this:
asynchronous - MessageListener: Use this on a server that listens to a queue. When a message arrives, then deal with it immediately. The server keeps listening to this queue.
synchronous - consumer.receive(1000): Use this on a client applications that now and then needs to check if a message is intend for this client. Example: poll every 60 seconds. This only opens a connection to the server shortly. The 1000 milliseconds will keep this connection open. If a message arrives within these 1000 milliseconds, then the message is consumed and the connection is closed.
You are looking at it end-to-end: from publisher to the consumer. Yes, it is asynchronous delivery from publisher to consumer irrespective of Sync/Async consumer. However Sync/Async in your question is for consumer only, i.e from the JMS broker (eg: ApacheMQ) to the consumer. As others have pointed out, Sync consumers pull messages sequentially from the broker and are waiting for messages. Async consumers register a callback where messages pushed to them (onMessage). Async consumers can go about doing other things while these messages are delivered to them asynchronously from the JMS broker.
I understand synchronous/asynchronous differently.
Synchronous: Caller(Sender) has to wait till the response from consumer has been received(till the time-out) -- request/reply pattern
Asynchronous: Caller(Sender) just post message and continue with its work, while the consumer processes as soon as the message reaches it -- one way request
Any MOM(Message Oriented Middle ware) follows service activator pattern which promotes asynchronous communication. One of my project has implemented a framework around JMS to make communication really synchronous.
Any message has 2 parts.
a. Metadata attributes
b. Payload
Set attribute "reply-to-queue" to a randomly generated value
Make sure the MOM framework creates temporary queue with name from #2
Make sure the sender spawns thread, which listens to temporary queue created in #3
Publish message and block sender till it receives message to temporary queue
Make sure the consumer intercepts "reply-to-queue" header and publishes response to it
This is one of the ways to make MOM based communication acts like synchronous. You may find other implementations like request-reply mechanism.
I want to use some mechanism to do something like this - I have a application server getting request to process orders. One request can have one order or two orders. To process the order the application will need to make a call to outside vendor, vendor can process at the most 5 orders at a time. I want to create a component that will batch maximum up to 5 orders if available and fire one request to the vendor and then when I get the response re-distribute the reponse to the appropriate request. The component would wait, say, 2 seconds before firing a batch request ones the first request arrives.
I thought of using JMS but in JMS a consumer will process the message (order) as soon as it arrives and a consumer processes one message at a time. I am thinking of usinga regular queue (ArrayBlockingQueue) and batch up the orders before firing using a ScheduledTimerTask. Does anybody have any ideas - Thanks in advance.
Most of what you describe can be done with Apache Camel. It can be configured to throttle, to batch, and forward request.
The Aggregator does what "max wait for X messages and then forward request"
There is nothing stopping you from using JMS. Just set a timeout on the operation that fetches from your queue, and when you have 5 messages or a timeout, you process the batch.