I am trying to write a camel route that consumes incoming message from a JMS queue and handles them concurrently in different threads. The camel route I have got is like this:
<camel:endpoint id="requestQueue" uri="jms:queue.request" camelContextId="camel"/>
<camel:endpoint id="responseQueue" uri="jms:queue.response" camelContextId="camel"/>
<camel:camelContext id="camel">
<camel:threadPool id="serviceThreadPool" poolSize="10" threadName="workerThread" maxPoolSize="20"/>
<camel:route id="requestServingRoute">
<camel:from ref="requestQueue"/>
<camel:threads executorServiceRef="serviceThreadPool">
<camel:to uri="bean:doSomething"/>
<camel:to ref="responseQueue"/>
</camel:threads>
</camel:route>
</camel:camelContext>
However, what I can observe is that the incoming message is indeed handled by separate threads, but they are handled in a sequential order.
What I tried to achieve is that camel handles each request in the doSomething bean in separate threads for each incoming request.
How can I achieve this?
Thank you very much.
You need to specify the number of concurrent consumers for the JMS request queue. See http://camel.apache.org/jms.html. You don't need to message around with the thread pools. eg in Java DSL syntax you would write something like:
from("jms:queue.request?concurrentConsumers=10")
.beanRef("bean", "doSomething")
.to("jms:queue.response)
Related
I'm trying to implement the following JMS message flow using camel routes:
there is a topic published on external message broker. My program is listening for messages on this topic. Each incoming message triggers specific route to be executed - ONE TIME ONLY (some kind of ad-hoc, disposable route). This route is supposed to move messages between queues within my internal message broker based on some selector (get all messages from queue A matching given selector and move them to queue B). I'm only starting with camel and so far I figured out just the first part - listening for messages on topic:
<bean id="somebroker" class="org.apache.camel.component.jms.JmsComponent"
p:connectionFactory-ref="rmAdvisoriesConnectionFactory"/>
<camelContext id="camel" xmlns="http://camel.apache.org/schema/spring">
<endpoint id="jms" uri="somebroker:topic:sometopic"/>
<route id="routeAdvisories">
<from ref="jms"/>
<to>???</to>
</route>
</camelContext>
Can you suggest a destination for these advisory messages? I need to read some of their JMS properties and use these values to construct JMS selector that will be used for the "move messages" operation. But I have no idea how to declare and trigger this ad-hoc route. It would be ideal if I could define it within the same camelContext using only Spring DSL. Or alternatively, I could route advisories to some java method, which would be creating and executing this ad-hoc routes. But if this isn't possible, I'll be grateful for any suggestion.
Thanks.
As far as I understand, it will be useful to use the 'selector' option, in your JMS consumer route, for example:
from("activemq:queue:test?selector=key='value1'").to("mock:a");
from("activemq:queue:test?selector=key='value2'").to("mock:b");
Maybe, another option is to implement some routes based on 'Content Based Router Pattern" through "choice" option. You can find more info here: http://camel.apache.org/content-based-router.html
I hope it helps.
I couldn't get it working the way I intended, so I had to abandon my original approach. Instead of using camel routes to move messages between queues (now I'm not sure camel routes are even intended to be used this way) I ended up using ManagedRegionBroker - the way JMX operation "moveMatchingMessagesTo" is implemented - to move messages matching given selector.
I am bit confused about Camel routes and its two endpoints : Direct and Seda. Well let's say i have a route like this :
public void configure()
{
from("direct:services")
.process(//Some processing here)
.to("http://ThirdPartyServers")
}
On top of this I have a rest web service which receives several requests, does some processing and then hands over the message to this route to get response from some third party servers. I have instantiated Camel Context through Spring framework like this :
<camelContext id="appCamelContext" xmlns="http://camel.apache.org/schema/spring"
trace="true" streamCache="true">
<propertyPlaceholder id="properties"
location="classpath:camel.properties" />
<camel:routeBuilder ref="oneRouteBuilder" />
<camel:routeBuilder ref="photosRouteBuilder" />
</camelContext>
Now the question is that in a instant I send multiple different messages to this route. Now Camel documentation says that direct component is invoked in single thread and is synchronous. So will all the messages be processed concurrently or one by one processing will happen ?
Also if i change the direct component to seda, will it make any difference ?
TIA
Update [after Petter's answer]:
Although Petter's answer has clarified but i have new doubt regarding the same Direct and Seda components. Lets say my route is now like this :
public void configure(){
from("direct:services")
.choice()
.when("some predicate here-Predicate1")
.to("seda:predicate1")
.otherwise()
.to("seda:fallback")
.end();
from("seda:predicate1")
.process("some processing")
.to("http://ThirdPartyServers");
from("seda:fallback")
.process("some processing")
.to("jms:fallbackqueue");
}
Now if i send 5 messages to direct component from different threads, so these messages would be processed concurrently. As you can see in the above route, direct component sends the message to seda component. So now will there be only one thread of seda component which will process all the different 5 messages? Meaning in the end all the messages will be processed one by one ?
The direct component runs in the thread of the caller. Simplified, it's as a regular java method invocation. Multiple messages can run through the route as long as multiple threads are calling the direct endpoint.
When invoking the SEDA (or VM) endpoint, you message is put on a queue (in memory). Another thread (in the route) picks messages from the queue one by one and processes them. You can configure how many threads the seda consumer should have by setting the concurrentConsumers option. By default, the one thread consuming messages makes sure only one message at a time is processed no matter how many threads are producing to this route.
It boils down to your statement
in a instant I send multiple different messages to this route
If this means different threads, or just from one single thread in a sequence,
I have a Spring Integration project where I want to process a message concurrently through multiple actions. So I have set up a publish-subscribe-channel with a task-executor. However I want to wait for all processing to complete before moving on. How would I do this?
<publish-subscribe-channel id="myPubSub" task-executor="my10ThreadPool"/>
<channel id="myOutputChannel"/>
<service-activator input-channel="myPubSub" output-channel="myOutputChannel"
ref="beanA" method="blah"/>
<service-activator input-channel="myPubSub" output-channel="myOutputChannel"
ref="beanB" method="blah"/>
<service-activator id="afterThreadingProcessor" input-channel="myOutputChannel" .../>
So in the above case, I want my afterThreadingProcessor to be invoked only once after both beanA and beanB have completed their work. However, in the above afterThreadingProcessor will be invoked twice.
Add apply-sequence="true" to the pub-sub channel (this adds default correlation data to the messages, including correlationId, sequenceSize, and sequenceNumber and allows default strategies to be used on downstream components).
Add an <aggregator/> before afterThreadingProcessor and route the output from the two <service-activator/>s to it.
Add a <splitter/> after the aggregator - the default splitter will split the collection made by the aggregator into two messages.
afterThreadingProcessor will be invoked once for each message on the second thread that completes its work.
You can make the configuration easier by using a chain...
<chain input-channel="myOutputChannel">
<aggregator />
<splitter />
<service-activator id="afterThreadingProcessor" input-channel="myOutputChannel" .../>
</chain>
To make a single call to the final service, just change your service to take a Collection<?> instead of adding the splitter.
EDIT:
In order to do what you want in comment #3 (run the final service on the original thread), something this should work...
<int:channel id="foo" />
<int:service-activator ref="twoServicesGateway" input-channel="foo"
output-channel="myOutputChannel" />
<int:gateway id="twoServicesGateway" default-request-channel="myPubSub"/>
<int:publish-subscribe-channel id="myPubSub" task-executor="my10ThreadPool"
apply-sequence="true"/>
<int:service-activator input-channel="myPubSub" output-channel="aggregatorChannel"
ref="beanA" method="blah"/>
<int:service-activator input-channel="myPubSub" output-channel="aggregatorChannel"
ref="beanB" method="blah"/>
<int:aggregator input-channel="aggregatorChannel" />
<int:service-activator id="afterThreadingProcessor" input-channel="myOutputChannel" .../>
In this case, the gateway encapsulates the two other services and the aggregator; the default service-interface is a simple RequestReplyExchanger. The calling thread will wait for the output. Since the aggregator has no output-channel the framework will send the reply to the gateway, and the waiting thread will receive it, return to the <service-activator/> and the result will then be sent to the final service.
You would probably want to put a reply-timeout on the gateway because, by default, it will wait indefinitely and, if one of the services returns a null, no agreggated response will ever be received.
Note that I indented the gateway flow just to show it runs from the gateway, they are NOT child elements of the gateway.
Same kind of behavior can now be achieved using a more cleaner approach introduced in Spring Integration 4.1.0 as an implementation of EIP Scatter-Gather pattern.
Checkout Scatter-Gather example gist:
https://gist.github.com/noorulhaq/b13a19b9054941985109
We're currentyly developping an application using JMS and activemq (5.5.1).
We would like to define a higher priority for some messages, which would make them consumed first.
After setting the producer and the consumer (through spring (3.1) JMSTemplate), the priority does not fully work.
Indeed, when we "turn off" the consumer, and send some messages, the priority is respected, but when we add messages while the consumer is on, the messages are received in the same order they were sent.
The configuration is quite simple:
Priority was activated in the activemq config file:
<policyEntries>
<policyEntry queue=">" prioritizedMessages="true"/>
...
</policyEntries>
And QoS was enabled in the producer template configuration:
<bean id="jmsOCRTemplate" class="org.springframework.jms.core.JmsTemplate">
<property name="connectionFactory" ref="connectionFactory" />
<property name="defaultDestination" ref="destination_ocr" />
<property name="explicitQosEnabled" value="true" />
</bean>
To send a message with a high priority, we just change the template priority property on the producer side:
template.setPriority(9);
Any idea? Is this the normal behaviour, or is there some configuration we would have forgotten?
If my opinion you are not missing anything, I had a similar issue a couple of weeks ago (but with TTL and QPid).
First the JMS is not Strict about this :
JMS does not require that a provider strictly implement priority ordering of messages; however, it should do its best to deliver expedited messages ahead of normal messages.
Second, ActiveMQ does not YET implement priority queues, they say it will somewhere in 6.x version.
So, what you see is actually normal.
As a work-around you can use the Resequencer pattern if it fits your case.
http://camel.apache.org/resequencer.html
Here is another discussion on this subject:
http://activemq.2283324.n4.nabble.com/Priority-message-td2352179.html
I know it is late but this answers may help somebody.
If you want your consumer to consume message based on priority (Priority Queue) then you can use client side message priority. This means, when messages are being sent to your consumer (even before your consumer is receiving them, using prefetch), they will be cached on the consumer side and prioritized by default. This is regardless of whether you’re using priority support on the broker side. This could impact the ordering you see on the consumer so just keep this in mind.
To enable it, set the following configuration option on your broker URL, e.g.,
tcp://0.0.0.0:61616?jms.messagePrioritySupported=true
To disable it, tcp://0.0.0.0:61616?jms.messagePrioritySupported=false
So you do not require to use Camel (if you want to avoid complication)
I have this:
<si:poller max-messages-per-poll="10" id="defaultPoller" default="true">
<si:interval-trigger interval="5000"/>
</si:poller>
<si:channel id="emailIn"/>
<si:channel id="emailOut"/>
<si:service-activator input-channel="emailIn" output-channel="emailOut" ref="mailService" method="recieveMessage"/>
<si:gateway id="gateway" service-interface="com.blah.MailSender" default-request-channel="emailIn"/>
<si:outbound-channel-adapter channel="emailOut" ref="mailService" method="recieveMessage" />
And I thought what I was configuring was an async Queue. I want to be able to drop messages onto it, and have a nother thread pick them up and process then later. However, at the momment it seems to do it in a synchronous way.
Am i doing it wrong (obvioulsy yes), but wondering if there is something i'm missing in this config, or whether i just have the wrong approach?
Cheers
By default all channels in Spring Integration are synchronous. This is a conscious design decision that will help you keep transaction boundaries and security contexts for example. When you want to do asynchronous hand-off you should add a task executor to your dispatcher or a queue to your channel:
<channel>
<dispatcher task-executor="pool"/>
</channel>
<channel>
<queue capacity="10"/>
</channel>
Look at channel configurations in the reference guide for some details on dispatchers and queues. See also the section on DirectChannel and the section on ExecutorChannel below that one.