I have a Camel route that dequeues a message off a queue, sends it to a bean for processing, then enqueues the message back onto a different queue.
I am trying to eliminate "duplicate messages" on the 2nd queue. Does Camel have any endpoints, processors, EIPs, etc. that I could configure to dedupe message en route, before they get sent to the 2nd queue?
Example:
<route id="myRoute">
<from uri="{{queue-1-uri}}" />
<to uri="bean:myBean?method=process" />
<!-- How to dedupe right here??? -->
<to uri="{{queue-2-uri}}" />
</route>
Update: perhaps something like this:
<route id="myRoute">
<from uri="{{queue-1-uri}}" />
<to uri="bean:myBean?method=process" />
<filter>
<method>what goes here???</method>
<to uri="{{queue-2-uri}}" />
</filter>
</route>
Per Ralf's suggestion, I could then reference a bean inside <method></method> that then used a cache to keep messages in memory.
Say this new bean was called FilterBean and it had a dedupe() method on it: how do I wire it up in the Spring XML, and what classes/interfaces does the bean need to implement to be called from inside the route?
I think you are looking for the Idempotent Consumer that Camel provides. There are different approaches depending on your needs like Memory, JDBC, Hazelcast... I am not really sure if it will work for you since you are using a bean after the consumer but it worth to give it a try. Simple example from Camel website is as follows:
<!-- repository for the idempotent consumer -->
<bean id="myRepo" class="org.apache.camel.processor.idempotent.MemoryIdempotentRepository"/>
<camelContext xmlns="http://camel.apache.org/schema/spring">
<route>
<from uri="direct:start"/>
<idempotentConsumer messageIdRepositoryRef="myRepo">
<!-- use the messageId header as key for identifying duplicate messages -->
<header>messageId</header>
<!-- if not a duplicate send it to this mock endpoint -->
<to uri="mock:result"/>
</idempotentConsumer>
</route>
</camelContext>
You can find more info here: Idempotent Consumer
Related
I want to log the incoming SOAP in my beans setup but I don't know how to intercept the requested body.
I found out if I add ?dataFormat=MESSAGE to my CxfEndpoint it shows the xml input but messes up the dataFormat that provides the addBookTransformer.
<from uri="cxf:bean:CxfEndpoint?dataFormat=MESSAGE" />
This is my setup
<cxf:cxfEndpoint id="CxfEndpoint"
address="/host/addBook"
endpointName="a:addBookEndpoint"
serviceName="a:addBookService"
wsdlURL="wsdl/add-book.wsdl"
serviceClass="com.library.AddBookEndpoint"
xmlns:a="http://library.com"/>
<bean id="addBookTransformer" class="com.library.bookshelf.AddBookTransformer"/>
<camelContext xmlns="http://camel.apache.org/schema/spring">
<route id="addBook" streamCache="true">
<from uri="cxf:bean:CxfEndpoint" />
<process ref="addBookTransformer" />
<log message="${body}"/>
</route>
</camelContext>
Is there a way to itercept and log the incoming post request data?
You could do wiretap for this
<camelContext xmlns="http://camel.apache.org/schema/spring">
<route id="addBook" streamCache="true">
<from uri="cxf:bean:CxfEndpoint" />
<process ref="addBookTransformer" />
<wireTap uri="direct:tap"/>
<to uri="mock:result"/>
<log message="${body}"/>
</route>
</camelContext>
A copy of the exchange is sent to direct:tap which you can read and process however you want.
For example we could just log the message like this. You could also add another processor.
<route id="wiretapped" streamCache="true">
<from uri="direct:tap" />
<log message="${body}"/>
</route>
Here is the documentation for this.
What you are looking for is a very well known requirement. The pattern that provides the solution is a Filter and exists as a concept in the very early versions of J2EE spec. Back in times when we (the dinosaurs) wrote servlets, you could add a Filter to your servlet that intercepts both, request before it gets to your servlet and response after it is generated by your servlet but before it is sent to the client-side. In your Filter, you can log info, redirect it to a different destination, deny it or do anything you like. The same concept remains intact. You need to define a filter for all or some of your end-points and in your filter do the logging. If you work with Spring boot here is the article that describes how to add Filters for end-points in Spring Boot: How to Define a Spring Boot Filter?
A short question about Apache Camel.
I have the following scenario, where my server receives jms messages and then transform to csv file and then insert DB.
For this purpose i have 2 beans:
xml2csv
insertDB
I use routing like:
<route id="route1" errorHandlerRef="myErrorHandler">
<from uri="file://{someFolder1}}
?...
<to uri="bean:xml2csv" />
<log message="transformed to xml file" />
</route>
<route id="route2" errorHandlerRef="myErrorHandler">
<from uri="file://{{someFolder2}}
?...
<to uri="direct:csvOnboardingChannel" />
</route>
<route id="csvOnboarding" errorHandlerRef="myErrorHandler">
<from uri="direct:csvOnboardingChannel" />
<to uri="bean:insertDB" />
</route>
When "route" a file from-to, is it move like a message? or putting the question different, does Apache Camel take a file, wrap it as a message and route it to a bean or a component?
Do I understand it correct or am in a wrong directation.
Yes, your understanding is correct. Camel reads in the file's data and sends it as a message through the route to a bean. Might also be simpler as a single route, like so:
<route id="route1" errorHandlerRef="myErrorHandler">
<from uri="file://{someFolder1}}">
<to uri="bean:xml2csv" />
<to uri="bean:insertDB" />
</route>
I built a web service in apache camel running as a bundle on karaf which takes the requests and saves the information in a database.
After this another bundle takes this inserted record, modifies the data and saves it back to the db.
Now I need the response of the original request to contain the modified data so besides the route for the web service that looks like this
<route id="cxf">
<from uri="cxf:bean:getHopEndpoint" />
<recipientList>
<simple>direct:${header.operationName}</simple>
</recipientList>
</route>
<route id="getHop">
<from uri="direct:getHop" />
<process ref="getHopToDbProcessor" />
<to
uri="sql:INSERT INTO myTable (field1, field2) VALUES (:#field1, :#field2)"/>
</route>
I would need another one like this
<route id="cxfResponse">
<from uri="sql:SELECT * FROM myTable"/>
<!-- to web service response in any way -->
</route>
Is there any way to do this?
You would have to set the body to the desired response as the last step in the first route. There is no way to set the response outside this route.
So I think there are two way to solve your problem.
Do all you need to do inside the first route in a synchronous way
Change your service to be completely asynchronous
For Variant 2 you can either use messaging like jms or you give the first call a webservice uri to call back.
If you have high load on the service a completely async approach might make your system work better.
I found another solution: I'm using the Direct VM Component. My Routes look like this now:
Bundle 1 recieve web service request and save the data in the database:
<route id="cxf">
<from uri="cxf:bean:getHopEndpoint" />
<recipientList>
<simple>direct:${header.operationName}</simple>
</recipientList>
</route>
<route id="getHop">
<from uri="direct:getHop" />
<process ref="getHopToDbProcessor" />
<to uri="sql:INSERT INTO myTable (field1, field2) VALUES (:#field1, :#field2)" />
<to uri="direct-vm:processHop" />
</route>
In bundle 2 which processes the data I just inserted:
<route>
<from uri="direct-vm:processHop"/>
<to uri="sql:SELECT * FROM myTable WHERE processed = false" />
<process ref="getHopComputopUrlProcessor" />
<to uri="sql:UPDATE webshop_gethop_requests SET new_data = :#newData, processed = true WHERE some_id = :#someId" />
<to uri="direct-vm:response"/>
</route>
And then back to bundle 1 again to send the response with the processed data:
<route>
<from uri="direct-vm:response" />
<to uri="sql:SELECT some_id,new_data FROM myTable WHERE some_id = :#someId AND processed = true" />
<process ref="getHopResponseProcessor" />
</route>
This works just fine for me :)
I have been trying to come up with a camel route that would read from an activemq and write to Oracle AQ.
However, when a message is succesfully written to Oracle-aq, I have to write a successful message to another Active mq queue("something like message with id 41 has been sent to OracleAQ")
Is there any "Auto-acknowledge" type of feature in camel that can be useful here?
This is the basic route that i have that routes from active mq to oracle aq.
<route>
<from uri="jms:queue:Q.Customer1"/>
<setHeader headerName="prop">
<simple>header1Value</simple>
</setHeader>
<to uri="oracleQueue:queue:Q.Customer2"/>
</route>
Just add another to that points to the queue you want camel to post to after it writes to oracle, like so:
<onException>
<exception>some.sql.Exception</exception>
<to uri="some:error:handler:uri" /> <!-- like a bean or dead letter channel or whatever -->
</onException>
<route>
<from uri="jms:queue:Q.Customer1"/>
<setHeader headerName="prop">
<simple>header1Value</simple>
</setHeader>
<to uri="oracleQueue:queue:Q.Customer2"/>
<!-- this step will not be called until the previous one is finished -->
<to uri="jms:queue:SomeOtherQueue"/>
</route>
I've got my route's autoStart set to false:
<route id="myRoute" autoStartup="false">
Everything I can find online about how to start it after that is for starting it in java and says to call startRoute("myRoute"); on the camelContext.. but I can't find anything about how to call that from within a route in Spring XML.
Here's my setup:
<route id="myRoute" autoStartup="false">
<from uri="ftp://remote/dir" />
<to uri="file:///local/dir" />
</route>
<route id="kickOff">
<from uri="timer://runOnce?repeatCount=1&delay=30000" />
<!-- START myRoute HERE -->
<to uri="bean:postProcessor?method=postProcess" />
</route>
My goal is to have the FTP get all the files on the FTP once, then stop that route. Currently it'll continue polling the FTP indefinitely. I tried adding a org.apache.camel.impl.LimitedPollingConsumerPollStrategy with a limit of 1 but that didn't seem to change anything.
See the controlbus eip / component where you can start routes:
http://camel.apache.org/controlbus.html
The code should be something alike:
<route id="kickOff">
<from uri="timer://runOnce?repeatCount=1&delay=30000" />
<to uri="controlbus:route?routeId=myRoute&action=start" />
<to uri="bean:postProcessor?method=postProcess" />
</route>