We Keep Coding

Transactions in camel routes with Splitter parallel processing

We have an XML file which consists of multiple customer elements in it and we want to save customer information to the DB using transactions. From my understanding, transactions needs to be run in a single thread to roll back the whole transaction in case of errors.
Here's my XML:
<root>
<customers>
<customer>...</customer>
<customer>...</customer>
<customer>...</customer>
<customer>...</customer>
</customers
</root>
Here is my route:
<route id="routeA">
<from uri="direct-vm:sample" />
<transacted />
<splitter parallelProcessing = "true" stopOnException="true"
strategyRef="combine" />
<xpath>/root/customers/customer</xpath>
<bean ref="customerService" method="saveCustomer" />
<onException>java.lang.Exception</onException>
<handled><constant>true</constant></handled>
<rollback markRollbackOnly="true" />
</route>
The method saveCustomer() runs a lot of business logic before saving customers to the Database and if for some reason an exception is thrown for 1 or 2 customers, I see multiple rollback messages, and it seems like this is happening for each thread. Do transactions in camel routes with parallel processing work? Is there any other way to save customers in parallel to the DB in a single DB transaction?

No you cannot do parallel work in the same transaction. The work must occur on the same thread.

You can use shareUnitOfWork() in combination with your parallelProcessing.
As the Camel Documentation of the Splitter EIP mentions it will rollback the entire unit of work not only the sub units:
When the Splitter is done, it checks the state of the shared unit of work and checks if any errors occurred. And if an error occurred it will set the exception on the Exchange and mark it for rollback. The error handler will yet again kick in, as the Exchange has been marked as rollback and it had an exception as well. No redelivery attempts is performed (as it was marked for rollback) and the Exchange will be moved into the dead letter queue.
see Sharing Unit of work chapter in the link.

Server-side paging possible?

In a Java application, I am using Spring-Data to access a Neo4j database via the REST binding.
The spring.xml used as a context contains the following lines:
<neo4j:config graphDatabaseService="graphDatabaseService" />
<neo4j:repositories base-package="org.example.graph.repositories"/>
<bean id="graphDatabaseService"
class="org.springframework.data.neo4j.rest.SpringRestGraphDatabase">
<constructor-arg index="0" value="http://example.org:1234/db/data" />
</bean>
My repository is very simple:
public interface FooRepository extends GraphRepository<Foo> {
}
Now, I would like to loop through some Foos:
for (Foo foo : fooRepository.findAll(new PageRequest(0, 5))) //...
However, the performance of this request is awful: It takes over 400 seconds (!) to complete.
After a bit of debugging, I found out that Spring-data generates the following query:
START `foo`=node:__types__(className="org.example.Foo") RETURN `foo`
It then looks like as if paging is done on the client and all Foos (more than 100,000) are transferred to the client. When issuing the above query to the Neo4j server using the web interface, it takes around 60 seconds. However, if I manually append a "LIMIT 5", the execution time reduces to around 0.5 seconds.
What am I doing wrong so that spring-data does not use server-side, CYPHER pagination?
According to Programming Model
the expensive operations like traversals and querying are executed efficiently on the server side by using the REST API to forward those calls.
Or does this exclude the pagination?
What other options do I have in this case?

You can do the below to handle this server side.
Provide your own query method in the repository
The cypher query should use order, skip and limit and parameterize them so that you can pass in the skip and limit values on a per page basis.
E.g.
start john=node:users("name:pangea")
match john-[:HAS_SEEN]-(movie)
return movie
order by movie.name?
skip 20
limit 10

Spring JDBC Adapter in Cluster mode

I am using spring JDBC inbound channel adapter in my web application. If I deploy this application in clustered environment, two or more instances pickup the same job and run.
Can anybody help to overcome this issue by changing the spring configuration ?
I have attached my spring configuration.
<int-jdbc:inbound-channel-adapter
query=" SELECT JOBID,
JOBKEY,
JOBPARAM
FROM BATCHJOB
WHERE JOBSTATUS = 'A' "
max-rows-per-poll="1" channel="inboundAdhocJobTable" data-source="dataSource"
row-mapper="adhocJobMapper"
update=" delete from BATCHJOB where JOBKEY in (:jobKey)"
>
<int:poller fixed-rate="1000" >
<int:advice-chain>
</int:advice-chain>
</int:poller>
</int-jdbc:inbound-channel-adapter>

Unfortunately this will not be possible without some sort of syncing. Additionally using the database as some sort of message queue is not a good idea (http://mikehadlow.blogspot.de/2012/04/database-as-queue-anti-pattern.html). I'd try to follow different approaches:
Use some sort of message bus + message store to store the jobs objects rather than executing SQL directly. In this case you'll have to change the way jobs are being stored. Either by using some sort of message store backed channel (Spring integration only) or push them to a message queue like RabbitMQ to store these jobs.
I'm not 100% sure but remember that Spring Batch offers something similar like Master-Slave-Job splitting and synchronization. Maybe you have a look there.

Spring integration - Appropriate pattern for collating/batching service calls

I have a remote service that I'm calling to load pricing data for a product, when a specific event occurs. Once loaded, the product pricing is then broadcast for another consumer to process elsewhere.
Rather than call the remote service on every event, I'd like to batch the events into small groups, and send them in one go.
I've cobbled together the following pattern based on an Aggregator. Although it works, lots of it 'smells' -- especially my SimpleCollatingAggregator. I'm new to Spring Integration, and EIP in general, and suspect I'm misusing components.
The Code
My code is triggered elsewhere in code by calling a method on the below #Gateway:
public interface ProductPricingGateway {
#Gateway(requestChannel="product.pricing.outbound.requests")
public void broadcastPricing(ProductIdentifer productIdentifier);
}
This is then wired to an aggregator, as follows:
<int:channel id="product.pricing.outbound.requests" />
<int:channel id="product.pricing.outbound.requests.batch" />
<int:aggregator input-channel="product.pricing.outbound.requests"
output-channel="product.pricing.outbound.requests.batch" release-strategy="releaseStrategy"
ref="collatingAggregator" method="collate"
correlation-strategy-expression="0"
expire-groups-upon-completion="true"
send-partial-result-on-expiry="true"/>
<bean id="collatingAggregator" class="com.mangofactory.pricing.SimpleCollatingAggregator" />
<bean id="releaseStrategy" class="org.springframework.integration.aggregator.TimeoutCountSequenceSizeReleaseStrategy">
<!-- Release when: 10 Messages ... or ... -->
<constructor-arg index="0" value="10" />
<!-- ... 5 seconds since first request -->
<constructor-arg index="1" value="5000" />
</bean>
Here's the aggregator implementation:
public class SimpleCollatingAggregator {
public List<?> collate(List<?> input)
{
return input;
}
}
Finally, this gets consumed on the following #ServiceActivator:
#ServiceActivator(inputChannel="product.pricing.outbound.requests.batch")
public void fetchPricing(List<ProductIdentifer> identifiers)
{
// omitted
}
Note: In practice, I'm also using #Async, to keep the calling code as quick-to-return as possible. I have a bunch of questions about that too, which I'll move to a seperate question.
Question 1:
Given what I'm trying to acheive, is an aggregator pattern an appropriate choice here? This feels like a lot of boilerplate -- is there a better way?
Question 2:
I'm using a fixed collation value of 0, to effectively say : 'It doesn't matter how you group these messages, take 'em as they come.'
Is this an appropriate way of achieving this?
Question 3:
SimpleCollatingAggregator simply looks wrong to me.
I want this to receive my individual inbound ProductIdentifier objects, and group them into batches, and then pass them along. This works, but is it appropriate? Are there better ways of acheiving the same thing?

Q1: Yes, but see Q3 and the further discussion below.
Q2: That is the correct way to say 'no correlation needed' (but you need the expire-groups-on-completion, which you have).
Q3: In this case, you don't need a custom Aggregator, just use the default (remove the ref and method attributes).
Note that the aggregator is a passive component; the release is triggered by the arrival of a new message; hence the second part of your release strategy will only kick in when a new message arrives (it won't spontaneously release the group after 5 seconds).
However, you can configure a MessageGroupStoreReaper for that purpose: http://static.springsource.org/spring-integration/reference/html/messaging-routing-chapter.html#aggregator

Priority with activemq

We're currentyly developping an application using JMS and activemq (5.5.1).
We would like to define a higher priority for some messages, which would make them consumed first.
After setting the producer and the consumer (through spring (3.1) JMSTemplate), the priority does not fully work.
Indeed, when we "turn off" the consumer, and send some messages, the priority is respected, but when we add messages while the consumer is on, the messages are received in the same order they were sent.
The configuration is quite simple:
Priority was activated in the activemq config file:
<policyEntries>
<policyEntry queue=">" prioritizedMessages="true"/>
...
</policyEntries>
And QoS was enabled in the producer template configuration:
<bean id="jmsOCRTemplate" class="org.springframework.jms.core.JmsTemplate">
<property name="connectionFactory" ref="connectionFactory" />
<property name="defaultDestination" ref="destination_ocr" />
<property name="explicitQosEnabled" value="true" />
</bean>
To send a message with a high priority, we just change the template priority property on the producer side:
template.setPriority(9);
Any idea? Is this the normal behaviour, or is there some configuration we would have forgotten?

If my opinion you are not missing anything, I had a similar issue a couple of weeks ago (but with TTL and QPid).
First the JMS is not Strict about this :
JMS does not require that a provider strictly implement priority ordering of messages; however, it should do its best to deliver expedited messages ahead of normal messages.
Second, ActiveMQ does not YET implement priority queues, they say it will somewhere in 6.x version.
So, what you see is actually normal.
As a work-around you can use the Resequencer pattern if it fits your case.
http://camel.apache.org/resequencer.html
Here is another discussion on this subject:
http://activemq.2283324.n4.nabble.com/Priority-message-td2352179.html

I know it is late but this answers may help somebody.
If you want your consumer to consume message based on priority (Priority Queue) then you can use client side message priority. This means, when messages are being sent to your consumer (even before your consumer is receiving them, using prefetch), they will be cached on the consumer side and prioritized by default. This is regardless of whether you’re using priority support on the broker side. This could impact the ordering you see on the consumer so just keep this in mind.
To enable it, set the following configuration option on your broker URL, e.g.,
tcp://0.0.0.0:61616?jms.messagePrioritySupported=true
To disable it, tcp://0.0.0.0:61616?jms.messagePrioritySupported=false
So you do not require to use Camel (if you want to avoid complication)