We Keep Coding

Batch consumer camel kafka

I am unable to read in batch with the kafka camel consumer, despite following an example posted here. Are there changes I need to make to my producer, or is the problem most likely with my consumer configuration?
The application in question utilizes the kafka camel component to ingest messages from a rest endpoint, validate them, and place them on a topic. I then have a separate service that consumes them from the topic and persists them in a time-series database.
The messages were being produced and consumed one at a time, but the database expects the messages to be consumed and committed in batch for optimal performance. Without touching the producer, I tried adjusting the consumer to match the example in the answer to this question:
How to transactionally poll Kafka from Camel?
I wasn't sure how the messages would appear, so for now I'm just logging them:
from(kafkaReadingConsumerEndpoint).routeId("rawReadingsConsumer").process(exchange -> {
// simple approach to generating errors
String body = exchange.getIn().getBody(String.class);
if (body.startsWith("error")) {
throw new RuntimeException("can't handle the message");
}
log.info("BODY:{}", body);
}).process(kafkaOffsetManager);
But the messages still appear to be coming across one at a time with no batch read.
My consumer config is this:
kafka:
host: myhost
port: myport
consumer:
seekTo: beginning
maxPartitionFetchBytes: 55000
maxPollRecords: 50
consumerCount: 1
autoOffsetReset: earliest
autoCommitEnable: false
allowManualCommit: true
breakOnFirstError: true
Does my config need work, or are there changes I need to make to the producer to have this work correctly?

At the lowest layer, the KafkaConsumer#poll method is going to return an Iterator<ConsumerRecord>; there's no way around that.
I don't have in-depth experience with Camel, but in order to get a "batch" of records, you'll need some intermediate collection to "queue" the data that you want to eventually send downstream to some "collection consumer" process. Then you will need some "switch" processor that says "wait, process this batch" or "continue filling this batch".
As far as databases go, that process is exactly what Kafka Connect JDBC Sink does with batch.size config.

We solved a similar requirement by using the Aggregation [1] capability provided by Camel
A rough code snippet
#Override
public void configure() throws Exception {
// 1. Define your Aggregation Strat
AggregationStrategy agg = AggregationStrategies.flexible(String.class)
.accumulateInCollection(ArrayList.class)
.pick(body());
from("kafka:your-topic?and-other-params")
// 2. Define your Aggregation Strat Params
.aggregate(constant(true), agg)
.completionInterval(1000)
.completionSize(100)
.parallelProcessing(true)
// 3. Generate bulk insert statement
.process(exchange -> {
List<String> body = (List<String>) exchange.getIn().getBody();
String query = generateBulkInsertQueryStatement("target-table", body);
exchange.getMessage().setBody(query);
})
.to("jdbc:dataSource");
}
There are a variety of strategies that you can implement, but we chose this particular one because it allows you to create a List of strings for the message contents that we need to ingest into the db. [2]
We set a variety of different params such as completionInterval & completionSize. The most important one for us was to set parallellProcessing(true) [3] ; without that our performance wasn't nearly getting the required throughput.
Once the aggregation has either collected 100 messages or 1000 ms has passed, then the processor generates a bulk insert statement, which then gets sent to the db.
[1] https://camel.apache.org/components/3.18.x/eips/aggregate-eip.html
[2] https://camel.apache.org/components/3.18.x/eips/aggregate-eip.html#_aggregating_into_a_list
[3] https://camel.apache.org/components/3.18.x/eips/aggregate-eip.html#_worker_pools

Count number of successful processed messages on camel parallel split

I'm diggin' into camel, in order to process a lot of records in parallel. I have something like this:
from(CAMEL_START_ROUTE_CTE)
.multicast().parallelProcessing()
.to(CAMEL_PROCESS_DOMAINS_ROUTE)
.to(CAMEL_PROCESS_OTHERS_ROUTE)
.end()
.onCompletion()
.to(EndCamelRouteBuilder.CAMEL_ROUTE);
Where CAMEL_START_ROUTE_CTE is
timer:foo?delay=100&repeatCount=1
And then, CAMEL_PROCESS_DOMAINS_ROUTE looks like this:
from(CAMEL_PROCESS_DOMAINS_ROUTE)
.setHeader("domains").constant(config.getDomains())
.split(header("domains"))
.parallelProcessing()
.to(ProcessDomainCamelRoute.CAMEL_ROUTE)
.end()
.end();
Simplifying, domains is a json list.
What I'm trying to achieve is a way to count the number of messages and be able to read it in the route EndCamelRoute
I've tried with exchange.setProperty into the processor (using CamelSplitSize), setHeader, etc... but I always get a null when reading.
Does anybody know a way to achieve something like this? Some kind... of a reporting stuff (number of failed, successful messages), but consumed in a different route

I was fighting a similar issue while using split. Ended up having a very simple bean holding a hashmap where I store my counters.
I was looking into MicroMeter Component but it is consumer only.

Check file is processed through all routes of camel using aggregator

I have a camel application where, I am reading files from FTP source.
Then the file goes through multiple routes, like one route goes to cassandra for storage, one route process the data and push pivoted data to Kafka topic etc
I want to mark the file processed when it goes through all routes and reaches till the end. This way I can build a processing completed log based on file name.
One way, That I can think of is to implement aggregator, where each route will send completion notification in exchange header and then based on the completion criteria logic in aggregator, I will mark that file as processed.
How will I write such aggregator in java?

You could try using multicast.
from("direct:start")
.multicast()
.to("direct:a","direct:b")
.end()
// Won't run until the sub routes are complete
.process(new MarkFileAsCompletedProcessor())
.log("Finished multicast");
from("direct:a")
.log("Processing a")
.to("mock:endOfA");
from("direct:b")
.log("Processing b")
.to("mock:endOfB");

continue behavior in camel route execution

I want to put continue behaviour in route, my route is like following
from("file:D:\\?fileName=abc.csv&noop=true").split().unmarshal().csv()
.to("direct:insertToDb").end();
from("direct:insertToDb")
.to("direct:getDataId")
.to("direct:getDataParameters")
.to("direct:insertDataInDb");
from("direct:getDataId")
.to("sql:SELECT id FROM data WHERE name = :#name)
.choice()
.when(header("id").isGreaterThan(0) )
.setProperty("id", header("id"))
.otherwise()
.log("Error for")
.endChoice().end();
I want that if direct:getDataId dont find any record , my execution of route for current record from CSV get skip and program process next request. it would be equal to continue keyword.
How i can achieve this in Apache Camel route?

You can modify your routes like this:
from("file:D:\\?fileName=abc.csv&noop=true").split().unmarshal().csv()
.to("sql:SELECT id FROM data WHERE name = :#name?outputHeader=id&outputType=SelectOne)
.choice().when(header("id").isGreaterThan(0))
.to("direct:getDataParameters")
.to("direct:insertDataInDb")
.end();

Have you got a test for this? I suggest you try using CamelTestSupport because what you want is how camel will execute by default.
From Camel Split Docs:
stopOnException
default:false
description: Whether or not to stop continue processing immediately when an exception occurred. If disable, then Camel continue splitting and process the sub-messages regardless if one of them failed. You can deal with exceptions in the AggregationStrategy class where you have full control how to handle that.

Apache Camel multicast routes interfering with each other

I am using a multicast in Camel DSL because I need to send a copy of the same message to two different endpoints. However, it seems that the routes are interfering with each other. Have I got the syntax wrong, or some other issue?
from("{{in.endpoint}}")
.routeId(this.getClass().getSimpleName())
.multicast().parallelProcessing()
.to("{{update.in}}", "{{add.ibmmq.topic}});
where
in.endpoint = seda:addOrder?waitForTaskToComplete=Never
update.in = seda:updateData?waitForTaskToComplete=Never
add.ibmmq.topic = an ibmmq topic
I expect the 'update' route to receive the 'in' message, and the 'ibmmq topic' to receive the same message, presumably cloned. However, in the logs I am getting exceptions like:
Exchange[
Id ID-slon12d10628-1228-1386074869307-0-44746
ExchangePattern InOnly
Headers {breadcrumbId=ID-slon12d10628-1228-1386074869307-0-41682, calendar=null, CamelMyBatisResult=[integration.model.EInquiry#19eb77c, integration.model.EInquiry#12059ce, xxxxxxx
BodyType message.BulkAddOrderMsg
Body message.BulkAddBondOrderMsg#77df22
]
but the EInquiry objects are read in by a completely separate route, nothing to do with this route except it, too, sends messages to 'in.endpoint'.
The other thing is because I read from Tibco and send to IBMMQ, I have to clear the JMS header codes because they are not compatible, so I have put:
exchange.getIn().getHeaders().clear();
in my 'update' route. Could this be clearing Camel's exchange tracing headers and causing this issue, basically like some weird concurrency issue?

Its hard to find the error without full source code, but bear in mind that multicast does not do deep copy.
If you have child objects in the Order object they are not duplicated and they are shared between both SEDA routes.
Probably you will have to make a custom deep clone of the object

The body of your Exchange is a custom POJO: message.BulkAddBondOrderMsg#77df22... which means there is no deep cloning available unless you add it. Same thing would happen if the body were DOM XML node...
Serialize the POJO to a String prior to the multicast so it can be shared across Exchanges.