We Keep Coding

Apache Camel - Kafka component - single producer multiple consumer

I am creating two apache camel (blueprint XML) kafka projects, one is kafka-producer which accepts requests and stores it in kafka server, and other is kafka-consumer which picks ups messages from kafka server and processes them.
This setup is working fine for single topic and single consumer. However how do I create separate consumer groups within same kafka topic? How to route multiple consumer specific messages within same topic inside different consumer groups? Any help is appreciated. Thank you.

Your question is quite general as it's not very clear what's the problem you are trying to solve, therefore it's hard to understand if there's a better way to implement the solution.
Anyway let's start by saying that, as far as I can understand, you are looking for a Selective Consumer (EIP) which is something that's not supported out-of-the-box by Kafka and Consumer API. Selective Consumer can choose what message to pick from the queue or topic based on specific selectors' values that are put in advance by a producer. This feature must be implemented in the message broker as well, but kafka has not such a capability.
Kafka does implement a hybrid solution between pure pub/sub and queue. That being said, what you can do is subscribing to the topic with one or more consumer groups (more on that later) and filter out all messages you're not interested in, by inspecting messages themselves. In the messaging and EIP world, this pattern is known as Array of Filters. As you can imagine this happen after the message has been broadcasted to all subscribers; therefore if that solution does not fit your requirements or context, then you can think of implementing a Content Based Router which is intended to dispatch the message to a subset of consumers only under your centralized control (this would imply intermediate consumer-specific channels that could be other Kafka topics or seda/VM queues, of course).
Moving to the second question, here is the official Kafka Component website: https://camel.apache.org/components/latest/kafka-component.html.
In order to create different consumer groups, you just have to define multiple routes each of them having a dedicated groupId. By adding the groupdId property, you will inform the Consumer Group coordinators (that reside in Kafka brokers) about the existence of multiple separated groups of consumers and brokers will use those in order to discriminate and treat them separately (by sending them a copy of each log message stored in the topic)...
Here is an example:
public void configure() throws Exception {
from("kafka:myTopic?brokers={{kafkaBootstrapServers}}" +
"&groupId=myFirstConsumerGroup"
.log("Message received by myFirstConsumerGroup : ${body}");
from("kafka:myTopic?brokers={{kafkaBootstrapServers}}" +
"&groupId=mySecondConsumerGroup"
.log("Message received by mySecondConsumerGroup : ${body}");
}
As you can see, I created two routes in the same RouteBuilder, not to say in the same Java process. That's a very bad design decision in most of the use cases I can think of, because there's no single responsibility, segregated concerns and they will not scale. But again, it depends on your requirements/context.
Out of completeness, please consider taking a look at all other Kafka Component properties, as there may be many other configurations of your interest such as the number of consumer threads per group.
I tried to stay high level, in order to initiate the discussion... I'll edit my answer in case of new updates from you. Hope I helped!

Cancel ActiveMQ message

I'm not sure if ActiveMQ is a right tool here...
I have a task queue and multiple consumers, so my idea was to use ActiveMQ to post tasks, which are then consumed by consumers.
But I need to be able to cancel the task, if it was not processed yet...
Is there an API for removing Message from Queue in ActiveMQ?
Destination destination = session.createQueue(TOPIC_NAME);
MessageProducer producer = session.createProducer(destination);
ObjectMessage message = session.createObjectMessage(jobData);
producer.send(message);
...
producer.cancel(message); (?)
The use-case is that, for any reason, performing the task is no longer needed, and the task is resource-consuming.

What about setting an expiry time on the message?
http://activemq.apache.org/how-do-i-set-the-message-expiration.html
If you want a message to be deleted if it has not been processed / consumed in a particular time frame, then message expiry seems the answer to me.

ActiveMQ exposes a JMX interface that allows for operations of this kind. The MBean that models a Queue (e.g., org.apache.activemq:type=broker,brokerName=amq,destinationType=Queue,destinationName=my_queue) exposes a method removeMessage (String id). There are also methods that remove messages that match a particular pattern.
So far as I know, this functionality is not exposed outside JMX.
But...
I have a nasty feeling that JMX operations that work on specific messages only work on messages that are paged into memory. By default that would usually be the 400 messages nearest the head of the queue. I know this is true for selector operations, although I'm not sure about JMX.
Some ActiveMQ message stores (e.g., the JDBC store) might also provide a way to get to the underlying message data and manipulate it. On a relational database this is usually safe to do, because messages that are 'in flight' in a JMS operation will be locked at the database level. However, this is a lot of hassle for what ought to be a simple job.
I wonder if JMS is really the right technology for this job? It isn't really intended for random access. Perhaps some sort of distributed data cache would work better (jgroups, Hazelcast,...)?

For those who are looking for a direct answer, there's a JMS API to control this behaviour:
Per JMS API docs:
setTimeToLive(long timeToLive)
Specifies the time to live of messages that are sent using this JMSProducer.
So you can set this value on the producer before sending:
...
producer.setTimeToLive(30000L);
producer.send();
With this particular setting, messages will be retained for 30 seconds before being deleted by the Message Broker

JMS - How do message selectors work with multiple queue and topic consumers?

Say you have a JMS queue, and multiple consumers are watching the queue for messages. You want one of the consumers to get all of a particular type of message, so you decide to employ message selectors.
For example, you define a property to go in your JMS message header named, targetConsumer. Your message selector, which you apply to the consumer known as, A, is something like WHERE targetConsumer = 'CONSUMER_A'.
It's clear that consumer A will now just grab messages with the property set like it is in in the example. Will the other consumers have awareness of that, though? IOW, will another consumer, unconstrained by a message selector, grab the CONSUMER_A messages, if it looks at the queue before Consumer A? Do I need to apply message selectors like, WHERE targetConsumer <> 'CONSUMER_A' to the others?
I am RTFMing and gathering empirical data now, but was hoping someone might know off the top of their head.

When multiple consumers use the same queue, message selectors need to configured correctly across these consumers so that there is no conflict in determining the intended consumer.
In the case of message-driven-beans (a consumer of JMS messages), the selector can be specified in the ejb-jar.xml file thereby allowing for the configuration to be done at deployment time (instead of the opposing view of specifying the message selector during development).
Edit: In real life, this would make sense when different consumers are responsible for processing messages containing the same headers (often generated by the same producer) written onto the same queue. For instance, message selectors could be used in a trading application, to differentiate between buy and sell orders, when the producer is incapable of writing the JMS messages onto two separate buy and sell queues.

Yes, another consumer which is not using any message selector will get message intended for consumer A (or for that matter any message on top of the queue). Hence when sharing a queue, consumer applications must be disciplined and pick only those messages intended for them.

The 'first' JMS message consumer from a queue will pick up the message if the selector matches. What 'first' means is an implementation detail (could be round-robin, based on priority or network closeness). So when using selectors on queues you need to make sure that these selectors are 'non overlapping'.
More formally: no message must exist that matches 2 selectors on the same queue
This is yet another disadvantage of queues versus topics - in practice you should always consider using topics first. With a topic each matching consumer receives the message.

Message Driven Beans - Single Bus, Multiple Activation Specs

I have 2 Message driven beans. 2 Activation Specs for these beans. I have one Message bus and both the activation specs are configured to this one bus. I have 2 different queues and one queue connection factory configured for that one Message bus.
Now, I would write my code to send a message to one of the queues at runtime after determining the queue. However, both my MDBs receive the same message.
How is this configuration done in general? Do I always configure 1 Queue -> 1 Queue Connection Factory -> 1 Message Bus -> 1 MDB? Is it all a one-to-one relationship?
Oh, I forgot to mention this: I am using Websphere Application Server v6.1

In general the concept is that:
a message is sent(Queue)/published(Topic) to a destination (Queue/Topic)
the ActivationSpec listens to messages at a particular destintation (Queue/Topic)
ActivationSpec : Destination is a 1:1 relationship
A bean (MDB which is a consumer) is configured to listen to an ActivationSpec.
What this means is that in effect the bean is linked to a destination with a layer of indirection provided by the activationSpec.
Where does the bus come in - SIBus is the messaging infrastructure that makes all this possible. Destinations are hosted on the bus.
Coming to the question - the ActivationSpec would be configured to listen to a destination on the bus to which messages would be sent. The connection factory decides the bus to which message would be sent. As long as the destination name is unique and targetted to a specific queue (JMS Queue is linked to destination on the bus) one message would only be received by one ActivationSpec.
how many destinations (under SIBus link in WAS admin console) have been created on the bus ? Could you check/validate if the configuration is correct?
to answer your questions - "Is it one bus per activation spec and one queue connection factory per queue." - the answer is NO.
Bus is the underlying infrastructure that can host "n" destinations. One ActivationSpec listens to one destination.
With queue connection factory is a factory (J2EE factory pattern) for creating queues.

I think you're saying you want both MDBs to receive the same message, right?
If this is the case then the MDBs should be listening to a topic not a queue.
Alternatively, there are ways you can configure IBM MQ to forward messages, so for example a message posted to a particular queue could be re-posted to n other queues, but I've only seen that used when some sort of message enrichment takes place before the re-posting and so I suspect would be overkill for what you're trying to achieve.

Why do you need the message bus?
Usually I associate an MDB with a queue - it's a 1:1 relationship. Send a message to a queue, the listener gets it. What is the bus buying you?
I've done JMS with WebLogic, and there's no such construct as a message bus required. I think it's an IBM thing.
Here's an example of how to do JMS with Spring. That's how I'd recommend proceeding.
UPDATE: I misinterpreted your question. When you said both your queues were getting the same message, I didn't think that was the desired behavior. If it is, then topics are the correct way to go. Queues are point-to-point messaging; topics are publish/subscribe.

I suspect the configuration you have is not set up the way you think it is. We use the same configuration your described, with many MDB's (with a queue and activation spec), a single factory and message bus and everything works as expected.
To get the behaviour you are seeing is not actually possible unless you either send the same message to both queues, or have defined a topic instead of a queue. I am pretty sure that even if both MDB's are reading from the same queue, only one will get the message, since a queue only supports point to point messaging. What you have described is topic based behaviour.

do message queues provide transactional support?

Say I load messages in a queue from multiple nodes.
Then, one or many nodes are pulling messages from the queue.
Is it possible (or is this normal usage?) that the queue guarantees to not hand out a message to more than one server/node?
And does that server/node have to tell the queue it has completed the operation and the queue and delete the message?

A message queuing system that did not guarantee to hand out a given message to just one recipient would not be worth the using. Some message queue systems have transactional controls. In that case, if a message is collected by one receiver as part of a transaction, but the receiver does not then commit the transaction (and the message queue can identify that the original recipient is no longer available), then it would be reissued. However, the message would not be made available to two processes concurrently.

What messaging/queuing technology are you using ? AMQP can certainly guarantee this behaviour (amongst many others, including pub/sub models)

If you want this in Java - then a JMS compliant messaging system will do what you want - and most messaging systems have a JMS client. You can Use Spring's JmsTemplate for real ease of use too.
With JMS - a message from a Queue will only be consumed by one and only one client - and once it is consumed (acknowledged) - it will be removed from the messaging system. Also when you publish a message using JMS - if its persistent - it will be sent synchronously, and the send() method won't return until the message is stored on the broker's disk - this is important - if you don't want to run the risk of loosing messages in the event of failure.