We Keep Coding

Camel consumer route with input from another consumer route

I have a kafka consumer route from where I get some data.
from("Kafka:foo?brokers=localhost:9092")
Once I receive data from the consumer, use that data in the topic name for a paho mqtt consumer.
from("paho:#?brokerUrl=tcp://localhost:1883")
I'm not able to figure out how to set the dynamic header CamelMqttTopic, from first consumer, as both seems independent flows. I'm using camel with Spring framework. Excuse me if my basic camel understanding is flawed.

You can override the MQTT topic using the CamelPahoOverrideTopic message header with a value being the Kafka topic accessed through the kafka.TOPIC message header:
from("kafka:foo?brokers=localhost:9092")
.setHeader(PahoConstants.CAMEL_PAHO_OVERRIDE_TOPIC, simple("${headers[kafka.TOPIC]"))
.to("paho:#?brokerUrl=tcp://localhost:1883");

Kafka consumer in test only works with "auto.offset.reset"="earliest"

I'm struggling to understand my Kafka consumer behaviours in some integration tests.
I have a Spring boot service which uses a default, autowired KafkaTemplate<String, String> to produce messages to a topic. In my integration tests, I create a KafkaConsumer in each test:
KafkaConsumer<String, String> consumer = new KafkaConsumer<>(
Map.of( ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, KAFKA_CONTAINER.getBootstrapServers(),
ConsumerConfig.GROUP_ID_CONFIG, "test-consumer-group-" + UUID.randomUUID(),
ConsumerConfig.GROUP_INSTANCE_ID_CONFIG, UUID.randomUUID().toString() ),
new StringDeserializer(), new StringDeserializer() );
consumer.subscribe( topics );
return consumer;
with the intent of having a test flow that looks something like:
Create a new consumer for the topics we're testing
Perform action under test which sends messages to some topics
Poll the topics of interest and verify the messages are there
Close consumer
My expectation was that since the default behaviour of a new consumer is to have auto.offset.reset set to latest I would only get messages sent after I create the consumer, which looks fine in this case. However my consumer never receives any messages! I have to set the consumer to earliest - but this is problematic since I don't want messages created by other tests interfering.
The messages don't have any kind of unique identifier on them, which makes consuming the entire topic each time a tricky proposition in terms of test verifications.
I've tried various permutations of auto committing, polling before running the test but after subscribing, manual syncs but nothing seems to work - how can manage my test lifecycle as described above (or is it not possible)?
The kafka instance is managed using TestContainers in case that's relevant.

Difference between KafkaTemplate and KafkaProducer send method?

My question is in a spring boot microservice using kafka what is appropriate to use KafkaTemplate.send() or KafkaProducer.send()
I have used KafkaConsumer and not KafkaListner to poll the records because KafkaListner was fetching the records as and when they were coming to the topics, I wanted the records to be polled periodically based on business needs.
Have gone through the documentation of KafkaProducer https://kafka.apache.org/10/javadoc/org/apache/kafka/clients/producer/KafkaProducer.html
and Spring KafkaTemplate
https://docs.spring.io/spring-kafka/reference/html/#kafka-template
I am unable to make a decision like what is ideal to use or atleast the reason of using one over the other is unclear?
What my need is I want the operation to be sync i.e. I want to know if the published happened successfully or not because If the record is not delivered I need to retry publishing.
Any help will be appreciated.

For your first question, which one should I use kafka Template or Kafka producer?
The Kafka Producer is defined in Apache Kafka. The KafkaTemplate is Spring's implementation of it (although it does not implement Producer
directly) and so it provides more methods for you to use.
Read this link::
What is the difference between Kafka Template and kafka producer?
For retry mechanism, in case of failure in publishing.
I have answered this in another question.
The acks parameter control how many partition replicas must receive
the record before the producer can consider the write successful.
There are 3 values for the acks parameter:
acks=0, the producer will not wait for a reply from the broker before
assuming the message sent successfully.
acks=1, the producer will receive a successful response from the
broker the moment the leader replica received the message. If the
message can't be written to the leader, the producer will receive an
error response and can retry.
acks=all, the producer will receive a successful response from the
broker once all in-sync replicas received the message.
Best way to configure retries in Kaka Producer

Two instances consuming from topic

I have deployed two instances of an application. Both applications runs the same code and consumes from the same topic.
#KafkaListener( offsetReset = OffsetReset.EARLIEST, offsetStrategy = OffsetStrategy.DISABLED )
public class AppConsumer implements ConsumerRebalanceListener, KafkaConsumerAware {
#Topic("topic")
public void consumeAppInfo(#KafkaKey String name, #Body #Nullable String someString) {
...
}
}
I have a problem where only one of the applications consumes the message. The topic has only one partition, partition 0, which i believe is default.
I have tried to add group-id and threads to the KafkaListener. This seems to work sometimes and other time not.
#KafkaListener(groupId="myGroup", threads=10)
What is the simplest solution to getting both applications to consume the same message?

You could not do the group and just give each application a separate consumer id each consumer consumes all messages (unless they are also assigned to a group).
Groups are used for parallel processing of messages each consumer in a group get assigned to a partition for processing messages.
More info => difference between groupid and consumerid in Kafka consumer

In kafka, only one consumer within consumer group is assigned to each partition. If you want to consume the data from the same partition by different applications, you need to specify different consumer groups for each different application / instance.

Is there an option to create consumers that accept only messages with high priority in RabbitMQ and Spring?

I'm creating an application that sends messages for time-expensive processing to a consumer using RabbitMQ. However, I need to prioritize messages. When a message with high priority arrives, it must be processed even if all consumer instances are processing other messages.
AFAIK there is no possibility to preempt processing low-priority messages and switch to processing high-priority messages in Spring Boot and RabbitMQ.
Is it possible to create consumers that accept only high-priority messages or to run additional set of consumers on the fly when all other are busy and high-priority messages arrive?
I tried to add queues with x-max-priority=10 flag and to increase number of consumers but it doesn't solve my problem.
Imagine that we run 50 consumers and send 50 messages with low priority. While time-expensive processing is being performed, a new message arrives with high priority but it cannot be processed at once because all 50 consumers are busy.
There is a part of configuration that sets number of consumers
#Bean
public SimpleRabbitListenerContainerFactory
rabbitListenerContainerFactory(SimpleRabbitListenerContainerFactoryConfigurer configurer,
#Qualifier("rabbitConnectionFactory") ConnectionFactory connectionFactory) {
SimpleRabbitListenerContainerFactory factory = new SimpleRabbitListenerContainerFactory();
configurer.configure(factory, connectionFactory);
factory.setConcurrentConsumers(50);
factory.setMaxConcurrentConsumers(100);
return factory;
}
Is there a way to create a set of consumers that accept messages high-priority messages (e.g. higher than 0) or to create consumer on the fly for high-priority messages?

I don't know about a way to implement the preemptive strategy you describe, but there's a number of alternative things that you could consider.
Priority Setting
The first thing to take into account is the priority support in RabbitMQ itself.
Consider this excerpt from RabbitMQ in Depth by Gavin M. Roy:
“As of RabbitMQ 3.5.0, the priority field has been implemented as per the AMQP specification. It’s defined as an integer with possible values of 0 through 9 to be used for message prioritization in queues. As specified, if a message with a priority of 9 is published, and subsequently a message with a priority of 0 is published, a newly connected consumer would receive the message with the priority of 0 before the message with a priority of 9”.
e.g.
rabbitTemplate.convertAndSend("Hello World!", message -> {
MessageProperties properties = MessagePropertiesBuilder.newInstance()
.setPriority(0)
.build();
return MessageBuilder.fromMessage(message)
.andProperties(properties)
.build();
});
Priority-based Exchange
A second alternative is to define a topic exchange and define a routing key that considers your priority.
For example, consider an exchange of events using a routing key of pattern EventName.Priority e.g. OrderPlaced.High, OrderPlaced.Normal or OrderPlaced.Low.
Based on that you could have a queue bound to just orders of high priority i.e. OrderPlaced.High and a number of dedicated consumers just for that queue.
e.g.
String routingKey = String.format("%s.%s", event.name(), event.priority());
rabbit.convertAndSend(routingKey, event);
With a listener like the one below where the queue high-priority-orders is bound to the events exchange for event OrderPlaced and priority High using routing key OrderPlaced.High.
#Component
#RabbitListener(queues = "high-priority-orders", containerFactory="orders")
public class HighPriorityOrdersListener {
#RabbitHandler
public void onOrderPlaced(OrderPlacedEvent orderPlaced) {
//...
}
}
Obviously, you will need a dedicated thread pool (in the orders container factory above) to attend the high priority requests.

There is no mechanism in the AMQP protocol to "select" messages from a queue.
You might want to consider using discrete queues with dedicated consumers instead.
BTW, this is not spring related; general questions about RabbitMQ should be directed to the rabbitmq-users Google group.