We Keep Coding

Camel consumer route with input from another consumer route

I have a kafka consumer route from where I get some data.
from("Kafka:foo?brokers=localhost:9092")
Once I receive data from the consumer, use that data in the topic name for a paho mqtt consumer.
from("paho:#?brokerUrl=tcp://localhost:1883")
I'm not able to figure out how to set the dynamic header CamelMqttTopic, from first consumer, as both seems independent flows. I'm using camel with Spring framework. Excuse me if my basic camel understanding is flawed.

You can override the MQTT topic using the CamelPahoOverrideTopic message header with a value being the Kafka topic accessed through the kafka.TOPIC message header:
from("kafka:foo?brokers=localhost:9092")
.setHeader(PahoConstants.CAMEL_PAHO_OVERRIDE_TOPIC, simple("${headers[kafka.TOPIC]"))
.to("paho:#?brokerUrl=tcp://localhost:1883");

Should i create NewTopics in each service spring kafka?

I'm using Kafka for sending messages between services. I use NewTopic bean for configuring number of partitions, for example:
#Bean
fun kafkaTopic(kafkaProperties: KafkaProperties): NewTopic = NewTopic(
kafkaProperties.topics.schedulerCalculationTopic.name,
kafkaProperties.topics.schedulerCalculationTopic.partitions,
1
)
My question is simple, should i add this bean into consumer service and producer service or only in one of them?

I would put it in the producer service and then consider the producer as 'owner' of those topics.
But it get a bit complicated if you have a scenario if you would have several producers to the same topic(s).

If you are not creating the topic on the fly, the best practice is to create topic before reading/writing to it.
Rationale is to prevent brokers to create topic whenever they receive metadata fetch request or consume request with the same topic name. Otherwise, if the consumer starts before the producer, you might end up wrong number of partition. (Broker will create your topic with default number of partitions setting.)

Eagerly connect to topic as a Kafka Producer

I'm implementing a service which sends messages to a downstream service via a Kafka topic. This will only happen when my service's API is called, which is likely to be infrequently, at least at first.
I've found that the reactive Kafka producer API connects to Kafka lazily, which I'm sure is great for most use-cases, but I'd like to know that my Kafka connection configuration is correct at the point I deploy my service. I don't want to have to wait for the first API call only to find out that something somewhere is wrong.
The solution I've got for this at the moment is simply to send an initialisation message to the topic at start-up, but this feels clunky. Is there anything better I can be doing to force an initial connection to the topic, or at least to validate the connection configuration?
val response = kafkaSender.send(Mono.just(initialise))
.next()
.block();
if (response == null) {
throw new RuntimeException("empty Mono from Kafka initialisation");
}
if (response.exception() != null) {
throw propagate(response.exception());
}

You can use AdminClient API to describe the cluster or the topic you're going to use if you prefer not to be sending heartbeats via a producer

Can producers have the same clientId and publish to a topic in Artemis?

I was wondering if it's possible to have multiple producers using the same clientId to send messages to a durable topic. And on the consuming side, what would happen if the clientID is the same as the producer side but the subscription name is different?
E.g. The producer has a clientId of 123abc and sends messages to a durable topic. A consumer is subscribed to this durable topic and this consumer has a clientId of 123abc but also a subscriptionName of abc123? Would the consumer still be able to pick up the message? What would happened if I bring another consumer in the mix?

Section 6.1.2 of the JMS 2 specification states:
By definition, the client state identified by a client identifier can be ‘in use’ by only one client at a time.
By "client" the specification really means "connection." Therefore, the same client identifier can only be in use by one connection at a time. So if you create multiple producers from the same connection that's OK. However, creating multiple connections with the same client ID will fail before you even get to the point where you can create a producer as the broker will validate the client ID when the connection is created.
That said, there's no real point in setting the client ID on a connection that's just used for producing messages. Section 6.1.2 of the JMS 2 specification also states:
The only use of a client identifier defined by JMS is its mandatory use in identifying an unshared durable subscription or its optional use in identifying a shared durable or non-durable subscription.
Therefore, it's not really necessary to set the client ID unless you're creating an unshared durable subscription or possibly a shared durable or non-durable subscription.

Two subscribers cannot have the same clientid: when they both try to connect to the broker, the second will go into exception. However, you can override the clientid: using TomEE or Tomcat you can add a simple line to system.properties file like this:
<classname>.activation.clientId=<newclientid>
No problem for producers.

Asynchronous request-reply with Spring Boot and RabbitMQ

We want to implement the following scenario:
A producer service sends some input params to another service asking for the details based on these params.
A producer wants to specify the queue where it will be listening for the reply.
Moreover, a producer wants to provide some metadata so that it can correlate the params it sent with a result it got.
Please advice how to do this properly.

See the AsyncRabbitTemplate.
It uses the correlationId and replyTo properties to convey that information to the service that handles the request.