We Keep Coding

How to run multiple instances of the same app as producer/consumer in Kafka

i have an simple rest api that have a h2 database so my plan is when i run multiple instances of the same app they will have different in memory databases.Now i want to syncronize these databases beetwen them.I thought kafka to be a good solution , so for example when i get an POST for instance with port 8080 , i should post also for all other instances. Now my app acts as a producer/consumer at the same time and i do not know why only one instance receive the message.
The code:
#EnableKafka
#Configuration
public class KafkaProducerConfigForDepartment {
#Value(value = "${kafka.bootstrapAddress}")
private String bootstrapAddress;
#Bean
public ProducerFactory<String, MessageEventForDepartment> producerFactoryForDepartment() {
Map<String, Object> configProps = new HashMap<>();
configProps.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, bootstrapAddress);
configProps.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class);
configProps.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, JsonSerializer.class);
return new DefaultKafkaProducerFactory<>(configProps);
}
#Bean
public KafkaTemplate<String, MessageEventForDepartment> kafkaTemplate() {
return new KafkaTemplate<>(producerFactoryForDepartment());
}
}
#Configuration
public class KafkaTopicConfig {
#Value(value = "${kafka.bootstrapAddress}")
private String bootstrapAddress;
#Bean
public ConsumerFactory<String, MessageEventForDepartment> consumerFactoryForDepartments() {
Map<String, Object> props = new HashMap<>();
props.put(JsonDeserializer.TRUSTED_PACKAGES, "*");
props.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, bootstrapAddress);
props.put(ConsumerConfig.GROUP_ID_CONFIG, "groupId");
return new DefaultKafkaConsumerFactory<>(props, new StringDeserializer(), new JsonDeserializer<>(MessageEventForDepartment.class));
}
#Bean
public NewTopic topic1() {
return TopicBuilder.name("topic12")
.partitions(10)
.replicas(10)
.build();
}
#Bean
public ConcurrentKafkaListenerContainerFactory<String, MessageEventForDepartment>
kafkaListenerContainerFactoryForDepartments() {
ConcurrentKafkaListenerContainerFactory<String, MessageEventForDepartment> factory =
new ConcurrentKafkaListenerContainerFactory<>();
factory.setConsumerFactory(consumerFactoryForDepartments());
return factory;
}
}
#Component
#Slf4j
public class DepartmentKafkaService {
#Autowired
private DepartmentService departmentService;
#KafkaListener(topics = "topic12" , groupId = "groupId",containerFactory = "kafkaListenerContainerFactoryForDepartments")
public void listenGroupFoo(MessageEventForDepartment message) {
log.info(message.toString());
}
}
Why is this happening ? or maybe my approach is not very good , what are your thoughts ,guys?

Have you considered Kafka Streams? In my opinion, your solution is already done by internal RocksDB and Global KTable implementation in Kafka Streams.
RocksDB will behave exactly like the H2 database which you've mentioned. GlobalKTables functionality allows you to broadcast the current state to all running KafkaStreams instances and read data with ease.
Example:
Producer part:
#RestController
class MessageEventForDepartmentController {
#Autowired
KafkaTemplate<String, MessageEventForDepartment> kafkaTemplate;
#PostMapping(path = "/departments", consumes = "application/json")
#ResponseStatus(HttpStatus.ACCEPTED)
void(#RequestBody MessageEventForDepartment event) {
kafkaTemplate.send("topic-a", event.getId(), event);
}
}
Consumer part - KafkaStreams GlobalKTable
#Component
public class StreamsBuilderMessageEventForDepartment {
#Autowired
void buildPipeline(StreamsBuilder streamsBuilder) {
KeyValueBytesStoreSupplier storeSupplier = Stores.inMemoryKeyValueStore("MessageEventForDepartmentGlobalStateStore");
Materialized<String, MessageEventForDepartment, KeyValueStore<Bytes, byte[]>> materialized = Materialized.<String, MessageEventForDepartment>as(storeSupplier)
.withKeySerde(Serdes.String())
.withValueSerde(new JsonSerde(MessageEventForDepartment.class));
GlobalKTable<String, MessageEventForDepartment> messagesCount = messagesGroupedByUser.globalTable("topic-a", materialized);
}
}
Read data from RocksDB
#RestController
class MessageEventForDepartmentReadModelController {
#Autowired
KafkaStreams kafkaStreams
#Get(path = "/departments")
MessageEventForDepartment getMessageEventForDepartment(String eventId) {
ReadOnlyKeyValueStore<String, MessageEventForDepartment> store = kafkaStreams.store(StoreQueryParameters.fromNameAndType("MessageEventForDepartmentGlobalStateStore", QueryableStoreTypes.keyValueStore()));
return store.get(eventId);
}
}

The reason why only one instance of the application receives each message is that each instance has the same ConsumerConfig.GROUP_ID_CONFIG. Kafka's consumer protocol is such that each consumer group gets each message delivered once (obviously, there's a lot more nuance to it, but this is basically how it works).
Pawel's suggestion to use KafkaStreams is a good one—a GlobalKTable would provide what you want.
Luca Pette wrote a great primer on Kakfa Streams here: https://lucapette.me/writing/getting-started-with-kafka-streams/

My understanding to your qus is that your using multiple instances for the same app which uses IN-MEMEORY so for Eventually consistency your going with Kafka stream.
MY SOLUTIONS:
I have used Rabbitmq mirroring which solves the same problem you have in Kafka also supports mirroring find the doc: https://cwiki.apache.org/confluence/plugins/servlet/mobile?contentId=27846330#content/view/27846330
Consider redis cluster or master slave for In-memory db

How do I get a Kafka message before the #KafkaListener method?

My task is to get the Kafka message before the method with the #KafkaListner annotation, check the correlationId and requestId headers in it. If they're present, flush them to MDC or generate them otherwise.
And my question is how to get Kafka message with headers before method with the #KafkaListner?

You can try to write your own ConsumerInterceptor following instructions from here.
Apache Kafka provides a mechanism to add interceptors to producers and consumers. These objects are managed by Kafka, not Spring, and so normal Spring dependency injection won’t work for wiring in dependent Spring Beans. However, you can manually wire in those dependencies using the interceptor config() method. The following Spring Boot application shows how to do this by overriding boot’s default factories to add some dependent bean into the configuration properties.
ConsumerFactory definition:
#Bean
public ConsumerFactory<?, ?> kafkaConsumerFactory(SomeBean someBean) {
Map<String, Object> consumerProperties = new HashMap<>();
// consumerProperties.put(..., ...)
// ...
consumerProperties.put(ConsumerConfig.INTERCEPTOR_CLASSES_CONFIG, MyConsumerInterceptor.class.getName());
consumerProperties.put("some.bean", someBean);
return new DefaultKafkaConsumerFactory<>(consumerProperties);
}
Interceptor definition:
public class MyConsumerInterceptor implements ConsumerInterceptor<String, String> {
private SomeBean bean;
#Override
public void configure(Map<String, ?> configs) {
this.bean = (SomeBean) configs.get("some.bean");
}
#Override
public ConsumerRecords<String, String> onConsume(ConsumerRecords<String, String> records) {
this.bean.someMethod("consumer interceptor");
return records;
}
#Override
public void onCommit(Map<TopicPartition, OffsetAndMetadata> offsets) {
}
#Override
public void close() {
}
}

Add a RecordInterceptor to the listener container (or factory that creates it).

Disable Kafka connection in SpringBoot tests

I'm working on a springboot project following a microservice architecture and I use Kafka as an event bus to exchange data between some of them. I also have Junit tests which test some part of my application which doesn't require the bus and others that require it by using an embedded Kafka broker.
The problem I have is when I launch all my tests, they take so much time and they fail because each of then is trying to connect to the embedded Kafka broker (connection not available) whereas they don't need Kafka bus in order to achieve their task.
Is it possible to disable the loading of Kafka components for these tests and only allow them for the ones that require it ?

This is how I usually write my JUnit tester classes, that usually wont connect to KAFKA Brokers for each test.
Mocking the REST API, if your KAFKA Client (Producer/Consumer) integrated with a REST API
public class MyMockedRESTAPI {
public MyMockedRESTAPI() {
}
public APIResponseWrapper apiResponseWrapper(parameters..) throws RestClientException {
if (throwException) {
throw new RestClientException(....);
}
return new APIResponseWrapper();
}
}
A factory class to generate an incoming KAFKA Event and REST API request and response wrappers
public class mockFactory {
private static final Gson gson = new Gson();
public static KAKFAEvent generateKAFKAEvent() {
KAKFAEvent kafkaEvent = new KAKFAEvent();
kafkaEvent.set...
kafkaEvent.set...
kafkaEvent.set...
return KAKFAEvent;
}
public static ResponseEntity<APIResponse> createAPIResponse() {
APIResponse response = new APIResponse();
return new ResponseEntity<>(response, HttpStatus.OK);
}
}
A Test Runner Class
#RunWith(SpringJUnit4ClassRunner.class)
public class KAFKAJUnitTest {
Your assertion should be declared here
}
You can also refer : https://www.baeldung.com/spring-boot-kafka-testing

a good practice would be to avoid sending messages to Kafka while testing code in your isolated microservice scope. but when you need to make an integration test ( many microservices in the same time ) sometimes you need to activate Kafka messages.
So my purpose is :
1- Activate/Deactivate loding Kafka configuration as required
#ConditionalOnProperty(prefix = "my.kafka.consumer", value = "enabled", havingValue = "true", matchIfMissing = false)
#Configuration
public class KafkaConsumerConfiguration {
...
}
#ConditionalOnProperty(prefix = "my.kafka.producer", value = "enabled", havingValue = "true", matchIfMissing = false)
#Configuration
public class KafkaProducerConfiguration {
...
}
and then u will be able to activate/deactivate loading consumer and producer as you need...
Examples :
#SpringBootApplication
#Import(KafkaConsumerConfiguration.class)
public class MyMicroservice_1 {
public static void main(String[] args) {
SpringApplication.run(MyMicroservice_1.class, args);
}
}
or
#SpringBootApplication
#Import(KafkaProducerConfiguration.class)
public class MyMicroservice_2 {
public static void main(String[] args) {
SpringApplication.run(MyMicroservice_2.class, args);
}
}
or maybe a microservice that need both of configurations
#SpringBootApplication
#Import(value = { KafkaProducerConfiguration.class, KafkaConsumerConfiguration.class })
public class MyMicroservice_3 {
public static void main(String[] args) {
SpringApplication.run(MyMicroservice_3.class, args);
}
}
2 - You need also to make sending messages depending on the current spring profile. To do that u can override the send method of the Kafka template object:
#ConditionalOnProperty(prefix = "my.kafka.producer", value = "enabled", havingValue = "true", matchIfMissing = false)
#Configuration
public class KafkaProducerConfiguration {
...
#Resource
Environment environment;
#Bean
public KafkaTemplate<String, String> kafkaTemplate() {
return new KafkaTemplate<>(producerFactory()) {
#Override
protected ListenableFuture<SendResult<String, String>> doSend(ProducerRecord<String, String> producerRecord) {
if (Arrays.asList(environment.getActiveProfiles()).contains("test")) {
return null;
}
return super.doSend(producerRecord);
}
};
}
#Bean
public ProducerFactory<String, String> producerFactory() {
Map<String, Object> props = new HashMap<>();
...
return new DefaultKafkaProducerFactory<>(props);
}
}

Embedded Kafka Spring test executes before embedded Kafka is ready

I have a Spring Boot project that has a Kafka listener that I want to test using Embedded Kafka. I have the Kafka Listener log out the message "record received". Which will only be be logged out if I add a Thread.sleep(1000) to the start of the method.
Test class:
#SpringBootTest
#DirtiesContext
#EnableKafka
#EmbeddedKafka(partitions = 1, topics = { "my-topic" }, ports = 7654)
class KafkaTest {
private static final String TOPIC = "my-topic";
#Autowired
EmbeddedKafkaBroker kafkaBroker;
#Test
void testSendEvent() throws ExecutionException, InterruptedException {
// Thread.sleep(1000); // I wont see the Listener log message unless I add this sleep
Producer<Integer, String> producer = configureProducer();
ProducerRecord<Integer, String> producerRecord = new ProducerRecord<>(TOPIC, "myMessage");
producer.send(producerRecord).get();
producer.close();
}
private Producer<Integer, String> configureProducer() {
Map<String, Object> producerProps = new HashMap<>(KafkaTestUtils.producerProps(kafkaBroker));
return new DefaultKafkaProducerFactory<Integer, String>(producerProps).createProducer();
}
}
I don't want to use the fickle Thread.sleep() The test is obviously executing before some setup processes have completed. I clearly need to wait on something, but I am not sure what nor how to do it.
Using:
Java 11
Spring Boot 2.5.6
JUnit 5
spring-kafka-test 2.7.8

Add an #EventListener bean to the test context and (for example) count down a CountDownLatch when a ConsumerStartedEvent is received; then in the test
assertThat(eventListner.getLatch().await(10, TimeUnit.SECONDS)).isTrue();
See https://docs.spring.io/spring-kafka/docs/current/reference/html/#events
and
https://docs.spring.io/spring-kafka/docs/current/reference/html/#event-consumption
Or add a ConsumerRebalanceListener and wait for partition assignment.

I clearly need to wait on something, but I am not sure what nor how to do it.
You need to use a different method to give Kafka time to process and route the message ...
Look at this line ...
ConsumerRecord<String, String> message = records.poll(500, TimeUnit.MILLISECONDS);
When testing Kafka listeners we always specify a poll delay. This is because your message is given to kafka, which will then process it in another thread. And you need to wait for it.
Here's how it looks in context of the code its used in.
class UserKafkaProducerTest {
#Test
void testWriteToKafka() throws InterruptedException, JsonProcessingException {
// Create a user and write to Kafka
User user = new User("11111", "John", "Wick");
producer.writeToKafka(user);
// Read the message (John Wick user) with a test consumer from Kafka and assert its properties
ConsumerRecord<String, String> message = records.poll(500, TimeUnit.MILLISECONDS);
assertNotNull(message);
assertEquals("11111", message.key());
User result = objectMapper.readValue(message.value(), User.class);
assertNotNull(result);
assertEquals("John", result.getFirstName());
assertEquals("Wick", result.getLastName());
}
}
This is a code piece from this article, which makes stuff clear.

You can use this small library for testing.
All output records will be collected to blocking queue and you can poll them with timout:
#OutputQueue(topic = TOPIC_OUT, partitions = 1)
private BlockingQueue<ConsumerRecord<String, String>> consumerRecords;
#Test
void shouldFilterRecordWithoutHeader() throws ExecutionException, InterruptedException, TimeoutException {
final String messageIn = "hello world";
try (var producer = producer()) {
producer.send(new ProducerRecord<>(TOPIC_IN, messageIn)).get(5, TimeUnit.SECONDS);
}
ConsumerRecord<String, String> record = consumerRecords.poll(5, TimeUnit.SECONDS);
Assertions.assertThat(record).isNotNull();
}

How does spring.kafka.consumer.auto-offset-reset works in spring-kafka

KafkaProperties java doc:
/**
* What to do when there is no initial offset in Kafka or if the current offset
* does not exist any more on the server.
*/
private String autoOffsetReset;
I have hello world appllication which contains application.properties
spring.kafka.consumer.group-id=foo
spring.kafka.consumer.auto-offset-reset=latest
At this case #KafkaListener method is invoked for all entries. But expected result was that #KafkaListener method is invoked only for latest 3 options I send. I tried to use another option:
spring.kafka.consumer.auto-offset-reset=earlisest
But behaviour the same.
Can you explain this stuff?
P.S.
code sample:
#SpringBootApplication
public class Application implements CommandLineRunner {
public static Logger logger = LoggerFactory.getLogger(Application.class);
public static void main(String[] args) {
SpringApplication.run(Application.class, args).close();
}
#Autowired
private KafkaTemplate<String, String> template;
private final CountDownLatch latch = new CountDownLatch(3);
#Override
public void run(String... args) throws Exception {
this.template.send("spring_kafka_topic", "foo1");
this.template.send("spring_kafka_topic", "foo2");
this.template.send("spring_kafka_topic", "foo3");
latch.await(60, TimeUnit.SECONDS);
logger.info("All received");
}
#KafkaListener(topics = "spring_kafka_topic")
public void listen(ConsumerRecord<?, ?> cr) throws Exception {
logger.info(cr.toString());
latch.countDown();
}
}
Update:
Behaviour doesn't depends on
spring.kafka.consumer.auto-offset-reset
it is only depends on spring.kafka.consumer.auto-offset-reset=earliest
spring.kafka.consumer.enable-auto-commit
if I set spring.kafka.consumer.enable-auto-commit=false - I see all records.
if I set spring.kafka.consumer.enable-auto-commit=true - I see only 3 last records.
Please clarify menaning of spring.kafka.consumer.auto-offset-reset property

The KafkaProperties in Spring Boot does this:
public Map<String, Object> buildProperties() {
Map<String, Object> properties = new HashMap<String, Object>();
if (this.autoCommitInterval != null) {
properties.put(ConsumerConfig.AUTO_COMMIT_INTERVAL_MS_CONFIG,
this.autoCommitInterval);
}
if (this.autoOffsetReset != null) {
properties.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG,
this.autoOffsetReset);
}
This buildProperties() is used from the buildConsumerProperties() which, in turn in the:
#Bean
#ConditionalOnMissingBean(ConsumerFactory.class)
public ConsumerFactory<?, ?> kafkaConsumerFactory() {
return new DefaultKafkaConsumerFactory<Object, Object>(
this.properties.buildConsumerProperties());
}
So, if you use your own ConsumerFactory bean definition be sure to reuse those KafkaProperties: https://docs.spring.io/spring-boot/docs/1.5.7.RELEASE/reference/htmlsingle/#boot-features-kafka-extra-props
UPDATE
OK. I see what's going on.
Try to add this property:
spring.kafka.consumer.enable-auto-commit=false
This way we won't have async auto-commits based on some commit interval.
The logic in our application is based on the exit fact after the latch.await(60, TimeUnit.SECONDS);. When we get 3 expected records we exit. This way the async auto-commit from the consumer might not happen yet. So, the next time you run the application the consumer polls data from the uncommited offset.
When we turn off auto-commit, we have an AckMode.BATCH, which is performed synchronously and we have an ability to see really latest recodrs in the topic for this foo consumer group.