We have a simple application. One micro service will send messages to be consumed by two other micro services. Of the two services, one is able to successfully process the messages and we don't see a lag there. The other service consistently fails and after 25th message no other message is consumed. Is there a reason for this?
The Kafka topic is created with one partition and one replication factor.
Service 1 - Working Fine:
Service Working Fine
Service 2 - Lag increasing:
Lag Increasing
Is there a configuration in Kafka that will make the consumer to stop consuming the messages after a particular amount of failure or what can we do to avoid this behavior?
you are saying a topic with one partition and you have two different services, I'm assuming different consumer groups names consuming from the same topic partition, one service is consuming fine and other one stop consuming at 25th message. In this case you can check in that consumer logs why it is not consuming, this could be malformed message where the consumer is able to consume.
here is how I'd debug
restart or redeploy the same consumer and see it is alway stop at 25th message
if step 1 is true, check 24th and 25th message and see the difference or reset offset to 26th to see, if consumer is moving forward after offset reset, then message 25th having some issue where consumer unable to consume.
Related
I am building an event driven software in java to listen to a Kafka Topic and sent the messages to a other server from my application. I want the Kafka consumer to keep polling the same message if my application wasn't able to sent the data successfully to the second server. to do this I had set manual commit offset and only incremented the offset when the message was sent successfully to the second server but the broker will only resend the message if my application(consumer) restarts. its a issue since I don't want my application to restart. Let me know if you have any solutions to this issue.
You would need to track and manually seek the consumer to the last un-processed offset(s) for each topic partition. You may also want to pause() the consumer and halt any poll loop until each record is processed
i am facing a problem with my kafka consumer. i have two kafka brokers running with replication factor 2 for the topic. everytime a broker restarts and if i restart my consumer service, it starts to read records which it has already read. e.g. before i restarted the consumer this was the state.
and consumer was sitting idle not receiving any messages as it has read all of them.
i restart my consumer, and all of a sudden it starts receiving messages which it has processed previously and here is the offset situation now.
also what is this LOG-END-OFFSET and LAG, looks like these are something to consider here.
note that it only happens when 1 of the broker gets restarted due to kubernetes shifting it to another node.
this is the topic configuration
Based on the info you posted, a couple of things that immediately come to mind:
The first screenshot shows a lag of 182, which means the consumer either was not running, or it has some weird configuration that made it stop consuming. Was it possible one of the brokers was down when the consumer stopped consuming?
On restart, the consumer finally consumed all the remaining messages, because it now shows lag of 0. This is correct, expected Kafka behavior.
Make sure that the consumer group name is not changing between restarts. Some clients default to "randomized" customer group names, which works as long as the consumer is not restarted.
Currently when starting consumer application it will receive old messages that have not been processed by KafkaListener and I only want to receive the latest messages since starting the consumer application ignore those old messages, I have to do that any?
This is pretty brief introduction into your issue - it would be handy to show versions of libraries you are working with, configurations, etc.
Nevertheless, if you do not want to receive old messages, that has not been ack before, you need to move offset for you consumer group.
Offset is basically pointer at last successfully read item, so when consumer is stopped, it remains here until consumer starts reading again - that is the reason why "old" messages are read. In this thread are some answers, but it is difficult to answer completely without further information.
Set consumer settings as auto.offset.reset=latest and enable.auto.commit=false, then your app will always start reading from the very end of the Kafka topic and ignore whatever was written while the app is stopped (between restarts, for example)
You could also add a random UUID to the group.id to ensure no other consumer would easily join that consumer group and "take away" events from your app.
Kafka Consumer API also has a method seekToEnd that you can try.
How can i get all messages are published before subscribed a Topic In Kafka? I m using Kafka 2.12-2.3.0 Producer for sending message and Consumer to receive message. Actually i m building a chat application in java using Kafka. But problem is if a producer posted some message and a consumer subscribed later it wont get that messages. Please give some suggestion.
Your consumer has configuration option auto.offset.reset (exact name depends on the SDK you are using). Set it to 1 (Earliest). This will work when you connect for the first time.
Also you can explicitly assign start offset for every partition, like: consumer.Assign(offsets); - exact code depends on your scenario, but assigning 0 will ensure you get the earliest.
I am running a simple set up for Apache Kafka using the APIs for Producer and Consumer.
In order to simulate heavy load, I am
running multiple instances of the Producer (say 2),
all of which are sending the same message (message content is a don't care)
multiple times (say 1000 for each topic)
to a large number of topics (say 5)
I am running a single Consumer to read messages from all the topics and keep a count of the number of messages processed.
I would expect at the Consumer end to get (2 x 5 x 1000) = 10000 messages.
But the number of messages received is less than expected.
This behavior does not exist for a smaller set of messages (say 50 messages sent to each topic). So I know that it cannot be something wrong with my setup.
Are there some configurations that I am missing here? Or perhaps, Kafka did not receive some messages from the Producer instances and the API is not notifying me?
FYI: This is being run on a single VM hosted in my personal machine. Both Kafka and Zookeeper are on the same machine. I'm not really interested in the performance of the setup as of now. Performance metrics are not valid if some messages go missing.