I am running a simple set up for Apache Kafka using the APIs for Producer and Consumer.
In order to simulate heavy load, I am
running multiple instances of the Producer (say 2),
all of which are sending the same message (message content is a don't care)
multiple times (say 1000 for each topic)
to a large number of topics (say 5)
I am running a single Consumer to read messages from all the topics and keep a count of the number of messages processed.
I would expect at the Consumer end to get (2 x 5 x 1000) = 10000 messages.
But the number of messages received is less than expected.
This behavior does not exist for a smaller set of messages (say 50 messages sent to each topic). So I know that it cannot be something wrong with my setup.
Are there some configurations that I am missing here? Or perhaps, Kafka did not receive some messages from the Producer instances and the API is not notifying me?
FYI: This is being run on a single VM hosted in my personal machine. Both Kafka and Zookeeper are on the same machine. I'm not really interested in the performance of the setup as of now. Performance metrics are not valid if some messages go missing.
Related
I have a situation where in my dev environment, my Kafka consumer groups will rebalance and distribute partitions to consumer instances just fine after increasing the partition count of a subscribed topic.
However, when we deploy our product into its kubernetes environment, we aren't seeing the consumer groups rebalance after increasing the partition count of the topic. Kafka recognized the increase which can be seen from the server logs or describing the topic from the command line. However, the consumer groups won't rebalance and recognize the new partitions. From my local testing, kafka respects metadata.max.age.ms (default 5 mins). But in kubernetes the group never rebalances.
I don't know if it affects anything but we're using static membership.
The consumers are written in Java and use the standard Kafka Java library. No messages are flowing through Kafka, and adding messages doesn't help. I don't see anything special in the server or consumer configurations that differs from my dev environment. Is anyone aware of any configurations that may affect this behavior?
** Update **
The issue was only occurring for a new topic. At first, the consumer application was starting before the producer application (which is intended to create the topic). So the consumer was auto creating the topic. In this scenario, the topic defaulted to 1 partition. When the producer application started it, updated the partition count per configuration. After that update, we never saw a rebalance.
Next we tried disabling consumer auto topic creation to address this. This prevented the consumer application from auto creating the topic on subscription. Yet still after the topic was created by the producer app, the consumer group was never rebalanced, so the consumer application would sit idle.
According to the documentation I've found, and testing in my dev environment, both of these situations should trigger a rebalance. For whatever reason we don't see that happen in our deployments. My temporary workaround was just to ensure that the topic is already created prior to allowing my consumer's to subscribe. I don't like it, but it works for now. I suspect that the different behavior I'm seeing is due to my dev environment running a single kafka broker vs the kubernetes deployments with a cluster, but that's just a guess.
Kafka defaults to update topic metadata only after 5 minutes, so will not detect partition changes immediately, as you've noticed. The deployment method of your app shouldn't matter, as long as network requests are properly reaching the broker.
Plus, check your partition assignment strategy to see if it's using sticky assignment. This will depend on what version of the client you're using, as the defaults changed around 2.7, I think
No messages are flowing through Kafka
If there's no data on the new partitions, there's no real need to rebalance to consume from them
I need to build scalable application that have several Java applications (Spring RabbitMQ producers) that consume messages from other applications by HTTP protocol, calculating priority of message, and send them in to 'priority queue' that matches the params.
So far under load of hundreds of messages per second, one application works just fine, but there is need to scale up applications.
The problem is that I don't really understand how does RabbitMQ producers work with 'priority queues'. I've been searching information in RabbitMQ documentation and I found docs that says that every producer needs to get acks to make sure that messages have proceed successfully.
So the questions are
Docs says that priority of messages calculated under hood AMQP protocol, so do RabbitMQ will send acks to producer after the position for messages will be selected or before
How does messages will be treated if assume that we have 2 producers that produce 2 different messages with same priority to the same 'priority queue'
I will be appreciated for any hint that will help me with that!
i am facing a problem with my kafka consumer. i have two kafka brokers running with replication factor 2 for the topic. everytime a broker restarts and if i restart my consumer service, it starts to read records which it has already read. e.g. before i restarted the consumer this was the state.
and consumer was sitting idle not receiving any messages as it has read all of them.
i restart my consumer, and all of a sudden it starts receiving messages which it has processed previously and here is the offset situation now.
also what is this LOG-END-OFFSET and LAG, looks like these are something to consider here.
note that it only happens when 1 of the broker gets restarted due to kubernetes shifting it to another node.
this is the topic configuration
Based on the info you posted, a couple of things that immediately come to mind:
The first screenshot shows a lag of 182, which means the consumer either was not running, or it has some weird configuration that made it stop consuming. Was it possible one of the brokers was down when the consumer stopped consuming?
On restart, the consumer finally consumed all the remaining messages, because it now shows lag of 0. This is correct, expected Kafka behavior.
Make sure that the consumer group name is not changing between restarts. Some clients default to "randomized" customer group names, which works as long as the consumer is not restarted.
We have a simple application. One micro service will send messages to be consumed by two other micro services. Of the two services, one is able to successfully process the messages and we don't see a lag there. The other service consistently fails and after 25th message no other message is consumed. Is there a reason for this?
The Kafka topic is created with one partition and one replication factor.
Service 1 - Working Fine:
Service Working Fine
Service 2 - Lag increasing:
Lag Increasing
Is there a configuration in Kafka that will make the consumer to stop consuming the messages after a particular amount of failure or what can we do to avoid this behavior?
you are saying a topic with one partition and you have two different services, I'm assuming different consumer groups names consuming from the same topic partition, one service is consuming fine and other one stop consuming at 25th message. In this case you can check in that consumer logs why it is not consuming, this could be malformed message where the consumer is able to consume.
here is how I'd debug
restart or redeploy the same consumer and see it is alway stop at 25th message
if step 1 is true, check 24th and 25th message and see the difference or reset offset to 26th to see, if consumer is moving forward after offset reset, then message 25th having some issue where consumer unable to consume.
How can i get all messages are published before subscribed a Topic In Kafka? I m using Kafka 2.12-2.3.0 Producer for sending message and Consumer to receive message. Actually i m building a chat application in java using Kafka. But problem is if a producer posted some message and a consumer subscribed later it wont get that messages. Please give some suggestion.
Your consumer has configuration option auto.offset.reset (exact name depends on the SDK you are using). Set it to 1 (Earliest). This will work when you connect for the first time.
Also you can explicitly assign start offset for every partition, like: consumer.Assign(offsets); - exact code depends on your scenario, but assigning 0 will ensure you get the earliest.