I have a topic (let's call it "my-custom-topic") with 6 partitions and (actually) two consumers in it with the same group. The group is for a service, let's call it "myService". Time to time one of the services receives a REST request and then it will send an event to the "my-custom-topic". In this case I would like to let both services to receive the event. (yes, the one which sent it should also receive it). Later on if I will start other instances from "myService" I would also let them automatically receive this event.
So, shortly, I would like to let all my services in the same group to receive all the messages there. (once / message)
How can I achieve this with Kafka?
Thank you for helping me.
Only one consumer in consumer group will actually get the message. To achieve this kind of broadcast, you need to assign each consumer to each own consumer group.
You can get more details here How Kafka broadcast to many Consumer Groups
Related
There are 2 services with reading from the the same topic, the configurations are the same except for groupId, 1 partition, in the logs I see the same consumer configuration and successful connection. One service reads messages from the queue, the other does not. Of the differences found, there are logs in the working service:
Setting offset for partition topic-0 to the committed offset FetchPosition{offset=420, offsetEpoch=Optional.empty, currentLeader=LeaderAndEpoch
And in the second service there are no such. Tried different auto.offset.reset, doesn't help. The implementation of working with kafka in services is identical. What could be the problem?
Do both services read from the same topic?
If the topic has only 1 partition, then only 1 consumer will receive messages.
If you want to distribute messages across multiple consumers, then you'd have to specify more than 1 partition (more than number of consumers).
However, if you want all consumers to receive all messages, then you'd have to specify different consumer group id for each consumer (which in your case is: each service).
The problem was in the #Builder annotation for the domain class of the message, after replacing it with #SuperBuilder, everything began to work as it should. It's weird, I didn't have inheritance from this class.
How to consume bulk (something like prefetchCount = 10) messages in one shot in spring framework of rabbitMq?
Note - I am implementing Consumer not Listener
As of now I'm using
Message message = amqpTemplate.receive("Queue_Name");
But the problem with the above solution is, it fetches only one message in one shot
I'm curious why you want to process 10 at a time. Typically messages are discreet and processed individually. That's why RabbitMQ will only pass a single message to a given instance of a consumer at a time. A PrefetchCount of 10 will call the consumer 10 times, with one message each. If you have to process 10 messages at once for some reason, you would need to receive messages individually, acknowledge each one and store them in a collection as they're received. Then when your count = 10, start processing them.
I have a topic named mytopic. There are one producer and 2 consumer for this topic. What I need to do is filtering the messages that produces by the producer, according to their prefix. For example, if a message starts with 'a' prefix, then only the first consumer must take it. If it starts with 'b' prefix, then only the second consumer must take it.
I made a lot search, what I found is filtering messages that came from a topic and then send them to the different topics after filtering. But as said above, I need to do filtering over one topic. How can I do that in Kafka?
Allow both the consumers to consume all data, once you get the records filter them using java stream with the filter logic specific to the consumer.
In short, I mean just get the data as is and filter them using java code instead of doing it at Kafka Level.
Update:
If you want to filter at Kafka Level, you can use partitions, while sending message to kafka topic, send messages with prefix 'a' to Partition-1, and messsages with prefix 'b' to Partition-2.
Now, while consuming just consume that particular partition in respective consumers.
That's easy,and dont't need to write back to different topics.
'2 consumers' means 2 consumer groups or 2 consumer threads in 1 consume group?
I will talk it both.
If it is 2 consumer threads in 1 consumer group,you can use the message 'Key' field.
Kafka send same 'Key' messages to same 'Partition'.
For example, a message prefix 'a' with key field 'a',b message prefix 'b' with key field 'b',then Kafka will send the a message to 'Partition-1',b message to 'Partition-2'.
The consumer thread A can subscribe the specified 'mytopic-Partition-1'，thread B can subscribe 'mytopic-Partition-2' using the 'assign' method in class 'org.apache.kafka.clients.consumer.KafkaConsumer'.
If it is 2 consumer group,just subscribe the topic and filter in the code.If unsatisfactory,use the same method above.
The tricks is sending specific prefix message to specific 'Partition'.
If you really want filter,maybe you can use Kafka Connect plugin.
I've seen a lot of code and examples on how to send a message from the server/producer to the client, but I'm lost as to how to send a message from the client to the server. Does it follow the same format?
For example, say my server has 5 topics: A, B, C, D, E. My client is listening to only topic C.
So my server creates a queue and sends messages to subscribers/clients.
My client gets messages that are on topic C. It performs some computation, and I want to send a message back to the publisher (say an int for example). What are the required steps?
Do I need to set a destination/topic for the producer to listen to?
Does my producer automatically listen to all clients (once they connect)?
I think you may need to do a bit of reading around the various communication models available e.g. this on JMS Models.
You have started by using Topics, but your statements hint at a One-to-One relationship between sender and receiver. For this queues may be more appropriate. Topics are generally for publishing information where the publisher doesn't know (nor care) about the subscribers - and there can be many.
On the other hand Queuing is intended to deliver to a specific target. With this you can still do "fire-and-forget" (where the sender puts the message out to a queue and doesn't wait for the expected consumer to process it)
Or you can add response handling to that - essentially by adding a "parallel" response queue (where the initial sender will processes the response messages back from the consumer - usually in a different thread).
ActiveMQ and JMS have "reply-to" functionality - where the sender indicates on send the queue name where it wants to see responses come back to - and the consumer complies with that. The reply-to queue can be fixed or a temporary one that the sender creates up-front.
All depends on what model you are going for.
I want to build a broadcasting system. It consists of several groups. Each group has one User who can broadcast a message to the other members of the same group. What is the appropriate JMS architecture for this kind of system?
Should I use a topic with durable subscription? I don't know much about pub/sub messaging style, can topic have multiple subscription? If so, then each subscription represents a group in the broadcasting system. If not, should I use a queue whereas each message has a header specifying the group and then use a JMS selector to filter the message so that each member receive only messages from groups he's in?
Also, I am thinking of persisting the messages after consumption. I decided to make each message to expire after one hour, and each member should check the database for every message he misses. But, how can I fire an action upon message expiration?
Topics can be used... Topics can have multiple subscribers. Each group can use a different topic. The user can send message to that topic and all the subscribers would receive it.
The durable subscribers are required only if the subscribers can go offline for sometime and the messages for the subscriber shouldn't be lost.
Queue doesn't suite well in one-to-many scenarios. But if you have a pre-defined set of receivers you can use a queue for each one of them and route messages to that. But this is a overhead to route the messages to the receiver's queue. JMS selector idea you have suggested would work but for a queue only one client can receive a message. In topics it distributed to all the clients interested in that topic.
Normally one would persist data to database and not the message itself. So you can persist to database and then create the message for delivery.