Is it possible to validate / filter messages that are being sent to Kafka topic?
Like, I want to ensure that only valid clients / producers send messages to my topic. I can certainly perform validation on the consumer side by discarding invalid messages based on certain parameters / criteria. But, what if I want to do it before the messages are written into the topic.
Say, Kafka receives a message, performs some validation and accordingly decides if it needs to discard or write that message into a topic. Is this possible?
A short answer - current versions of Kafka has no support for such functionality out of the box. And since Kafka producers are designed to communicate with multiple brokers during single session, there is no easy way to implement such ad-hoc filtering.
There are couple of reasonable options still exists:
Use 2 topics: one "public" topic opened to everyone which will allow all messages, and another non-public "filtered" topic which will be populated by your own application with data from "public" after applying your filtering rules.
If you absolutely need to validate incoming messages before writing them down, then you could hide actual Kafka brokers behind some form of proxy application, which will do validation before writing messages into Kafka
Related
I need to build scalable application that have several Java applications (Spring RabbitMQ producers) that consume messages from other applications by HTTP protocol, calculating priority of message, and send them in to 'priority queue' that matches the params.
So far under load of hundreds of messages per second, one application works just fine, but there is need to scale up applications.
The problem is that I don't really understand how does RabbitMQ producers work with 'priority queues'. I've been searching information in RabbitMQ documentation and I found docs that says that every producer needs to get acks to make sure that messages have proceed successfully.
So the questions are
Docs says that priority of messages calculated under hood AMQP protocol, so do RabbitMQ will send acks to producer after the position for messages will be selected or before
How does messages will be treated if assume that we have 2 producers that produce 2 different messages with same priority to the same 'priority queue'
I will be appreciated for any hint that will help me with that!
I have a chat app that will create a queue for each user that is online and I'm trying to get all the queued messages for this user and the problem is that I only know the name of the queue through the message that comes through and therefore I cant use #RabbitListener to give it a queue name.
Is there any way that I can get all the message queued for a user other than using rabbitTemplate
convert ? since it only gives me one single message other than all of them
I would say it is better to look into a STOMP over WebSocket protocol which is supported as a plugin on RabbitMQ. It indeed creates an individual queue for every user and there is a mechanism to consume all the messages sent to that user.
See WebSocket support in Spring Framework: https://docs.spring.io/spring-framework/docs/current/reference/html/web.html#websocket
If you can't do that, you probably should look into a custom solution where you send a queue name to some static exchange for the consumer to be aware of such a new queue which you can add to the ListenerContainer at runtime (and remove respectively later). See more info in Spring AMQP: https://docs.spring.io/spring-amqp/docs/current/reference/html/#listener-queues
I am currently working on a PoC about Aeron MQ. As per PoC, i have 2 subscriptions and 1 publisher for a local channel with embedded media driver. The messages published are received in both the subscriptions whereas i would want to distribute the message in round-robin fashion. Is there a property or a way to achieve this?
Out the box, Aeron supports:
One-to-one (Single Publication to a single Subscription)
Many-to-one (Many Publications publishing to a single Subscription).
One-to-many (Single Publication publishing to many Subscriptions using either UDP multicast or MDC over UDP unicast).
In all cases, the receiver gets all messages from point that it joins the stream. There's nothing out of the box that would give you this round-robin semantic.
In theory, you could use the primitives provided by Aeron to build a central component that would distribute the message in round-robin fashion to a set of components that request work.
Depending on your use case, you may be better off considering a solution like RabbitMQ that provides this out of the box.
I understand JMS as depicted by the following diagram:
(source: techhive.com)
Is there any way for me to access the underlying database using JMS or some other thing? Further, the JDBC connections that the JMS server maintains, can I add new connections in it so as to access other databases also and do CRUD operations on them? If yes, how?
Where did you get this from?
Normally JMS is used to send messages to queue (or topics). You have message producers that push messages in the queue and message consumers consume them and process it.
In your exemple it seems that you have multiple queues. One for the messages that need to be processed, and one for each client to retrieve the result the processing of its messages.
With JMS Server you don't necessarily have a database behind. Everything can stay in memory, or can be written to files. You will need database server behind only if you configure your JMS server to be persistent (and to assure that even if server/application crash your messages won't be lost). But in that case you will never have to interact with the database. Only the JMS server will and you will interact with the JMS server sending and consuming messages.
I will be publishing to a single Activemq Topic and I will have many subscribers consuming from this Activemq. Some of my subscribers may connect at a later date, but when they do I want them to receive ALL MESSAGES ever published to that Activemq topic. How do I do this and what is this pub-sub type called where you get a full picture on first subscribe?
It's typically a lot better to create a separate initial load service. New clients connecting and wanting years of missed updates can trigger some sync from the source application and receive these message through some other channel (a queue for instance). Once up to sync, you simply use durable subscribers on your topic to guarantee that you miss no further updates.
ActiveMQ is not really built to store huge amount of data in the middle for long term. Kahadb is not like a regular database (although you can back it with a JDBC data source if you wish). Storing messages long term in MOM software is actually an anti-pattern.