I am currently working on a PoC about Aeron MQ. As per PoC, i have 2 subscriptions and 1 publisher for a local channel with embedded media driver. The messages published are received in both the subscriptions whereas i would want to distribute the message in round-robin fashion. Is there a property or a way to achieve this?
Out the box, Aeron supports:
One-to-one (Single Publication to a single Subscription)
Many-to-one (Many Publications publishing to a single Subscription).
One-to-many (Single Publication publishing to many Subscriptions using either UDP multicast or MDC over UDP unicast).
In all cases, the receiver gets all messages from point that it joins the stream. There's nothing out of the box that would give you this round-robin semantic.
In theory, you could use the primitives provided by Aeron to build a central component that would distribute the message in round-robin fashion to a set of components that request work.
Depending on your use case, you may be better off considering a solution like RabbitMQ that provides this out of the box.
Related
Is it possible to validate / filter messages that are being sent to Kafka topic?
Like, I want to ensure that only valid clients / producers send messages to my topic. I can certainly perform validation on the consumer side by discarding invalid messages based on certain parameters / criteria. But, what if I want to do it before the messages are written into the topic.
Say, Kafka receives a message, performs some validation and accordingly decides if it needs to discard or write that message into a topic. Is this possible?
A short answer - current versions of Kafka has no support for such functionality out of the box. And since Kafka producers are designed to communicate with multiple brokers during single session, there is no easy way to implement such ad-hoc filtering.
There are couple of reasonable options still exists:
Use 2 topics: one "public" topic opened to everyone which will allow all messages, and another non-public "filtered" topic which will be populated by your own application with data from "public" after applying your filtering rules.
If you absolutely need to validate incoming messages before writing them down, then you could hide actual Kafka brokers behind some form of proxy application, which will do validation before writing messages into Kafka
What happens behind the scene, when receiving messages with (spring or ejb) message listener container in ActiveMQ/HornetQ?
Does broker pushing messages to consumers? If so, how consumers register
themselves to broker?
Or consumers polling messages on the queue? If so, why each queue (in admin console) has a consumer-number field that shows number of registered consumers of the queue?
This link of O'Reilly book said:
The p2p messaging model has traditionally been a pull- or
polling-based model, where messages are requested from the queue
instead of being pushed to the client automatically. (The JMS
specification does not specifically state how the p2p and pub/sub
models must be implemented. Either one may use push or pull, but at
least conceptually pub/sub is push and p2p is pull).
You are not stating the protocol, since ActiveMQ and HornetQ are multi protocol brokers the exact implementation may vary a bit. However, most protocols except HTTP/REST based ones pushes messages to the client. It's not possible to achive high throughput without a push strategy on the wire protocol level.
The application level API allows for "polling", i.e. JMS MessageConsumer.receive, but that's really just a "sleep until a message is pushed" mechanism.
I have JMS implementation based on JBoss (to be precise, JBossMQ on JBoss4.2). There are 5 clusters, with each cluster having few nodes. One node in each of the cluster acts as master node. Out of the 5 clusters, one of the cluster is supposed to publish messages to a persistent Topic, and the other 4 clusters consumes those messages. The publishing and consuming is done by only the master node of each cluster.
I want to device a mechanism where the publisher knows that the message was consumed by all the subscribers or a subscriber knows that it has consumed all the messages produced by the publisher. How can this be achieved?
In principle, you use a JMS system in order to not care about that and only configure it the way you need. You could save state information in a shared resource like a database, but I wouldn't do it. Better use monitor features of the JMS system in order to track that. In case your application really needs to know about the successful processing of a message, then you could have a queue where process acknowledge go back to the sender.
For HornetQ, which you might use with JBoss, you'll find an example of a clustered topic here.
I will be publishing to a single Activemq Topic and I will have many subscribers consuming from this Activemq. Some of my subscribers may connect at a later date, but when they do I want them to receive ALL MESSAGES ever published to that Activemq topic. How do I do this and what is this pub-sub type called where you get a full picture on first subscribe?
It's typically a lot better to create a separate initial load service. New clients connecting and wanting years of missed updates can trigger some sync from the source application and receive these message through some other channel (a queue for instance). Once up to sync, you simply use durable subscribers on your topic to guarantee that you miss no further updates.
ActiveMQ is not really built to store huge amount of data in the middle for long term. Kahadb is not like a regular database (although you can back it with a JDBC data source if you wish). Storing messages long term in MOM software is actually an anti-pattern.
Using HornetQ (In JBoss AS 6.0) I would like to setup a JMS topic to which multiple clients can subscribe.
A producer periodically sends a message to this topic with a reply-to destination, to which all subscribers should reply.
The problem I'm having is that I'm not entirely sure how to check that all subscribers have indeed replied.
One solution could be that each subscriber first sends a message to the topic after subscription with its details (perhaps some GUID). The producer remembers these details and uses it to check later whether all subscribed clients have replied.
However, rather than inventing the wheel myself I would like to use something that already exists. This seems like a standard problem, but I could not find any existing solution.
You could use durable subscriptions, and then query the subscriptions and messages.
See http://hornetq.sourceforge.net/docs/hornetq-2.0.0.BETA5/user-manual/en/html/management.html#d0e5742
Note that usage of durable subscriptions and persistent messages will incur a performance penalty. You'll have to gauge the severity of the performance impact according to your specific needs.
JMS itself doesn't support this, it's too simple. If you didn't mind coupling your code to HornetQ, then you could use its native API to find out this stuff. Not ideal, but it's well written and has readable source code, so it wouldn't be too hard.