Say I have the following components:
Producer produces numbers and sends messages to Consumer
Both Producer and Consumer send messages to Monitor
Monitor, say randomly, decides when the produce / consume process should stop and sends a message to Stopper
Stopper then stops both Producer and Consumer cleanly
I know this is easy to accomplish in a mutable language such as Java. I know also this can be resolved by allowing partial mutability with interfaces, such as described here.
However, it's not a good practice to have cyclic dependencies even if possible. So, let's assume all references are constructor-injected and final:
Producer has final Consumer and final Monitor
Consumer has final Monitor
Monitor has final Stopper
Stopper has final Producer and final Consumer
I found references such as this, but they don't seem to apply.
How would one go about un-cycling this case and cases such as this in general? In other words, I'm mostly interested in how to accomplish not forming the cycles from a design standpoint. Any hints?
You're right, this won't work if all dependencies are final and injected via the constructor.
But may I ask, why do they have to be injected via the constructor? There is nothing wrong at the end of the day to use setters to wire up beans.
In fact, in Spring, beans are usually instantiated first and injected afterwards. So you could look at that approach.
Other than that, you could look at a different way to model your problem (that does not have circular dependencies).
For example, since you are using queues already to send messages between the producer and consumer, why not also send messages on queues to the monitor? The stopper could also send messages to the producer and consumer.
Or, as Taylor suggests, an ESB.
There are probably many other ways to design it, have a read about (for example) Apache Camel Enterprise Integration Patterns for some ideas.
Related
I have a situation in which Producer A writes on topics A,B and C however listener for topic C throws an exception. All writes are part of a transaction. I want to know if there is a way that all writes can be rolled back automatically, as if no there were no commits in the first place?
I don't think this can be achieved in Kafka out of the box. I would suggest to re-think the design since Kafka/ messaging system is not the best match for your requirement. Kafka consumers are meant to be independent business logic like a micro-service, even if one fails it should not affect the other. If its so critical you may consider a single topic/webservice with all required info in that topic/request and make the client transactional. Otherwise if non-critical(failure of a topic client is not affecting functionality of another topic client), then introduce some audit/alerting mechanism on top of clients to make sure that they are back online.
I want to publish a message on a EventBus channel and receive a response from everyone who's listening to that channel. The thing is, how to I know everyone has already responded if I don't know how many responses to expect?
I assume I need to know how many consumers there are in order to know that I already got all responses or not.
Is there any way to know how many consumers are currently "listening" to a Vert.x EventBus address?
EventBus works on a "best effort" basis, so even if you could track the number of subscribers, there is a chance that some of them would never return your message.
If you're still inclined to try it, there are a couple of ways I can thin of. None are really recommended, but I'll try to highlight the pros and cons anywya.
One is to get a reference to handlerMap inside EventBusImpl through reflection.
https://github.com/eclipse-vertx/vert.x/blob/master/src/main/java/io/vertx/core/eventbus/impl/EventBusImpl.java#L48
If you get it once, when your application starts, that should have much impact on the performance. Of course that will break at runtime if Vert.x team decides to as much as change the name of the field.
Another option is to use vertx.sharedData(), for example getLocalMap(). All consumers will add themselves to the map, and producer will check the map to figure out how many consumers are listening. The problem with that implementation:
Lots of logic to implement this
Unregistering consumers is hard (there's no guarantee that consumer will be able to unregister itself)
I am creating two apache camel (blueprint XML) kafka projects, one is kafka-producer which accepts requests and stores it in kafka server, and other is kafka-consumer which picks ups messages from kafka server and processes them.
This setup is working fine for single topic and single consumer. However how do I create separate consumer groups within same kafka topic? How to route multiple consumer specific messages within same topic inside different consumer groups? Any help is appreciated. Thank you.
Your question is quite general as it's not very clear what's the problem you are trying to solve, therefore it's hard to understand if there's a better way to implement the solution.
Anyway let's start by saying that, as far as I can understand, you are looking for a Selective Consumer (EIP) which is something that's not supported out-of-the-box by Kafka and Consumer API. Selective Consumer can choose what message to pick from the queue or topic based on specific selectors' values that are put in advance by a producer. This feature must be implemented in the message broker as well, but kafka has not such a capability.
Kafka does implement a hybrid solution between pure pub/sub and queue. That being said, what you can do is subscribing to the topic with one or more consumer groups (more on that later) and filter out all messages you're not interested in, by inspecting messages themselves. In the messaging and EIP world, this pattern is known as Array of Filters. As you can imagine this happen after the message has been broadcasted to all subscribers; therefore if that solution does not fit your requirements or context, then you can think of implementing a Content Based Router which is intended to dispatch the message to a subset of consumers only under your centralized control (this would imply intermediate consumer-specific channels that could be other Kafka topics or seda/VM queues, of course).
Moving to the second question, here is the official Kafka Component website: https://camel.apache.org/components/latest/kafka-component.html.
In order to create different consumer groups, you just have to define multiple routes each of them having a dedicated groupId. By adding the groupdId property, you will inform the Consumer Group coordinators (that reside in Kafka brokers) about the existence of multiple separated groups of consumers and brokers will use those in order to discriminate and treat them separately (by sending them a copy of each log message stored in the topic)...
Here is an example:
public void configure() throws Exception {
from("kafka:myTopic?brokers={{kafkaBootstrapServers}}" +
"&groupId=myFirstConsumerGroup"
.log("Message received by myFirstConsumerGroup : ${body}");
from("kafka:myTopic?brokers={{kafkaBootstrapServers}}" +
"&groupId=mySecondConsumerGroup"
.log("Message received by mySecondConsumerGroup : ${body}");
}
As you can see, I created two routes in the same RouteBuilder, not to say in the same Java process. That's a very bad design decision in most of the use cases I can think of, because there's no single responsibility, segregated concerns and they will not scale. But again, it depends on your requirements/context.
Out of completeness, please consider taking a look at all other Kafka Component properties, as there may be many other configurations of your interest such as the number of consumer threads per group.
I tried to stay high level, in order to initiate the discussion... I'll edit my answer in case of new updates from you. Hope I helped!
I am trying to figure out if I can switch from a blocking scenario to a more reactive pattern.
I have incoming update commands arriving in a queue, and I need to handle them in order, but only those regarding the same entity. In essence, I can create as many parallel streams of update events as I wish, as long as no two streams contain events regarding the same entity.
I was thinking that the consumer of the primary queue would possibly be able to leverage amqp's routing mechanisms, and temporary queues, by creating temporary queues for each entity id, and hooking a consumer to them. Once the subscriber is finished and no other events regarding the entity in question are currently in the queue, the queue could be disposed of.
Is this scenario something that is used regularly? Is there a better way to achieve this? In our current system we use a named lock based on the id to prevent concurrent updates.
There are at least 2 Options:
A single queue for each entity
And n Consumers on one Entity-Queue.
One queue with messages of all entities. Where the message contains data what it is for an entity. You could than split this up into several queues (One AMQP-Queue for one type of entity) or by using a BlockingQueue implementation.
Benefits of splitting up the Entities in qmqp-queues
You could create an ha-setup with rabbitmq
You could route messages
You could maybe have more than one consumer of an entity queue if it
is necessary someday (scalability)
Messages could be persistent and therefore recoverable on an
application-crash
Benefits of using an internal BlockingQueue implementation
It is faster (no net-io obviously)
Everything has to happen in one JVM
Anyway it does depend on what you want since both ways could have their benefits.
UPDATE:
I am not sure if I got you now, but let me give you some resources to try some things out.
There are special rabbitmq extensions maybe some of them can give you an idea. Take a look at alternate exchanges and exchange to exchange bindings.
Also for basic testing, I am not sure if it covers all rabbitmq features or at all all amqp features but this can sometimes be usefull. Keep in mind the routing key in this visualization is the producer name, you can also find there some examples. Import and Export your configuration.
I'm trying to write a multithreaded implementation for JMS message processing from a queue.
I've tried with DefaultMessageListenerContainer and SimpleMessageListenerContainer classes.
The problem I have is that it seems like just a single instance of the MessageListener class gets ever instantiated, no matter how I configure it. This forces me to unnecessarily write stateless or thread-safe MessageListener implementations, since I have the ListenerContainer configured to use multiple threads (concurrentConsumers=8).
Is there an obvious solution to this that I'm overlooking?
This is by design. The MessageListener is a dependency that you inject into Spring - it has no way of instantiating new ones.
This forces me to unnecessarily write stateless or thread-safe messageListener implementations
You make that sound like a bad thing. Making your MessageListener is a very good idea, Spring just removes the temptation to do otherwise.
Maybe this answer is too late, but it may benefit others who're searching for it. In short, the answer is using CommonsPoolTargetSource and ProxyFactoryBean.
Check out this link for details: http://forum.springsource.org/showthread.php?34595-MDB-vs-MDP-concurrency
If you want to do something similar for topic, check this: https://stackoverflow.com/a/12668538/266103
Configuring "concurrentConsumers" is just enough to process messages concurrently. This doesn't mean you will have "n" instances of MessageListenerContainer. The MessageListenerContainer may span "tasks" internally to process the messages. Optionally, you may have to configure your logging accordingly to see the information associated with the underlying tasks/threads.
See "Tuning JMS message consumption in Spring" for more details.