I have two microservices. Service #1 puts a certain object in the queue (a table in the database) that needs to be processed. After that, in service #2, the sheduler takes new records from the queue every few seconds and processes them, and then saves the result as json to the database. The question is, how do I notify service #1 about the result of processing?
You can achieve this using any messaging brokers activeMq or rabbitMq. Please refer: https://spring.io/guides/gs/messaging-rabbitmq/
Please make use of a rabbitmq receiver to fetch the notification and trigger the processing on you service 1 appropriately
You can set up Kafka for your microservice communication. For notifying service #1 you can send an event from service #2 through Kafka.
Suppose, from service #2 you'll send an event named "PROCESSING_DONE" and service #1 will listen to this event and can do the further processing.
This all will take place in realtime.
Related
I have to build a microservice that will listen to multiple queues, read the messages, and insert them into a database. There can be 1000's queues. If anyone can help what will be the best and clean way to scale up and down the service. How can I pass the queues name dynamically using docker and for example, service1 would only listen to queue1 and queue2 whereas after load increases another instance of the service is up which will listen to queue3 and process its messages? Each queue is independent and should be processed in an ordered way. Basically, you can consider having if we have 100 customers we will have 100 queues. Sometimes the queues for one customer get more messages for that case If we can spawn a new instance and separate that customer id queue to be processed on a different service that will be faster for that customer. As the load decreases one service can process all the queues.
Also, is there any way to dynamically pass the queues to listen/remove from the binding for a service without restarting?
I have a topic (let's call it "my-custom-topic") with 6 partitions and (actually) two consumers in it with the same group. The group is for a service, let's call it "myService". Time to time one of the services receives a REST request and then it will send an event to the "my-custom-topic". In this case I would like to let both services to receive the event. (yes, the one which sent it should also receive it). Later on if I will start other instances from "myService" I would also let them automatically receive this event.
So, shortly, I would like to let all my services in the same group to receive all the messages there. (once / message)
How can I achieve this with Kafka?
Thank you for helping me.
Only one consumer in consumer group will actually get the message. To achieve this kind of broadcast, you need to assign each consumer to each own consumer group.
You can get more details here How Kafka broadcast to many Consumer Groups
I'm working on an application in microservices architecture usingrabbitmq as messaging system.
calls between microservices are asynchronous http requests and each service is subscribed on specific queues
my question is seen that the calls are stateless, how to guarantee the parallelisation of the message commation not by routing-key in rabbitmq queue but by http call itself, that is to say for n call every service must be able to listen to only needed messages .
Sorry for the ambiguity, I'm trying to explain further:
The scenario is that we are in a micro service architecture, due to huge data response the call service will receive the answer in the listener rabbitmq queue.
So let's imagine that two calls are made simultaneously and both queries start loading data into the same queue, the calling service is waiting for messages and adds the received messages but cannot differentiate between the data of caller 1 and caller 2.
Is there a better implementation for the listener
Not sure I understood the question completely, but here is what I can suggest based on the description:
If each service is hooked to a particular listener and you don't want to associate a Routing-Key for the Queue+Listener integration, then can you try having header arguments. [You can use a QueueBuilder.withArguments API to set specific Header values that the Queue is supposed to listen to]
There needs to be a mechanism through which an exchange will bind to a particular queue and consequently to a Listener service.
Publisher -> Exchange ---> (with headers) binds to Queue -> Listener
I have the following setup: a number of devices send data via HTTP to my backend, where multiple instances a receiver component are running. I need to process the data and then send it to another external partner, who needs this data in timestamp order. So I came up with the following architecture:
There are n receiver instances running, with a load balancer in front of them, so they potientially get data from all devices. These instances process each incoming data by adding some information and then put the data into a Redis Sorted Set (there is one for each device). After this they send a message (via Redis) about how many data entries are currently in the set.
There are m processing instances whose task is it to send the data to the external partner. They listen to the messages sent by the receivers and if the number of entries inside a set is larger than some threshold, they retrieve the data from the queue, add some other information and then send it to the external partner.
The problem I have is the timestamp order requirement. I have n and m instances, each one running multiple threads. For the processing instances, who all receive the messages from the receiver, I thought about doing the retrieval of the data from the set and sending it to the external partner inside a shared Redis lock for the queue associated with the message (and the respective device). But currently there are multiple Spring Integration steps that are part of the processing flow: get the data from the queue -> transform it for sending -> send it via an HTTP outbound channel. I thought about using a lock that is obtained in the first step (getting the data from the queue) and released in the last step (after sending it via the outbound channel). In case of an error the lock would be released in the error processing step.
Are there any ideas for alternatives to this? I was thinking about sending the lock as part of the message header through the remaining flow and then release it at the end.
Since you say about ordering, you should consider to use PriorityChannel or Resequencer to reorder records before sending to the external partner.
Both of them can be configured with the shared MessageStore.
After reading some document of JMS, I totally puzzled by the phrase synchronous and asynchronouns.
See this page: http://docs.oracle.com/cd/E19798-01/821-1841/bncdq/index.html
Synchronous
You use the receive method to consume a message synchronously.
You can use this method at any time after you call the start method:
connection.start();
Message m = consumer.receive();
connection.start();
Message m = consumer.receive(1000); // time out after a second
To consume a message asynchronously, you use a message listener, described in the next section.
Asynchronous
JMS Message Listeners
A message listener is an object that acts as an asynchronous event handler for messages. This object implements the MessageListener interface, which contains one method, onMessage. In the onMessage method, you define the actions to be taken when a message arrives.
You register the message listener with a specific MessageConsumer by using the setMessageListener method. For example, if you define a class named Listener that implements the MessageListener interface, you can register the message listener as follows:
Listener myListener = new Listener();
consumer.setMessageListener(myListener);
I have two questions:
As what I understood, the nature of JMS is asynchronous. Producer publishes messages to the queue/topic, it doesn't need to wait consumer. This is asynchronous behaviour. How can it be "synchronous"?
If the "mesageListener" is asynchronous, but in my test with spring-jms, I found it always running in a thread. That means, if I write Thread.sleep(2000) in onMessage, it have to be wait 2 seconds before processing next message. Is it "asynchronous"?
If you understand it better like this, consumer.receive() uses a pull model: you read from a queue and are blocked waiting for this message until it comes, or some timeout has elapsed.
Using a listener uses a push model: you register a listener and, when a message comes in, the listener is called, in a separate thread.
Everything is done in a thread in Java, and the listener call is no exception. Whether the listener message handling prevents the processing of other messages in the queue depends on how many threads are dedicated to message processing. If you configure Spring to use a pool of 5 threads to process messages asynchronously, then 5 listeners will be able to process messages in parallel.
Like I understand this:
asynchronous - MessageListener: Use this on a server that listens to a queue. When a message arrives, then deal with it immediately. The server keeps listening to this queue.
synchronous - consumer.receive(1000): Use this on a client applications that now and then needs to check if a message is intend for this client. Example: poll every 60 seconds. This only opens a connection to the server shortly. The 1000 milliseconds will keep this connection open. If a message arrives within these 1000 milliseconds, then the message is consumed and the connection is closed.
You are looking at it end-to-end: from publisher to the consumer. Yes, it is asynchronous delivery from publisher to consumer irrespective of Sync/Async consumer. However Sync/Async in your question is for consumer only, i.e from the JMS broker (eg: ApacheMQ) to the consumer. As others have pointed out, Sync consumers pull messages sequentially from the broker and are waiting for messages. Async consumers register a callback where messages pushed to them (onMessage). Async consumers can go about doing other things while these messages are delivered to them asynchronously from the JMS broker.
I understand synchronous/asynchronous differently.
Synchronous: Caller(Sender) has to wait till the response from consumer has been received(till the time-out) -- request/reply pattern
Asynchronous: Caller(Sender) just post message and continue with its work, while the consumer processes as soon as the message reaches it -- one way request
Any MOM(Message Oriented Middle ware) follows service activator pattern which promotes asynchronous communication. One of my project has implemented a framework around JMS to make communication really synchronous.
Any message has 2 parts.
a. Metadata attributes
b. Payload
Set attribute "reply-to-queue" to a randomly generated value
Make sure the MOM framework creates temporary queue with name from #2
Make sure the sender spawns thread, which listens to temporary queue created in #3
Publish message and block sender till it receives message to temporary queue
Make sure the consumer intercepts "reply-to-queue" header and publishes response to it
This is one of the ways to make MOM based communication acts like synchronous. You may find other implementations like request-reply mechanism.