Problem:
In my api-gateway I call other microservices using a message broker and wait for response using method, annotated using Spring #MessageHandler. When response arrives how am I suppose to connect it with the waiting http request?
My solution is to hang request to CompletableFuture, generate a unique key and save the CompletableFuture with the key to the Spring Data Key-Value (or other in-memory) database and than call other services via Message Broker and send my request key along with the data. When async response arrives to the MessageHandler, check the in-memory database for my request key that is also part of the response and complete the stage.
If there are more nodes in cluster, I'll use a topic and check every key-value instance and complete stage only on the correct node, since I don't want to use any distributed cache for that.
Is it a good solution? Or how do you handle this particular problem?
Related
I am subscribed to a Azure servicebus topic made by an external department. The way I want the code to work is as follows:
Trigger an http endpoint that starts processorClient and listens to the topic.
Fetches one message
Does required actions to that message
Closes processorClient connection.
Repeat
I am using the ServiceBusProcessorClient class as shown in the following documentation Receive messages from a subscription
Is there a way to utilize this code in order to only fetch one message from the topic at a time before calling the processorClient.close();
I have tried using setting maxConcurrentCalls and setting prefetchCount(0).
There is a remote server keep bringing about the data in JSON format. Here is a REST API named
http://192.168.1.101:8000/v1/status,and if I want to collect the data continuously in Spring Boot.Here is a possible JSON from the REST API:
{
"run-status": 0,
"opr-mode": 0,
"ready": false,
"not-ready-reason": 1,
"alarms":["ps", "prm-switch"]
}
I want to keep collecting or just subscribe the REST API, if there is a JSON and then collect it.
There are two main approaches of achieving what you are looking for:
Polling - If this service already exists and you do not have control
over the code, then this might be your only option. You constantly
poll the given URL to check if data has been changed.
In spring, you can use #Scheduled annotation to execute and poll
at any given frequency (using cron expression or fixed delays).
https://www.baeldung.com/spring-scheduled-tasks - provides a detail
of how to create a scheduled tasks.
Webhooks - If you have control over your server code, you can use
webhooks to notify subscriber about availability of data. It is a
callback mechanism where caller will receive a notification about
data changes on the server, and subscriber can then call server to
fetch data immediately.
More about Polling and Webhooks can be found on this URL: https://dzone.com/articles/webhooks-vs-polling-youre-better-than-this-1
Make a "while" cycle what calls your function then goes to sleep (if needed) for the time you want.
Or just while (true) {}
I am currently migrating our existing Spring asynchronous REST architecture to Spring's new WebFlux library and have a question around joining multiple requests so that they can listen for the same published response.
Use Case is as follows:
Client A connects to our web server and requests data
We hit our cache to check if we have the data there
We don't, so we go and retrieve this data (Client A has subscribed and waits for a response)
Client B connects to our web server and requests the same data (hits the same endpoint)
We check the cache, data is still not there
As we are already fetching this data for Client A we don't want to make another request, however, we also do not want to turn Client B away. Client B should be able to listen for the same information
How can Client B subscribe to the same response stream that Client A is waiting for?
"Client A has subscribed and waits for a response"
I suppose the request is coded as a Mono and client A sibscribes to it literally:
Subscriber<Response> clientA = ...
Mono<Response> request = makeRequest(...);
request.subscribe(clientA);
then clientB should subscribe the same way:
Subscriber<Response> clientB = ...
request.subscribe(clientB);
Moreover, the cache should contain not the previously saved response data, but the requests themselves, of type Mono<Response>. Then, if such a request is found in the cache, new clients simply subscribe to it, regardless of was that request already completed or not.
Is there a way by which AWS SQS can call my REST API? Basically as soon as message is pushed to AWS SQS, I want to hear it and perform required action. I can schedule a listener that can pull messages every second but that won't be an optimizes solution and also the queue might be empty(sometimes).
Thanks In Advance!!
A couple of thoughts:
Use Publisher/Subscriber
Look into using a publisher-subscriber model with SNS/SQS, so that you publish a message to SNS and subscribe to it via SQS. If you absolutely need to handle a message as soon as it is published, you can publish to SNS and set another consumer in addition to your SQS subscription (such as a lambda subscriber that calls your Rest API?) to process it instead.
SQS Long Polling
Regarding SQS, it sounds like you would benefit from long polling. From the documentation:
Long polling helps reduce your cost of using Amazon SQS by reducing
the number of empty responses (when there are no messages available to
return in reply to a ReceiveMessage request sent to an Amazon SQS
queue) and eliminating false empty responses (when messages are
available in the queue but aren't included in the response):
Long polling reduces the number of empty responses by allowing Amazon
SQS to wait until a message is available in the queue before sending a
response. Unless the connection times out, the response to the
ReceiveMessage request contains at least one of the available
messages, up to the maximum number of messages specified in the
ReceiveMessage action.
Long polling eliminates false empty responses by querying all (rather than a limited number) of the servers.
Long polling returns messages as soon any message becomes available.
Also from the documentation, to enable long polling programmatically, use the following for any of these SQS actions:
ReceiveMessage: WaitTimeSeconds parameter
CreateQueue: ReceiveMessageWaitTimeSeconds attribute
SetQueueAttributes: ReceiveMessageWaitTimeSeconds attribute
Reference:
Publish–subscribe (PubSub) Pattern
SQS Documentation - Long Polling
Sounds like you would be much better of using SNS instead of SQS. What you are trying to get SQS to do, SNS was designed to do:
You can use Amazon SNS to send notification messages to one or more
HTTP or HTTPS endpoints. When you subscribe an endpoint to a topic,
you can publish a notification to the topic and Amazon SNS sends an
HTTP POST request delivering the contents of the notification to the
subscribed endpoint. When you subscribe the endpoint, you select
whether Amazon SNS uses HTTP or HTTPS to send the POST request to the
endpoint. If you use HTTPS, then you can take advantage of the support
in Amazon SNS for the following...
http://docs.aws.amazon.com/sns/latest/dg/SendMessageToHttp.html
I was going through the tutorial shared by RabbitMQ here
I am assuming that the client code below
while (true)
{
var ea = (BasicDeliverEventArgs)consumer.Queue.Dequeue();
if (ea.BasicProperties.CorrelationId == corrId)
{
return Encoding.UTF8.GetString(ea.Body);
}
}
Would receive all messages on the queue and will unnecessarily iterate through messages not designated for it. Is their anyway we can avoid it i.e we can modify the client to only receive the messages intended for it only.
The basic work that i intend to achieve through RabbitMQ is Request-Response pattern where a request would be received by web-service which will send data in a queue the data object would have a unique reference number . This would be received by an asynchronous tcp-client which will send data on a tcp/ip layer based on message it had received.
On receiving reply from the asynchronous channel of tcp/ip the channel would parse the data and respond back on the queue with the corresponding request reference number.
The RPC approach is well suited for it but the client code shared have this shortcoming would appreciate feedback on it.
Actually I didn’t understand well your aim, but when you create an RPC model, you have to create an “reply queue”, this queue is bound only to the client.
It means that you will receive back only the client messages, and not all messages.
Since the Rabbitmq RPC model is asynchronous you can execute more than one request without wait the responses and replies could not have the same publish order.
The correlation id is necessary to map your client requests with the replies, so there are not "unnecessarily" messages
hope it helps