I was going through the tutorial shared by RabbitMQ here
I am assuming that the client code below
while (true)
{
var ea = (BasicDeliverEventArgs)consumer.Queue.Dequeue();
if (ea.BasicProperties.CorrelationId == corrId)
{
return Encoding.UTF8.GetString(ea.Body);
}
}
Would receive all messages on the queue and will unnecessarily iterate through messages not designated for it. Is their anyway we can avoid it i.e we can modify the client to only receive the messages intended for it only.
The basic work that i intend to achieve through RabbitMQ is Request-Response pattern where a request would be received by web-service which will send data in a queue the data object would have a unique reference number . This would be received by an asynchronous tcp-client which will send data on a tcp/ip layer based on message it had received.
On receiving reply from the asynchronous channel of tcp/ip the channel would parse the data and respond back on the queue with the corresponding request reference number.
The RPC approach is well suited for it but the client code shared have this shortcoming would appreciate feedback on it.
Actually I didn’t understand well your aim, but when you create an RPC model, you have to create an “reply queue”, this queue is bound only to the client.
It means that you will receive back only the client messages, and not all messages.
Since the Rabbitmq RPC model is asynchronous you can execute more than one request without wait the responses and replies could not have the same publish order.
The correlation id is necessary to map your client requests with the replies, so there are not "unnecessarily" messages
hope it helps
Related
I am currently migrating our existing Spring asynchronous REST architecture to Spring's new WebFlux library and have a question around joining multiple requests so that they can listen for the same published response.
Use Case is as follows:
Client A connects to our web server and requests data
We hit our cache to check if we have the data there
We don't, so we go and retrieve this data (Client A has subscribed and waits for a response)
Client B connects to our web server and requests the same data (hits the same endpoint)
We check the cache, data is still not there
As we are already fetching this data for Client A we don't want to make another request, however, we also do not want to turn Client B away. Client B should be able to listen for the same information
How can Client B subscribe to the same response stream that Client A is waiting for?
"Client A has subscribed and waits for a response"
I suppose the request is coded as a Mono and client A sibscribes to it literally:
Subscriber<Response> clientA = ...
Mono<Response> request = makeRequest(...);
request.subscribe(clientA);
then clientB should subscribe the same way:
Subscriber<Response> clientB = ...
request.subscribe(clientB);
Moreover, the cache should contain not the previously saved response data, but the requests themselves, of type Mono<Response>. Then, if such a request is found in the cache, new clients simply subscribe to it, regardless of was that request already completed or not.
Is there a way by which AWS SQS can call my REST API? Basically as soon as message is pushed to AWS SQS, I want to hear it and perform required action. I can schedule a listener that can pull messages every second but that won't be an optimizes solution and also the queue might be empty(sometimes).
Thanks In Advance!!
A couple of thoughts:
Use Publisher/Subscriber
Look into using a publisher-subscriber model with SNS/SQS, so that you publish a message to SNS and subscribe to it via SQS. If you absolutely need to handle a message as soon as it is published, you can publish to SNS and set another consumer in addition to your SQS subscription (such as a lambda subscriber that calls your Rest API?) to process it instead.
SQS Long Polling
Regarding SQS, it sounds like you would benefit from long polling. From the documentation:
Long polling helps reduce your cost of using Amazon SQS by reducing
the number of empty responses (when there are no messages available to
return in reply to a ReceiveMessage request sent to an Amazon SQS
queue) and eliminating false empty responses (when messages are
available in the queue but aren't included in the response):
Long polling reduces the number of empty responses by allowing Amazon
SQS to wait until a message is available in the queue before sending a
response. Unless the connection times out, the response to the
ReceiveMessage request contains at least one of the available
messages, up to the maximum number of messages specified in the
ReceiveMessage action.
Long polling eliminates false empty responses by querying all (rather than a limited number) of the servers.
Long polling returns messages as soon any message becomes available.
Also from the documentation, to enable long polling programmatically, use the following for any of these SQS actions:
ReceiveMessage: WaitTimeSeconds parameter
CreateQueue: ReceiveMessageWaitTimeSeconds attribute
SetQueueAttributes: ReceiveMessageWaitTimeSeconds attribute
Reference:
Publish–subscribe (PubSub) Pattern
SQS Documentation - Long Polling
Sounds like you would be much better of using SNS instead of SQS. What you are trying to get SQS to do, SNS was designed to do:
You can use Amazon SNS to send notification messages to one or more
HTTP or HTTPS endpoints. When you subscribe an endpoint to a topic,
you can publish a notification to the topic and Amazon SNS sends an
HTTP POST request delivering the contents of the notification to the
subscribed endpoint. When you subscribe the endpoint, you select
whether Amazon SNS uses HTTP or HTTPS to send the POST request to the
endpoint. If you use HTTPS, then you can take advantage of the support
in Amazon SNS for the following...
http://docs.aws.amazon.com/sns/latest/dg/SendMessageToHttp.html
I have a Jetty endpoint receiving push-messages of different types from several data-providers - let´s call them type1, type2.
I also provide set of websocket-endpoints - one for each type - where clients can subscribe to push-updates of the different message-types (/ws_type1, /ws_type2).
The dataproviders and the clients are not the same.
Based on the type of incoming message, I am trying to get the camel route to push updates only to the clients subscribing to the specific messagetype.
My code is something like this:
from("jetty:http://0.0.0.0:8888/incoming_1?matchOnUriPrefix=true")
.log("incoming_1")
.to("websocket://ws_type1?sendToAll=true");
from("jetty:http://0.0.0.0:8888/incoming_2?matchOnUriPrefix=true")
.log("incoming_2")
.to("websocket://ws_type2?sendToAll=true");
from("jetty:http://0.0.0.0:8080/incoming?matchOnUriPrefix=true")
.log("incoming")
.to("websocket://woot?sendToAll=true");
from("websocket://ws_type1")
.log("ws_type1")
.to("websocket://ws_type1")
;
from("websocket://ws_type2")
.log("ws_type2")
.to("websocket://ws_type2")
;
The result of this is that ALL messages of ALL types are sent to ALL the connected clients - regardless if they are connected to the /ws_type1 or /ws_type2 endpoints.
Is it possible to filter the updates to different sets of clients? I.e. messages of type1 are pushed to the clients connected to /ws_type1, and messages of type2 are pushed to the clients connected to /ws_type2.
EDIT:
Using the code above, I am able to connect websockets to ws-endpoints "ws_type1", "ws_type2", and "woot". However - when I post data to the "/incoming_1" jetty-endpoint ALL three websockets receive the data, but only "incoming_1" is logged.
Am I missing something?
EDIT 2:
I looked at the camel-websocket source, and it seems like this is by design. There is no filtering on which websockets the message-contents are sent to, and the content is written directly to the socket. This also means that the camel-routes "ws_type1" and "ws_type2" in the example above will not be called when the server pushes data.
....
Collection<DefaultWebsocket> websockets = store.getAll();
Exception exception = null;
for (DefaultWebsocket websocket : websockets) {
try {
sendMessage(websocket, message);
...
Full source
I can't find documentation for reply processing with gateways and service activators.
If I have gateway which:
1) sends requests to channel ReqChannel
2) accepts replies on channel RepChannel
ReqChannel is connected to router, that routes incoming messages to one of some service activators, let say AServiceActivator and BServiceActivator and that service activators have a configured output-channel="RepChannel".
And if I execute more than one method call on gateway's interface asynchronously or simultaneously from different threads, how gateway will correlate incoming replies to actual service caller?
The gateway creates a temporary reply channel and puts it in the header of the message. This mechanism provides the necessary correlation because each message gets its own reply channel.
If the final consumer (say a service-activator) has no output-channel, the framework automatically sends the reply to the replyChannel header.
For this reason, it is generally not necessary to declare a reply-channel on the gateway for the final consumer to send to.
However, there are times when this is useful - such as if you want to wire-tap the reply channel, or make it a publish-subscribe channel, so the result goes to multiple places.
In this case (when there is a reply-channel on the gateway, and the final consumer sends a message there), the framework simply bridges the explicitly declared reply-channel to the temporary reply channel in the message header.
For this reason, it is critical to retain the replyChannel header in your flow. You can't send some arbitrary reply to a reply-channel, unless you include the original message's replyChannel header.
I have an webapp that is expected to fetch and display data from an External App which is accessible only via messaging (JMS).
So, if a user submits a request on a browser, the same HTTP request thread will have to interact with the Messaging system (MQ Series) such that the same request thread can display the data received from the Messaging System.
Is there a pattern I can make use of here? I saw some vague references on the net that use "Correlation ID" in this way:
Msg m = new TextMsg("findDataXYZ");
String cr_id = m.setCorrelationID(id);
sendQueue.send(m).
// now start listening to the Queue for a msg that bears that specific cr_id
Response r = receiverQueue.receive(cr_id);
Is there something better out there? The other patterns I found expect the response to be received asynchronously.. which is not an option for me, since I have to send the response back on the same HTTP request.
The request/reply messaging pattern is useful for your requirement. You typically use a CorrelationId to relate request & reply messages.
While sending request message you set JMSReplyTo destination on the message. Typically a temporary queue is used as JMSReplyTo destination. When creating a consumer to receive response use a selector with JMSCorrelationId, something like
cons = session.createConsumer(tempDestination,"JMSCorrelationId="+requestMsg.JMSMessageId);
At the other end, the application that is processing the request message must use the JMSReplyTo destination to send response. It must also use the MessageId of the request message and set it as CorrelationId of the response message.
First, open the response queue. Then pass that object to the set reply-to method on the message. That way the service responding to your request knows where to send the reply. Typically the service will copy the message ID to the correlation ID field so when you send the message, take the message ID you get back and use that to listen on the reply queue. Of course if you use a dynamic reply-to queue even that isn't neessary - just listen for the next message on the queue.
There's sample code that shows all of this. If you installed to the default location, the sample code lives at "C:\Program Files (x86)\IBM\WebSphere MQ\tools\jms\samples\simple\SimpleRequestor.java" on a Windows box or /var/mqm/toolsjms/samples/simple/SimpleRequestor.java on a *nix box.
And on the off chance you are wondering "install what, exactly?" the WMQ client install is downloadable for free as SupportPac MQC71.