I'm sending a message from an SNS topic to an SQS. When I'm checking the body of the SQS message on my client, the whole of the message metadata is being sent in the SQS body.
I.E. if I'm sending a message "Hello World" from the topic, my client is receiving:
BenFlowers {
"Type" : "Notification",
"MessageId" : "84102bd5-8890-4ed5-aeba-c15fafc926dc",
"TopicArn" : "arn:aws:sns:eu-west-1:534706846367:HelloWorld",
"Message" : "hello World",
"Timestamp" : "2012-06-05T13:44:22.360Z",
"SignatureVersion" : "1",
"Signature" : "Qzh0qXhijBKylaFwc9PGE+lQQDwHGWkIzCW2Ld1eVrxNfSem4yyBTgouqGX26V0m1qhFD4RQcBzE3oNqx5jFhJfV4hN45FNcsFVnmfLPGNUTmJWblSk8f6znWgTy8UtK9xrTeNYzK59k3VJ4WTJ5kCEj+2vH7sBV15fAXeCAtdQ=",
"SigningCertURL" : "https://sns.eu-west-1.amazonaws.com/SimpleNotificationService-f3ecfb7224c7233fe7bb5f59f96de52f.pem",
"UnsubscribeURL" : "https://sns.eu-west-1.amazonaws.com/?Action=Unsubscribe&SubscriptionArn=arn:aws:sns:eu-west-1:534706846367:HelloWorld:8a3acde2-cb0b-4a56-9b9c-b75ed7307556"
}
This is a bit annoying as I am having to split the message body up on the other end. Speed is pretty important in this application so i would like to eliminate this. Is there any way to just send the message from the SNS and ignore the rest of the metadata?
Thanks,
Ben
SNS recently rolled out a feature that allows you to set 'raw message delivery' on an SNS topic.
http://docs.aws.amazon.com/sns/latest/dg/large-payload-raw-message.html
SNS limits the message size to 8KB, so there's no way to go around this problem using this service.
You could just send your message directly to SQS queue, where the limit is 64KB.
If your issue is the speed of reception of the messages you put in the queue, you could use SNS to notify the client it's time to make a request to SQS for messages to receive.
Related
I have 5 verticles: V{1..5}. All the verticles subscribed at "topic" and send message to "topic". Is there a way how to publish message in a way that it won't be sent to sender? For example, V1 publishes "message" and receives it as well. I want to avoid it. I need "message" sent by V1 to be delivered only to V{2..5}, but not to V1. Is it possible?
It's not possible out of the box. But you could add a header to the message which identifies the sender and filter messages in the consumer.
I am working on an approach where i am required to send a message back to SQS.
I don't want it to go as a new message as that will reset the approximateRecieveCount parameter which is required by the code.
Please note that
I cannot send a NACK to the queue as i am reading it as a batch of 10 messages, I want to manually post it back in certain cases for individual message and not as a batch.
The code I am trying to use
I tried setting the JMSMessageId but it is not possible as according to the documentation -
After you send messages, Amazon SQS sets the following headers and properties for each message:
JMSMessageID
JMS_SQS_SequenceNumber (only for FIFO queues)
The code i am using right now is
defaultJmsTemplate.send(destinationName, new MessageCreator() {
#Override
public Message createMessage(Session session) throws JMSException {
Message message = session.createTextMessage(errorMessage);
message.setJMSCorrelationID(transactionId);
if (destinationName.endsWith(".fifo")) {
message.setStringProperty("JMSXGroupID", property.getMessageGroup());
message.setStringProperty("JMS_SQS_DeduplicationId", java.util.UUID.randomUUID().toString());
}
return message;
}
});
}
Is there anything that i can set/use to make sure the message is not treated as a new message and the approximate receive count is maintained?
Yes. This can be done. As you are using JMS for SQS while setting up your consumer you can define an UNORDERED_ACKNOWLEDGE mode in your consumer session. By doing so if you do not acknowledge a particular message it will be redelivered after its visibility timeout expires and the approximateRecieveCount will be incremented. This will not impact your other messages in the same batch. One downside of this is if you are using the fifo queue and the all your messages have same group id then you next message will only be processed after this unacknowledged message ends up in dead letter queue. This will only happen after your message is retried for the Maximum Receives that you have set up in fifo queue configuration. Note : The key here is to not acknowledge a particular message.
Is there a way by which AWS SQS can call my REST API? Basically as soon as message is pushed to AWS SQS, I want to hear it and perform required action. I can schedule a listener that can pull messages every second but that won't be an optimizes solution and also the queue might be empty(sometimes).
Thanks In Advance!!
A couple of thoughts:
Use Publisher/Subscriber
Look into using a publisher-subscriber model with SNS/SQS, so that you publish a message to SNS and subscribe to it via SQS. If you absolutely need to handle a message as soon as it is published, you can publish to SNS and set another consumer in addition to your SQS subscription (such as a lambda subscriber that calls your Rest API?) to process it instead.
SQS Long Polling
Regarding SQS, it sounds like you would benefit from long polling. From the documentation:
Long polling helps reduce your cost of using Amazon SQS by reducing
the number of empty responses (when there are no messages available to
return in reply to a ReceiveMessage request sent to an Amazon SQS
queue) and eliminating false empty responses (when messages are
available in the queue but aren't included in the response):
Long polling reduces the number of empty responses by allowing Amazon
SQS to wait until a message is available in the queue before sending a
response. Unless the connection times out, the response to the
ReceiveMessage request contains at least one of the available
messages, up to the maximum number of messages specified in the
ReceiveMessage action.
Long polling eliminates false empty responses by querying all (rather than a limited number) of the servers.
Long polling returns messages as soon any message becomes available.
Also from the documentation, to enable long polling programmatically, use the following for any of these SQS actions:
ReceiveMessage: WaitTimeSeconds parameter
CreateQueue: ReceiveMessageWaitTimeSeconds attribute
SetQueueAttributes: ReceiveMessageWaitTimeSeconds attribute
Reference:
Publish–subscribe (PubSub) Pattern
SQS Documentation - Long Polling
Sounds like you would be much better of using SNS instead of SQS. What you are trying to get SQS to do, SNS was designed to do:
You can use Amazon SNS to send notification messages to one or more
HTTP or HTTPS endpoints. When you subscribe an endpoint to a topic,
you can publish a notification to the topic and Amazon SNS sends an
HTTP POST request delivering the contents of the notification to the
subscribed endpoint. When you subscribe the endpoint, you select
whether Amazon SNS uses HTTP or HTTPS to send the POST request to the
endpoint. If you use HTTPS, then you can take advantage of the support
in Amazon SNS for the following...
http://docs.aws.amazon.com/sns/latest/dg/SendMessageToHttp.html
In my JAVA application, I'm using Amazon SQS and SNS; I did the below steps:
Step 1: I pushed the message to SQS like,
SendMessageResult aSendMessageStatus = Amazon_SQS_Client.sendMessage(new SendMessageRequest().withQueueUrl(AWS_SQS_URL).withMessageBody(theRequestString));
Step 2: Created topic in SNS like,
CreateTopicResult createRes = Amazon_SNS_Client.createTopic(createReq);
Step 3: Now I am trying to send email by receiving the messages from SQS to 100 of customer.
Can someone advice me on how to subscribe the topic in SNS and send the emails to multiple email addresses.
For SNS to deliver the message, the 100 email address would have to subscribe to the topic. The email address will get confirmation message which they will have to respond to.
For your scenario, another option could be to use the queue service. It can be done in multiple ways. I have a setup as follows:
An application component sends a message to a queue
Another application component polls the queue, retrieves the message
From the message an email is composed and use SES service to deliver emails.
Another option is to use SNS -> SQS -> SES setup, where initial notification goes to SNS, and SNS delivers the notification to SQS.
The notification message itself need not be the complete email message. It could be just a reference to the content and people to which the content is to be delivered. Your application could take care of forming the complete message.
For a scenario where email is delivered to a general application user, I think SES is the right solution rather than SNS.
I have an webapp that is expected to fetch and display data from an External App which is accessible only via messaging (JMS).
So, if a user submits a request on a browser, the same HTTP request thread will have to interact with the Messaging system (MQ Series) such that the same request thread can display the data received from the Messaging System.
Is there a pattern I can make use of here? I saw some vague references on the net that use "Correlation ID" in this way:
Msg m = new TextMsg("findDataXYZ");
String cr_id = m.setCorrelationID(id);
sendQueue.send(m).
// now start listening to the Queue for a msg that bears that specific cr_id
Response r = receiverQueue.receive(cr_id);
Is there something better out there? The other patterns I found expect the response to be received asynchronously.. which is not an option for me, since I have to send the response back on the same HTTP request.
The request/reply messaging pattern is useful for your requirement. You typically use a CorrelationId to relate request & reply messages.
While sending request message you set JMSReplyTo destination on the message. Typically a temporary queue is used as JMSReplyTo destination. When creating a consumer to receive response use a selector with JMSCorrelationId, something like
cons = session.createConsumer(tempDestination,"JMSCorrelationId="+requestMsg.JMSMessageId);
At the other end, the application that is processing the request message must use the JMSReplyTo destination to send response. It must also use the MessageId of the request message and set it as CorrelationId of the response message.
First, open the response queue. Then pass that object to the set reply-to method on the message. That way the service responding to your request knows where to send the reply. Typically the service will copy the message ID to the correlation ID field so when you send the message, take the message ID you get back and use that to listen on the reply queue. Of course if you use a dynamic reply-to queue even that isn't neessary - just listen for the next message on the queue.
There's sample code that shows all of this. If you installed to the default location, the sample code lives at "C:\Program Files (x86)\IBM\WebSphere MQ\tools\jms\samples\simple\SimpleRequestor.java" on a Windows box or /var/mqm/toolsjms/samples/simple/SimpleRequestor.java on a *nix box.
And on the off chance you are wondering "install what, exactly?" the WMQ client install is downloadable for free as SupportPac MQC71.