I am currently working with SNS and SQS to integrate disparate remote systems. The producer sends messages to an AWS SNS with a SQS subscribed. The consumer is a Spring Boot application with spring integration enabled that polls the SQS with an #SqsListener (default configuration with no tweaking). All this works fine.
The requirement is to process those messages in the right order mostly driven by the chronological creation time from the producer perspective. And as some of they could be dependent I have to process them one by one taking into account the original order.
The problem is that I am aware that SQS does not guarantees that those messages arrive in order when the Listener polls the SQS. I have probe this by programmatically sending a couple of messages to the SNS in the right order I want them to be processed and receive those messages in a slightly different order within the SqsListener.
To try to deal with this unwanted effect I put in place a Priority Channel right after the SqsListener to buffer those messages and let this channel reorder the messages.
Would this be the right approach to process standard SQS messages in order? Should I tweak the Listener config, for example to change it for a Long Polling?
Related
I need to build scalable application that have several Java applications (Spring RabbitMQ producers) that consume messages from other applications by HTTP protocol, calculating priority of message, and send them in to 'priority queue' that matches the params.
So far under load of hundreds of messages per second, one application works just fine, but there is need to scale up applications.
The problem is that I don't really understand how does RabbitMQ producers work with 'priority queues'. I've been searching information in RabbitMQ documentation and I found docs that says that every producer needs to get acks to make sure that messages have proceed successfully.
So the questions are
Docs says that priority of messages calculated under hood AMQP protocol, so do RabbitMQ will send acks to producer after the position for messages will be selected or before
How does messages will be treated if assume that we have 2 producers that produce 2 different messages with same priority to the same 'priority queue'
I will be appreciated for any hint that will help me with that!
I have a chat app that will create a queue for each user that is online and I'm trying to get all the queued messages for this user and the problem is that I only know the name of the queue through the message that comes through and therefore I cant use #RabbitListener to give it a queue name.
Is there any way that I can get all the message queued for a user other than using rabbitTemplate
convert ? since it only gives me one single message other than all of them
I would say it is better to look into a STOMP over WebSocket protocol which is supported as a plugin on RabbitMQ. It indeed creates an individual queue for every user and there is a mechanism to consume all the messages sent to that user.
See WebSocket support in Spring Framework: https://docs.spring.io/spring-framework/docs/current/reference/html/web.html#websocket
If you can't do that, you probably should look into a custom solution where you send a queue name to some static exchange for the consumer to be aware of such a new queue which you can add to the ListenerContainer at runtime (and remove respectively later). See more info in Spring AMQP: https://docs.spring.io/spring-amqp/docs/current/reference/html/#listener-queues
I'm using rabbitmqclient for RabbitMQ (from Scala). I subscribe to a queue via DefaultConsumer and consume the messages from few instances concurrently.
The problem is that when the first consumer starts, it immediately takes all existing messages from the queue, so other nodes will consume only newer messages. I'd like to configure the consumers to take, say, not more than 10 messages at a time. It's definitely possible to rewrite it using pull-based API and manage back pressure manually, but I'd like to avoid it.
I am pretty new to RabbitMQ, I want to to consume multiple messages from RabbitMQ so that work can be done parallely also sending acknowledgement only when any of the actor has finished it's task so as not to loose messages. How should I proceed, I want to use spring support for AKKA.
Can I use an actor as a consumer or it should be a plain consumer that can consume multiple messages without sending acknowledgement for any of the message or it should be that I have multiple classes/threads working as consumer instantiated to listen a single message at a time than calling actor (but that would be as if it had no actor or parallelism via AKKA model).
I haven't worked with RabbitMQ per se, but I would probably designate one actor as a dispatcher, that would:
Handle RabbitMQ connection.
Receive messages (doesn't matter if one-by-one or in a batch for efficiency).
Distribute work between worker actors, either by creating a new worker for each message, or by sending messages to a pre-created pool of workers.
Receive confirmation from worker once task is completed and results are committed, and send acknowledgement back to RabbitMQ. Acknowledgment token may be included as a part of worker's task, so no need to track the mappings inside the dispatcher.
Some additional things to consider are:
Supervision strategy: who supervises the dispatcher? Who supervises the workers? What happens if they crash?
Message re-sends when running in a distributed environment.
We're using a Tibco client implementation of the JMS API. We have a MessageListener with an onMessage() implementation.
Is there a way with the Tibco client to inspect past (received) messages in the queue? (I realise this totally ignores the logical concept of a queue - I wondered if the queue implementation provided this workaround.)
No. Not for "past" messages.
Messages acknowledged by the receiver are removed from the queue - as their "function" is already done.
You could have a Listener configured to persist your messages in some DB or file - but for future messages.
A client uses a QueueBrowser object to look at messages on a queue without removing them.
#hawkeye Its not possible to browse messages from the past... At any point of time , you can browse destinations only for the pending messages.
There is no way for you browse all the received messages as EMS server usually deletes the message once it has delivered ( acknowledged) for the given delivery mode.
One possible way is to a send copy of the messages to another queue (without any receivers) before actually confirming the messages.
Also it depends on your acknowledgement mode and logic involved.