I have set pubsub topic configuration as following:
Acknowledgement deadline 600 sec
max delivery attempt 10
dead letter topic is not enabled
My question is:
After 10 retry attempt, if pusub does not receive ack from subscriber, does it delete message from subscription or it will still be there in subscription till message retention period but will not be consumed.
And if it is there in subscription till retention period will it keep increasing oldest unacked message age
It is not possible to set a max delivery attempt count without setting a dead letter topic. The max delivery attempt count indicates the number of times to attempt delivery before the message is sent to the dead letter topic.
If the dead letter topic cannot be used because it was deleted or proper publish permission has not been set up, then the message will continue to be delivered to subscribers on the original subscription until it is acknowledged, the dead letter topic is available, or the retention period has passed. In this case, the oldest unacked message age will increase until one of these things happens.
Related
I have a spring cloud application running GCP PubSub messaging. I've got 2 message inbound channels that is subscribed to 2 different subscribers. The problem I face during load/stress test of the application is that, with a specific no.of threads set as below :
spring.cloud.gcp.pubsub.subscriber.executor-threads: 350
spring.cloud.gcp.pubsub.subscriber.parallel-pull-count: 2
spring.cloud.gcp.pubsub.subscriber.max-acknowledgement-threads: 700
when the processes pulled by messages of channel 1 are busy, I don't have sufficient threads for channel 2 to pull messages. The solution would be is to restrict/configure no.of threads for each channel. I am finding a very hard time to figure this out. Please do help me out here ! Below are the channels I was referring to :
#Bean
#ServiceActivator(inputChannel = "pubsubInputChannel1")
public MessageHandler extractionMessageReceiver() {
return message -> {
// do something
};
}
#Bean
#ServiceActivator(inputChannel = "pubsubInputChannel2")
public MessageHandler extractionMessageReceiver() {
return message -> {
// do something
};
}
Note, the subscriber thread remains busy until the end of a particular process pulled by a message.
I had the following problem: when there were a lot of messages and they queued up, the actuator health check for pubsub stopped working. My assumption was that all the executor threads were busy handling the messages and the check run into a deadline exceeded exception.
The following flow control properties helped me to fix the issue:
Config
Description
spring.cloud.gcp.pubsub.[subscriber,publisher.batching].flow-control.max-outstanding-element-count
Maximum number of outstanding elements to keep in memory before enforcing flow control
spring.cloud.gcp.pubsub.[subscriber,publisher.batching].flow-control.max-outstanding-request-bytes
Maximum number of outstanding bytes to keep in memory before enforcing flow control.
From https://docs.spring.io/spring-cloud-gcp/docs/1.1.0.M1/reference/html/_spring_cloud_gcp_for_pub_sub.html
When I set the max outstanding element count to 100 everything worked fine.
I think the outstanding messages get pulled with a stream. And with the properties above we can control that not all messages get processed at once. Instead, we split them up into, for example, 100 messages each. Maybe it will also switch between the channels. Quote from https://medium.com/google-cloud/things-i-wish-i-knew-about-google-cloud-pub-sub-part-2-b037f1f08318:
Note, streaming pull only guarantees flow control on a best-effort basis. Say you’ve noted your application can only handle 100 messages in any one period, so you set max outstanding messages to 100. The client will pause once it has pulled in 100 messages, which works most of the time. However, if you then publish 500 messages in a single publish batch, the client will receive all 500 messages at once but only be able to process 100 at a time, potentially leading to a growing backlog of expired messages. This is because streaming pull can’t split up messages from a single publish batch. To avoid this, either increase your number of subscribers or decrease your batch sizes to match subscriber message processing capacity while publishing.
Could these parameters maybe solve your problem?
I have a queue with 50 consumers and the prefetch count of around 100. All the consumer is from single JVM instance. So, when the application goes down, the messages were in READY state. The number of messages in ready state is equal to (prefetch_count * no. of consumers).
Now the question is, what will happen to the messages in READY state? Will it redelivered or will it get dead lettered?
They will be redelivered.
Messages will only be dead-lettered if the consumer specifically rejects (nacks) the message with requeue=false (and the queue is configured for dead-lettering).
I have a dead letter exchange which works as expected - when I nack a message it goes there:
#Override
public void onMessage(Message message, Channel channel) throws Exception {
// How to specify when the message will be put back to the original queue?
// This doesn't work.
message.getMessageProperties().setExpiration("3000");
channel.basicNack(message.getMessageProperties().getDeliveryTag(), false, false);
}
But what I cannot find is how to specify per-message when the message should be returned to the original queue. Please advise.
The expiration header is removed when the message is dead lettered. Secondly, messages do not automatically get sent back to your original queue from your deadletter queue.
It sounds like you want a retry system with per message wait times. This can be done but not the way you are doing it now.
The basic pattern is that you Ack your message and then send it to a "delay" exchange and queue. This queue sets it's dead letter exchange as your application's exchange. When a message arrives to the delay queue, it sits there until the message TTL is reached and gets dead lettered to your application exchange. However, this does not work when you have variable message expirations. Messages get dead lettered from the head of the queue so a message with a long expiration will block messages with shorter expirations behind it.
So you have various options to overcome this problem.
Have a few standard delay exchanges to select from which have different queue based TTLs and don't set message expiration.
Just before sending your message, declare a temporary delay exchange (that auto deletes) and queue (with a queue TTL that expires a few seconds after your message expires). Send your message and it will get dead lettered after the correct amount of time and then the exchange will auto delete and the queue will expire and be deleted. This could be very expensive if you have peaks of retries. However, if you name the temporary exchange and queue after the expiry time limit, then it will get reused by other messages with the same TTL being sent within the same period.
NServiceBus has an ingenious technique using multiple delay exchanges and queues: https://docs.particular.net/nservicebus/rabbitmq/delayed-delivery
I was wondering if someone here could clarify the following for me.
Let's say that I have a Topic, with 5 Durable Subscribers attached to it. Each subscriber was created with the following call (the name is obviously different for every one):
session.createDurableSubscriber(
(Topic) destination,
"NAME_n", // Unique for every subscriber
"PARTY_ID = '123'", // Message selector
true );
So, when I publish a message with PARTY_ID = '123', it will be retained in the Topic until all 5 durable subscribers got a copy, correct?
Now, let's say that we have 3 other durable subscribers, with message selector "PARTY_ID = '666'". When I publish a '666' message to the Topic, will the message be removed from the Topic after the 3 durable '666' subscribers receive the message, or will it keep it there until ALL subscribers receive the message, which in this case will never happen due to selector restrictions?
In other words, does message retention on a Topic take the message selector of the Message and Subscriber into account?
When a message is published, the messaging provider will put a copy of that message to all the subscribers that match the selection criteria specified by the subscribers. In your case a copy of the message will be put to 5 subscribers as they match the selector 'PARTY_ID=123'. Then message is discarded by the messaging provider. Similarly for selector 'PARTY_ID=666', messaging provider will copy the message to matching subscribers only and then discard the message.
Please note the messaging provider will put a copy of the message to the storage (typically a queue) assigned to the subscribers and will go on to process next message from publishers. It will not wait for the subscriber to consume the message from the queue.
Message retention or Retain Publication is another concept where the messaging provider will retain (or cache) a publication for topic until the next publication is received for the same topic. The next publication may be published in the next second or after 5 minutes or 5 hrs. Basically the time of next publication is unpredicted. This concept is quite useful when subscriber comes in after a publication has been maid and the next publication is made after sometime. With Retain publication, the a copy of the retained publication is delivered immediately to the subscriber so that the subscriber does not have to wait long for the next publication.
Let's assume that a publisher is publishing on score of a FIFA2014 ongoing match between The Netherlands and Spain on topic FIFA2014/MATCHUPDATE/NED-ESP. The publisher publishes a new message every time a goal is scored. Since the scoring of goal is unpredictable, messages are published with 'Retain Publication' property which tells the message provider to keep a copy of the message till the next message is published on FIFA2014/MATCHUPDATE/NED-ESP. Assume that there are two subscribers when the first publication was made and with score being 0:0. Now let's assume that a third subscriber comes in after the first publication was made, it will get a publication immediately with score being 0:0. It does not have to wait to know the score till a next goal is scored. See this link.
Hope I am clear.
we're using activemq as message queue of our Java stand-alone application. my problem is that based on the activemq web console, the queue has this certain number of messages enqueued and dequeued. however, based on sysout statements i added in the code, it seems that the application is consuming less than the number of messages displayed on the activemq web console. for example, on the activemq console, no. of messages enqueued and dequeued is around 1800. however, the number of messages dequeued as displayed on console (i increment a counter per message received) is only around 1700.
i really don't know where the approx. 100 messages went. so i'm thinking maybe i'll have some idea if i know how to make activemq log the message enqueued by the producer and dequeued by the consumer. is this possible? if yes, how can this be done?
enqueued == numbers of messages put into the queue since the last restart
dequeued == number of messages successfully processed by the consumers
the difference in the two numbers == number of messages in-flight, usually tracked by the "dispatched" counter. "in-flight" means sent to the consumer, but not yet ack'd.