I'm trying to read up and understand 3 fundamental methods in the RabbitMQ Java client:
Channel#basicConsume
Channel#basicPublish; and
DefaultConsumer#handleDelivery
These methods have several arguments that are cryptic and mysterious, and although the Javadocs do provide some explanation of what they are, don't really make it clear/obvious as to what these arguments do:
Channel#basicConsume
consumerTag - a client-generated consumer tag to establish context
noLocal - true if the server should not deliver to this consumer messages published on this channel's connection
exclusive - true if this is an exclusive consumer
arguments - a set of arguments for the consumer
Channel#basicPublish
exchangeName - the exchange to publish the message to
routingKey - the routing key
DefaultConsumer#handleDelivery
envelope - packaging data for the message
These methods, and using them correctly, are crucial to using RabbitMQ in its simplest form (basic publishing & consuming of messages to and from a queue). Until I understand what these arguments are - and what they imply/do on the server-side - I'm stuck and unsure of how to proceed with the library.
Can some battle-weary RabbitMQ veteran help a newbie like myself understand these 7 method arguments, and what they are used for? The Javadoc explanations just aren't clear enough. For example: "arguments - a set of arguments for the consumer". What?!?! Or: "exclusive - true if this is an exclusive consumer"...well what's an exclusive consumer?!?! Etc. Thanks in advance!
Follow below two link :-
https://www.rabbitmq.com/ttl.html
http://www.rabbitmq.com/amqp-0-9-1-quickref.html
Java creates a queue in which messages may reside for at most 60 seconds:
Map<String, Object> args = new HashMap<String, Object>();
args.put("x-message-ttl", 60000);
channel.queueDeclare("myqueue", false, false, false, args);
To throughly understand the implementation of RabbitMQ you might have to go through the AMQP protocol specification and reference. In general irrespective of the language the specification should explain all the parameters
Below link explains about consumer-tag
Exclusive:
Exclusive queues may only be accessed by the current connection, and are deleted when that connection closes. Passive declaration of an exclusive queue by other connections are not allowed.
The server MUST support both exclusive (private) and non-exclusive (shared) queues.
The client MAY NOT attempt to use a queue that was declared as exclusive by another still-open connection. Error code: resource-locked
Customer tag:
Specifies the identifier for the consumer. The consumer tag is local to a channel, so two clients can use the same consumer tags. If this field is empty the server will generate a unique tag.
The client MUST NOT specify a tag that refers to an existing consumer. Error code: not-allowed
The consumer tag is valid only within the channel from which the consumer was created. I.e. a client MUST NOT create a consumer in one channel and then use it in another. Error code: not-allowed
AMQP extensions to understand additional Args (Map)
http://www.rabbitmq.com/extensions.html
Example - TTL:
Per-Queue Message TTL
The x-message-ttl argument to queue.declare controls for how long a message published to a queue can live before it is discarded. A message that has been in the queue for longer than the configured TTL is said to be dead. Note that a message routed to multiple queues can die at different times, or not at all, in each queue in which it resides. The death of a message in one queue has no impact on the life of the same message in other queues.
The server guarantees that dead messages will not be included in any basic.get-ok or basic.deliver methods. Further, the server will try to reap messages at or shortly after their TTL-based expiry.
The value of the x-message-ttl argument must be a non-negative 32 bit integer (0 <= n <= 2^32-1), describing the TTL period in milliseconds. Thus a value of 1000 means that a message added to the queue will live in the queue for 1 second or until it is delivered to a consumer. The argument can be of AMQP type short-short-int, short-int, long-int, or long-long-int.
This example in Java creates a queue in which messages may reside for at most 60 seconds:
Map<String, Object> args = new HashMap<String, Object>();
args.put("x-message-ttl", 60000);
channel.queueDeclare("myqueue", false, false, false, args);
The original expiry time of a message is preserved if it is requeued (for example due to the use of an AMQP method that features a requeue parameter, or due to a channel closure).
Setting x-message-ttl to 0 causes messages to be expired upon reaching a queue unless they can be delivered to a consumer immediately. Thus this provides an alternative to basic.publish's immediate flag, which the RabbitMQ server does not support. Unlike that flag, no basic.returns are issued, and if a dead letter exchange is set then messages will be dead-lettered.
Per-Message TTL
A TTL can be specified on a per-message basis, by setting the expiration field in the basic AMQP class when sending a basic.publish.
The value of the expiration field describes the TTL period in milliseconds. The same constraints as for x-message-ttl apply. Since the expiration field must be a string, the broker will (only) accept the string representation of the number.
When both a per-queue and a per-message TTL are specified, the lower value between the two will be chosen.
This example in Java publishes a message which can reside in the queue for at most 60 seconds:
byte[] messageBodyBytes = "Hello, world!".getBytes();
AMQP.BasicProperties properties = new AMQP.BasicProperties();
properties.setExpiration("60000");
channel.basicPublish("my-exchange", "routing-key", properties, messageBodyBytes);
Related
TLDR; In the context of a topic exchange and queues created on the fly by the consumers, how to have a message redelivered / the producer notified when no consumer consumes the message?
I have the following components:
a main service, producing files. Each file has a certain category (e.g. pictures.profile, pictures.gallery)
a set of workers, consuming files and producing a textual output from them (e.g. the size of the file)
I currently have a single RabbitMQ topic exchange.
The producer sends messages to the exchange with routing_key = file_category.
Each consumer creates a queue and binds the exchange to this queue for a set of routing keys (e.g. pictures.* and videos.trending).
When a consumer has processed a file, it pushes the result in a processing_results queue.
Now - this works properly, but it still has a major issue. Currently, if the publisher sends a message with a routing key that no consumer is bound to, the message will be lost. This is because even if the queue created by the consumers is durable, it is destroyed as soon as the consumer disconnects since it is unique to this consumer.
Consumer code (python):
channel.exchange_declare(exchange=exchange_name, type='topic', durable=True)
result = channel.queue_declare(exclusive = True, durable=True)
queue_name = result.method.queue
topics = [ "pictures.*", "videos.trending" ]
for topic in topics:
channel.queue_bind(exchange=exchange_name, queue=queue_name, routing_key=topic)
channel.basic_consume(my_handler, queue=queue_name)
channel.start_consuming()
Loosing a message in this condition is not acceptable in my use case.
Attempted solution
However, "loosing" a message becomes acceptable if the producer is notified that no consumer received the message (in this case it can just resend it later). I figured out the mandatory field could help, since the specification of AMQP states:
This flag tells the server how to react if the message cannot be routed to a queue. If this flag is set, the server will return an unroutable message with a Return method.
This is indeed working - in the producer, I am able to register a ReturnListener :
rabbitMq.confirmSelect();
rabbitMq.addReturnListener( (int replyCode, String replyText, String exchange, String routingKey, AMQP.BasicProperties properties, byte[] body) -> {
log.info("A message was returned by the broker");
});
rabbitMq.basicPublish(exchangeName, "pictures.profile", true /* mandatory */, MessageProperties.PERSISTENT_TEXT_PLAIN, messageBytes);
This will as expected print A message was returned by the broker if a message is sent with a routing key no consumer is bound to.
Now, I also want to know when the message was correctly received by a consumer. So I tried registering a ConfirmListener as well:
rabbitMq.addConfirmListener(new ConfirmListener() {
void handleAck(long deliveryTag, boolean multiple) throws IOException {
log.info("ACK message {}, multiple = ", deliveryTag, multiple);
}
void handleNack(long deliveryTag, boolean multiple) throws IOException {
log.info("NACK message {}, multiple = ", deliveryTag, multiple);
}
});
The issue here is that the ACK is sent by the broker, not by the consumer itself. So when the producer sends a message with a routing key K:
If a consumer is bound to this routing key, the broker just sends an ACK
Otherwise, the broker sends a basic.return followed by a ACK
Cf the docs:
For unroutable messages, the broker will issue a confirm once the exchange verifies a message won't route to any queue (returns an empty list of queues). If the message is also published as mandatory, the basic.return is sent to the client before basic.ack. The same is true for negative acknowledgements (basic.nack).
So while my problem is theoretically solvable using this, it would make the logic of knowing if a message was correctly consumed very complicated (especially in the context of multi threading, persistence in a database, etc.):
send a message
on receive ACK:
if no basic.return was received for this message
the message was correctly consumed
else
the message wasn't correctly consumed
on receive basic.return
the message wasn't correctly consumed
Possible other solutions
Have a queue for each file category, i.e. the queues pictures_profile, pictures_gallery, etc. Not good since it removes a lot of flexibility for the consumers
Have a "response timeout" logic in the producer. The producer sends a message. It expects an "answer" for this message in the processing_results queue. A solution would be to resend the message if it hasn't been answered to after X seconds. I don't like it though, it would create some additional tricky logic in the producer.
Produce the messages with a TTL of 0, and have the producer listen on a dead-letter exchange. This is the official suggested solution to replace the 'immediate' flag removed in RabbitMQ 3.0 (see paragraph Removal of "immediate" flag). According to the docs of the dead letter exchanges, a dead letter exchange can only be configured per-queue. So it wouldn't work here
[edit] A last solution I see is to have every consumer create a durable queue that isn't destroyed when he disconnects, and have it listen on it. Example: consumer1 creates queue-consumer-1 that is bound to the message of myExchange having a routing key abcd. The issue I foresee is that it implies to find an unique identifier for every consumer application instance (e.g. hostname of the machine it runs on).
I would love to have some inputs on that - thanks!
Related to:
RabbitMQ: persistent message with Topic exchange (not applicable here since queues are created "on the fly")
Make sure the broker holds messages until at least one consumer gets it
RabbitMQ Topic Exchange with persisted queue
[edit] Solution
I ended up implementing something that uses a basic.return, as mentioned earlier. It is actually not so tricky to implement, you just have to make sure that your method producing the messages and the method handling the basic returns are synchronized (or have a shared lock if not in the same class), otherwise you can end up with interleaved execution flows that will mess up your business logic.
I believe that an alternate exchange would be the best fit for your use case for the part regarding the identification of not routed messages.
Whenever an exchange with a configured AE cannot route a message to any queue, it publishes the message to the specified AE instead.
Basically upon creation of the "main" exchange, you configure an alternate exchange for it.
For the referenced alternate exchange, I tend to go with a fanout, then create a queue (notroutedq) binded to it.
This means any message that is not published to at least one of the queues bound to your "main" exchange will end up in the notroutedq
Now regarding your statement:
because even if the queue created by the consumers is durable, it is destroyed as soon as the consumer disconnects since it is unique to this consumer.
Seems that you have configured your queues with auto-delete set to true.
If so, in case of disconnect, as you stated, the queue is destroyed and the messages still present on the queue are lost, case not covered by the alternate exchange configuration.
It's not clear from your use case description whether you'd expect in some cases for a message to end up in more than one queue, seemed more a case of one queue per type of processing expected (while keeping the grouping flexible). If indeed the queue split is related to type of processing, I do not see the benefit of setting the queue with auto-delete, expect maybe not having to do any cleanup maintenance when you want to change the bindings.
Assuming you can go with durable queues, then a dead letter exchange (would again go with fanout) with a binding to a dlq would cover the missing cases.
not routed covered by alternate exchange
correct processing already handled by your processing_result queue
problematic processing or too long to be processed covered by the dead letter exchange, in which case the additional headers added upon dead lettering the message can even help to identify the type of actions to take
We're using Oracle JMS APIs to read messages from Advanced Queue. We use the following piece of code to read the messages from the queue:
MessageConsumer consumer = sess.createConsumer(q);
for (Message m; (m = consumer.receive()) != null;)
{
new Timer().schedule(new QueueExample(m), 0);
}
The problem is, after the message is received from the queue, it is not completely removed from the queue table, only the STATE field is changed from 0 to 2. Is this the default behavior of the Oracle JMS Client? We would like to completely remove the record from the queue table, after the message has been read from the queue with consumer.receive() method. What is the appropriate api method to do that ?
I think you are experiencing this due to the retention_time parameter on your queue being configured to some high value.
Retention is used for:
Users can specify that messages be retained after consumption. The
systems administrator can specify the duration for which messages will
be retained. Oracle AQ stores information about the history of each
message, preserving the queue and message properties of delay,
expiration, and retention for messages destined for local or remote
recipients. The information contains the ENQUEUE/DEQUEUE time and the
identification of the transaction that executed each request. This
allows users to keep a history of relevant messages. The history can
be used for tracking, data warehouse and data mining operations.
You can verify this by checking the queue-creation script and alter the setting via admin interface or using ALTER_QUEUE.
In the RabbitMQ/AMQP Java client, you can create an AMQP.BasicProperties.Builder, and use it to build() an instance of AMQP.BasicProperties. This built properties instance can then be used for all sorts of important things. There are lots of "builder"-style methods available on this builder class:
BasicProperties.Builder propsBuilder = new BasicProperties.Builder();
propsBuilder
.appId(???)
.clusterId(???)
.contentEncoding(???)
.contentType(???)
.correlationId(???)
.deliveryMode(2)
.expiration(???)
.headers(???)
.messageId(???)
.priority(???)
.replyTo(???)
.timestamp(???)
.type(???)
.userId(???);
I'm looking for what fields these builer methods help "build-up", and most importantly, what valid values exist for each field. For instance, what is a clusterId, and what are its valid values? What is type, and what are its valid values? Etc.
I have spent all morning scouring:
The Java client documentation; and
The Javadocs; and
The RabbitMQ full reference guide; and
The AMQP specification
In all these docs, I cannot find clear definitions (besides some vague explanation of what priority, contentEncoding and deliveryMode are) of what each of these fields are, and what their valid values are. Does anybody know? More importantly, does anybody know where these are even documented? Thanks in advance!
Usually I use very simple approach to memorize something. I will provide all details below, but here is a simple picture of BasicProperties field and values. I've also tried to properly highlight queue/server and application context.
If you want me to enhance it a bit - just drop a small comment. What I really want is to provide some visual key and simplify understanding.
High-level description (source 1, source 2):
Please note Clust ID has been deprecated, so I will exclude it.
Application ID - Identifier of the application that produced the message.
Context: application use
Value: Can be any string.
Content Encoding - Message content encoding
Context: application use
Value: MIME content encoding (e.g. gzip)
Content Type - Message content type
Context: application use
Value: MIME content type (e.g. application/json)
Correlation ID - Message correlated to this one, e.g. what request this message is a reply to. Applications are encouraged to use this attribute instead of putting this information into the message payload.
Context: application use
Value: any value
Delivery mode - Should the message be persisted to disk?
Context: queue implementation use
Value: non-persistent (1) or persistent (2)
Expiration - Expiration time after which the message will be deleted. The value of the expiration field describes the TTL period in milliseconds. Please see details below.
Context: queue implementation use
Headers - Arbitrary application-specific message headers.
Context: application use
Message ID - Message identifier as a string. If applications need to identify messages, it is recommended that they use this attribute instead of putting it into the message payload.
Context: application use
Value: any value
Priority - Message priority.
Context: queue implementation use
Values: 0 to 9
ReplyTo - Queue name other apps should send the response to. Commonly used to name a reply queue (or any other identifier that helps a consumer application to direct its response). Applications are encouraged to use this attribute instead of putting this information into the message payload.
Context: application use
Value: any value
Time-stamp - Timestamp of the moment when message was sent.
Context: application use
Value: Seconds since the Epoch.
Type - Message type, e.g. what type of event or command this message represents. Recommended to be used by applications instead of including this information into the message payload.
Context: application use
Value: Can be any string.
User ID - Optional user ID. Verified by RabbitMQ against the actual connection username.
Context: queue implementation use
Value: Should be authenticated user.
BTW, I've finally managed to review latest sever code (rabbitmq-server-3.1.5), there is an example in rabbit_stomp_test_util.erl:
content_type = <<"text/plain">>,
content_encoding = <<"UTF-8">>,
delivery_mode = 2,
priority = 1,
correlation_id = <<"123">>,
reply_to = <<"something">>,
expiration = <<"my-expiration">>,
message_id = <<"M123">>,
timestamp = 123456,
type = <<"freshly-squeezed">>,
user_id = <<"joe">>,
app_id = <<"joe's app">>,
headers = [{<<"str">>, longstr, <<"foo">>},
{<<"int">>, longstr, <<"123">>}]
Good to know somebody wants to know all the details. Because it is much better to use well-known message attributes when possible instead of placing information in the message body. BTW, basic message properties are far from being clear and useful. I would say it is better to use a custom one.
Good example (source)
Update - Expiration field
Important note: expiration belongs to queue context. So message might be dropped by the servers.
README says the following:
expiration is a shortstr; since RabbitMQ will expect this to be
an encoded string, we translate a ttl to the string representation
of its integer value.
Sources:
Additional source 1
Additional source 2
At time of writing:
The latest AMQP standard is AMQP 1.0 OASIS Standard.
The latest version of RabbitMQ is 3.1.5 (server and client), which claims to support AMQP 0.9.1 (pdf and XML schemas zipped).
RabbitMQ provides it's own description of the protocol as XML schema including extensions (i.e. non-standard), plus XML schema without extensions (which is identical to the schema linked via (2)) and pdf doc.
In this answer:
links in (3) are the primary source of detail
(2) pdf doc is used as secondary detail if (3) is inadequate
The source code (java client, erlang server) is used as tertiary detail if (2) is inadequate.
(1) is generally not used - the protocol and schema have been (fairly) significantly evolved for/by OASIS and should apply to future versions of RabbitMQ, but do not apply now. The two exceptions where (1) was used was for textual descriptions of contentType and contentEncoding - which is safe, because these are standard fields with good descriptions in AMQP 1.0.
The following text is paraphrased from these sources by me to make a little more concise or clear.
content-type (AMQP XML type="shortstr"; java type="String"): Optional. The RFC-2046 MIME type for the message’s application-data section (body). Can contain a charset parameter defining the character encoding used: e.g., ’text/plain; charset=“utf-8”’. Where the content type is unknown the content-type SHOULD NOT be set, allowing the recipient to determine the actual type. Where the section is known to be truly opaque binary data, the content-type SHOULD be set to application/octet-stream.
content-encoding (AMQP XML type="shortstr"; java type="String"): Optional. When present, describes additional content encodings applied to the application-data, and thus what decoding mechanisms need to be applied in order to obtain the media-type referenced by the content-type header field. Primarily used to allow a document to be compressed without losing the identity of its underlying content type. A modifier to the content-type, interpreted as per section 3.5 of RFC 2616. Valid content-encodings are registered at IANA. Implementations SHOULD NOT use the compress encoding, except as to remain compatible with messages originally sent with other protocols, e.g. HTTP or SMTP. Implementations SHOULD NOT specify multiple content-encoding values except as to be compatible with messages originally sent with other protocols, e.g. HTTP or SMTP.
headers (AMQP XML type="table"; java type="Map"): Optional. An application-specified list of header parameters and their values. These may be setup for application-only use. Additionally, it is possible to create queues with "Header Exchange Type" - when the queue is created, it is given a series of header property names to match, each with optional values to be matched, so that routing to this queue occurs via header-matching.
deliveryMode (RabbitMQ XML type="octet"; java type="Integer"): 1 (non-persistent) or 2 (persistent). Only works for queues that implement persistence. A persistent message is held securely on disk and guaranteed to be delivered
even if there is a serious network failure, server crash, overflow etc.
priority (AMQP XML type="octet"; java type="Integer"): The relative message priority (0 to 9). A high priority message is [MAY BE?? - GB] sent ahead of lower priority messages waiting in the same message queue. When messages must be discarded in order to maintain a specific service quality level the server will first discard low-priority messages. Only works for queues that implement priorities.
correlation-id (AMQP XML type="octet"; java type="String"): Optional. For application use, no formal (RabbitMQ) behaviour. A client-specific id that can be used to mark or identify messages between clients.
replyTo (AMQP XML type="shortstr"; java type="String"): Optional. For application use, no formal (RabbitMQ) behaviour but may hold the name of a private response queue, when used in request messages. The address of the node to send replies to.
expiration (AMQP XML type="shortstr"; java type="String"): Optional. RabbitMQ AMQP 0.9.1 schema from (3) states "For implementation use, no formal behaviour". The AMQP 0.9.1 schema pdf from (2) states an absolute time when this message is considered to be expired. However, both these descriptions must be ignored because this TTL link and the client/server code indicate the following is true. From the client, expiration is only be populated via custom application initialisation of BasicProperties. At the server, this is used to determine TTL from the point the message is received at the server, prior to queuing. The server selects TTL as the minimum of (1) message TTL (client BasicProperties expiration as a relative time in milliseconds) and (2) queue TTL (configured x-message-ttl in milliseconds). Format: string quoted integer representing number of milliseconds; time of expiry from message being received at server.
message-id (AMQP XML type="shortstr"; java type="String"): Optional. For application use, no formal (RabbitMQ) behaviour. If set, the message producer should set it to a globally unique value. In future (AMQP 1.0), a broker MAY discard a message as a duplicate if the value of the message-id matches that of a previously received message sent to the same node.
timestamp (AMQP XML type="timestamp"; java type="java.util.Date"): Optional. For application use, no formal (RabbitMQ) behaviour. An absolute time when this message was created.
type (AMQP XML type="shortstr"; java type="String"): Optional. For application use, no formal (RabbitMQ) behaviour. [Describes the message as being of / belonging to an application-specific "type" or "form" or "business transaction" - GB]
userId (AMQP XML type="shortstr"; java type="String"): Optional. XML Schema states "For application use, no formal (RabbitMQ) behaviour" - but I believe this has changed in the latest release (read on). If set, the client sets this value as identity of the user responsible for producing the message. From RabbitMQ: If this property is set by a publisher, its value must be equal to the name of the user used to open the connection (i.e. validation occurs to ensure it is the connected/authenticated user). If the user-id property is not set, the publisher's identity remains private.
appId (RabbitMQ XML type="shortstr"; java type="String"): Optional. For application use, no formal (RabbitMQ) behaviour. The creating application id. Can be populated by producers and read by consumers. (Looking at R-MQ server code, this is not used at all by the server, although the "webmachine-wrapper" plugin provides a script and matching templates to create a webmachine - where an admin can provide an appId to the script.)
cluster Id (RabbitMQ XML type="N/A"; java type="String"): Deprecated in AMQP 0.9.1 - i.e. not used. In previous versions, was the intra-cluster routing identifier, for use by cluster applications, which should not be used by client applications (i.e. not populated). However, this has been deprecated and removed from the current schema and is not used by R-MQ server code.
As you can see above, the vast majority of these properties do not have enumerated / constrained / recommended values because they are "application use only" and are not used by RabbitMQ. So you have an easy job. You're free to write/read values that are useful to your application - as long as they match datatype and compile :). ContentType and contentEncoding are as per standard HTTP use. DeliveryMode and priority are constrained numbers.
Note: Useful, but simple constants for AMQP.BasicProperties are available in class MessageProperties.
Cheers :)
UPDATE TO POST:
With many thanks to Renat (see comments), have looked at erlang server code in rabbit_amqqueue_process.erl and documentation at RabbitMQ TTL Extensions to AMQP. Message expiration (time-to-live) can be specified
per queue via:
Map<String, Object> args = new HashMap<String, Object>();
args.put("x-message-ttl", 60000);
channel.queueDeclare("myqueue", false, false, false, args);
or per message via:
byte[] messageBodyBytes = "Hello, world!".getBytes();
AMQP.BasicProperties properties = new AMQP.BasicProperties();
properties.setExpiration("60000");
channel.basicPublish("my-exchange", "routing-key", properties, messageBodyBytes);
Here, the ttl/expiration is in millisecs, so 60 sec in each case.
Have updated above definition of expiration to reflect this.
The AMQP spec defines a generic, extensible model for properties.
AMQP properties are somewhat similar in concept to HTTP headers, in that they represent metadata about the messages in question. Just as in HTTP, they are framed separately to the message payload. But they are basically a key/value map.
Some brokers like RabbitMQ will interpret certain message properties like expiration to add extra vendor-specific value (in that case, enforcing a TTL).
But in the end, AMQP properties are just a big bunch of key/value pairs that get safely sent along with each message, should you choose to do so. Your AMQP broker's documentation will tell you which ones they interpret specially and how to send your own ones.
All that being said, if you're asking this question in the first place then you probably don't need to worry about them at all. You will be successfully able to send messages without having to worry about setting any message properties at all.
I have a Java client which monitors RabbitMQ queue. I am able to get the count of messages currently in queue with this code
#Resource
RabbitAdmin rabbitAdmin;
..........
DeclareOk declareOk = rabbitAdmin.getRabbitTemplate().execute(new ChannelCallback<DeclareOk>() {
public DeclareOk doInRabbit(Channel channel) throws Exception {
return channel.queueDeclarePassive("test.pending");
}
});
return declareOk.getMessageCount();
I want to get some more additional details like -
Message body of currently enqueued items.
Total number of messages that was enqueued in the queue since the queue was created.
Is there any way to retrieve these data in Java client?
With AMQP protocol (including RabbitMQ implementation) you can't get such info with 100% guarantee.
The closest number to messages count is messages count returned with queue.declare-ok (AMQP.Queue.DeclareOk in java AMQP client library).
Whilst messages count you receive with queue.declare-ok may match exact messages number enqueues, you can't rely on it as it doesn't count messages which waiting acknowledges or published to queue during transaction but not committed yet.
It really depends what kind of precission do you need.
As to enqueued messages body, you may want to manually extract all messages in queue, view their body and put them back to queue. This is the only way to do what you want.
You can get some information about messages count with Management Plugin, RabbitMQ Management HTTP API and rabbitmqctl util (see list_queues, list_channels).
You can't get total published messages count since queue was created and I think nobody implement such stats while it useless (FYI, with messages flow in average 10k per second you will not even reach uint64 in a few thousand years).
AMQP.Queue.DeclareOk dok = channel.queueDeclare(QUEUE_NAME, true, false, false, queueArgs);
dok.getMessageCount();
To access queue details via http api,
http://public-domain-name:15672/api/queues/%2f/queue_name
To access queue details via command from localhost cli promt,
curl -i -u guest_uname:guest_password http://localhost:15672/api/queues/%2f/queue_name
Where,
%2f is default vhost "/"
I have a publisher that is pushing messages to a topic. I have multiple subscribers each doing a different task once they consume the message from the topic.
Now I want my system to scale to multiple instances of the same process running on different hosts/same host. e.g. I want to run multiple copies of my application A on different hosts so that if one instance of A is slow, then the other instances can pull in subsequent messages and make forward progress..
I found out that this is possible using virtual destinations. I followed the steps here -
http://activemq.apache.org/virtual-destinations.html
But how do i setup my multiple subscribers to the same topic with the same client id? when i try to do that, i get errors. when i try some other way, it doesn't work. can someone help?
Normally, I start a subscriber by doing the below steps -
ActiveMQConnectionFactory connectionFactory = new ActiveMQConnectionFactory(ActiveMQConnection.DEFAULT_USER, ActiveMQConnection.DEFAULT_PASSWORD, ActiveMQConnection.DEFAULT_BROKER_URL;);
activeMQConnection = connectionFactory.createConnection();
activeMQConnection.setClientID("subscriber1");
activeMQConnection.setExceptionListener(exceptionListener);
activeMQSession = activeMQConnection.createSession(false, Session.CLIENT_ACKNOWLEDGE);
activeMQTopic = activeMQSession.createTopic("myTopic");
activeConsumer = activeMQSession.createDurableSubscriber(activeMQTopic, "myTopic");
activeConsumer.setMessageListener(messageListener);
activeMQConnection.start();
when i try to create a 2nd subscriber and pass the topic name as "VirtualTopic.myTopic", nothing happens.
thanks
The virtual topics feature is very simple and quite powerful once you understand it.
When using virtual topics - there is no need for durable consumers. That is because for each client you will get an instance of regular queue created. If you have 5 clients (application A, B, C, D, E) you will get 5 queues created and populated with the copy of the messages every time message is sent to the virtual topic.
Actually it is a limitation of durable consumer - that only ONE connection is allowed per clientId. Being a regular queue, you can create as many consumers as you like and queue will guarantee that 1 message will be received only by 1 consumer. So if you have application A that takes 1 minute to process a message, you can create 5 instances of it listening to the same queue. When you will post 5 messages within 1 second, each of your application will receive its own message to process.
There are not well documented requirements which are not intuitive. To make virtual topic work you need
Use VirtualTopic. in your topic name, for example VirtualTopic.Orders (this prefix can be configured)
Use Consumer. in the name of the queue you. Like Consumer.ApplicationA.VirtualTopic.Orders where ApplicationA is actually your client id
Use regular subscribers not durable ones for the queue above.
Example:
string activeMqConsumerTopic = "Consumer.AmqTestConsumer.VirtualTopic.Orders";
IQueue queue = SessionUtil.GetQueue(session, activeMqConsumerTopic);
IMessageConsumer consumer = session.CreateConsumer(queue);
Queue is created for automatically whenever the first instance of consumer is subscribed to it. Since that moment all messages that are sent to topic are duplicated/copied into all associated queues.
Hope this helps.
Virtual Topics is the answer for you. However you have to define a naming standard for all virtual topic queues. Here is the answer for this:
Virtual Topics helps with following prospective:
1. Load Balancing of messages
2. Fast Failover of Subscriber
3. Re-using same connection Factory for different Producers and Consumers. (Durable Subscribers needs a Unique JMS Client Id and same cannot be reused for any other Producer or consumer)
here is the way to do it, below example creates prefix VTCON.*. So every queue with this prefix and Topic Name at the end will consumer the message.
<virtualDestinations>
<virtualTopic name="TEST.TP01" prefix="VTCON.*." selectorAware="false"/>
</virtualDestinations>
http://workingwithqueues.blogspot.com/2012/05/activemq-virtual-topics-or-virtual.html