I'm working with JMS and queues (Azure queues) for the first time. I'm required to make a queue where Rubi server would write some data and Java would read it from queue and will do further executions.
This process is working fine locally on my machine. I've created a REST endpoint which is writing data in the queue and once data is written in the queue, the listener would take over and read the data and execute.
When we deploy it to Azure the error I can see in logs which is not letting the Queues start is
Setup of JMS message listener invoker failed for destination 'queue' - trying to recover. Cause: Identifier contains invalid JMS identifier character '-': 'x-request-id'
Zipkin is also present on the Azure server as a distributed tracing system and I guess this x-request-id is related to Zipkin which is creating the problem. I've searched Google for the issue but couldn't understand why its happening.
Following is detailed error message:
[36mc.m.s.l.NextGenRequestLoggingFilter [0;39m [2m:[0;39m
Before request [uri=/services/deal-service/api/v2/deals/ack;headers=
[x-request-id:"2d8d86d7-4fbf-9db6-8e95-28813f21a85c",
x-envoy-internal:"true", x-b3-parentspanid:"a209cdc649b0b890", content-
length:"575", x-forwarded-proto:"http", postman-token:"ad074595-
76a5-474b-9711-7e071b12b3b0", x-b3-sampled:"1", x-forwarded-
for:"10.244.2.1", accept:"*/*",
authorization: "some-token-YJc4tg--34jPRziJNSACqNQ", x-b3-
traceid:"6b40ff22781be67ba209cdc649b0b890", x-b3-
spanid:"702684ddb62cfe6b",
host:"portal-gateway.52.228.65.225.nip.io",
cache-control:"no-cache", accept-encoding:"gzip, deflate, br",
user-agent:"PostmanRuntime/7.22.0",
Content-Type:"application/xml;charset=UTF-8"]]
2020-02-18T15:19:34.197666458Z [2m2020-02-18 15:19:34.197[0;39m .
[32mDEBUG
[,6b40ff22781be67ba209cdc649b0b890,702684ddb62cfe6b,true][0;39m .
[35m9[0;39m [2m---[0;39m [2m[ XNIO-1 task-15][0;39m
Section 3.5.1 of the JMS 2 specification states this about message properties:
Property names must obey the rules for a message selector identifier. See
Section 3.8 “Message selection” for more information.
In regards to identifiers, section 3.8.1.1 states, in part:
An identifier is an unlimited-length character sequence that must begin with a Java identifier start character; all following characters must be Java identifier part characters. An identifier start character is any character for which the method Character.isJavaIdentifierStart returns true. This includes '_' and '$'. An identifier part character is any character for which the method Character.isJavaIdentifierPart returns true.
If you pass the character - into either Character.isJavaIdentifierStart or Character.isJavaIdentifierPart the return value is false. In other words, the - character in the name of a message property violates the JMS specification and therefore will cause an error.
From the error message its obvious that you are using qpid JMS client for communication through queues.
qpid client won’t allow any keys which violates java variable naming convention e.g. you won’t be able to send x-request-id in a queue’s header
which qpid jms client is consuming as it’ll throw error.
You need to take care of istio/zipkin to not to add certain headers (id you don’t need them actually) with the queue when its trying to communicate on azure bus.
So you have to disable the istio/zipkin libraries to intercept the request for queues so that request to/from queue can be made without headers. This will fix the issue.
Details of error (Java stack trace) would be really useful here.
By error message I assume, you are using qpid JMS client, that is performing check of message properties' names. These names can contain only characters, that are valid Java identifier characters.
In string 'queue-name' there is a '-' character, that is not Java identifier. To fix, you need to change 'queue-name' into something with valid characters, for example 'queue_name' (with underscore), or 'queueName' (camel case).
I am using Camel for wanting to push a resource to a topic.
The name of the topic is depending on the content of a resource.
For this one I've implemented the choice definition.
I've put the name of the generated topic in the header of the message:
final String resource = exchange.getIn().getBody(String.class);
String topic = resource + ".get.request";
exchange.getIn().setHeader("topic", topic);
When it comes to the endpoint part of the Camel route, then I wanted to read and re-use the topic out of the exchanged message:
// the logging shows everything up correctly, with both variants..
.log("${header.topic} : " + simple("${header.topic}").getText())
// this line is not woking, because the simple language fails here
.to("mqtt:camelMqtt?publishTopicName=${header.topic}?host=" + mqttHost);
If I am putting a hard coded topic instead of the ${header.topic}, then the message is published on the mqtt-topic.
As stated in the code, the to(...)sequence is not working with the Simple Language.
How to solve that kind of issue?
Ok, found it out by myself...
The Simple Language is only evaluated IF the endpoint method toDis used.
Makes totally sense, because this is a dynamic endpoint or at least the parameters of this endpoint are variable.
full line from above:
.toD("mqtt:camelMqtt?publishTopicName=${header.topic}?host=" + mqttHost);
I'm working on a Spring Boot REST API that handles document and can launch a check on a document.
I have a document resource: /doc:
Create a doc with POST /doc
Rest of the CRUD actions with /doc/{id}
Now I can launch a check on a doc, check can be seen either as an action or as a sub-resource.
It's pretty straightforward to launch (create) a check on a document: POST /doc/{id}/check
The check can however take some time so I want to give the user the choice to launch a synchronous or asynchronous check.
How would I handle this path wise?
Should the user choose sync or async check through a query parameter on POST /doc/{id}/check?
Should I create 2 separate paths?
Also in the case of an async check, I would create a temporary Task resource that can be pooled to know the status of the check.
But then if both check and task are returned from the same path it gets confusing, no?
I read an article that says the resource returned in async should be a check resource filled as much as possible but with a link to the task that can be pooled.
That seems like a good way; I would return a partial check if async with a link to the /task/{id} associated with the check.
However I'm still confused as to what path my API should offer to let the user pick between sync and async checks.
How would you handle it path and resource wise?
Basically it's up to you. Usually if it's a big chunk of data you want to query like /resource/{id} most APIs I have used use GET for synchronous requests and POST for async request returning task or job ID.
For POST in your case if the creation/checking takes time I would consider always doing it asynchronous and returning HTTP 202 Accepted and doc/{id}/check/{id} url where the user can see the result if it is ready or some status that it is still working.
If you want to give them a choice to wait or not it's up to you how to do it. There is a standard header that can be used to modify behavior. For example Expect: 202-accepted for async calls and no header or Expect: 201-created for synchronous calls. This makes the API a bit less clear even though it is a standard. Most people (including me) would probably stick to adding a parameter to the URL for clarification. I don't think it should be in the POST data because it should be data related to the object you are creating
There are multiple questions here. I would try to answer one by one
Checking the health of a resource can be done with query param
/doc/{id} - GET Get the resource details
/doc/{id}?healthCheck=true&async=true GET - Get the resource details and trigger an async health check
For the async health check the response as you mentioned will be 202 and the response contains the link to the health status URL
HTTP/1.1 202 Accepted
Location: /doc/12345/status
If the client sends a GET request to this endpoint, the response should contain the current status of the request. Optionally, it could also include an estimated time to completion or a link to cancel the operation.
Reference
https://learn.microsoft.com/en-us/azure/architecture/best-practices/api-design
I want to let my application to register for topic messaging, but each time I try to call Topic registration it is returning the error below
java.lang.IllegalArgumentException: Invalid topic name:
Using the method bellow
GcmPubSub.getInstance(context).subscribe(regToken, "/topics/testTopic", null);
I am using the tutorial on this link https://developers.google.com/cloud-messaging/topic-messaging. I will greatly appreciate your help on this error.
Please try this with the sample app. "/topics/testTopic" should be fine. As mentioned in the documentation, any topic name matching this regex should be fine: /topics/[a-zA-Z0-9-_.~%]+
For anyone running into this error: double check that you conform to the specification on the topic name: /topics/[a-zA-Z0-9-_.~%]+.
The original question actually does conform to the spec and I have no answer for why the error occurs. However another way to trigger the error is not conforming to the spec. In my case I mistakenly assumed path separator characters are allowed and used a topic like /topics/users/123 to mimic my REST API. The last slash is not allowed.
In the RabbitMQ/AMQP Java client, you can create an AMQP.BasicProperties.Builder, and use it to build() an instance of AMQP.BasicProperties. This built properties instance can then be used for all sorts of important things. There are lots of "builder"-style methods available on this builder class:
BasicProperties.Builder propsBuilder = new BasicProperties.Builder();
propsBuilder
.appId(???)
.clusterId(???)
.contentEncoding(???)
.contentType(???)
.correlationId(???)
.deliveryMode(2)
.expiration(???)
.headers(???)
.messageId(???)
.priority(???)
.replyTo(???)
.timestamp(???)
.type(???)
.userId(???);
I'm looking for what fields these builer methods help "build-up", and most importantly, what valid values exist for each field. For instance, what is a clusterId, and what are its valid values? What is type, and what are its valid values? Etc.
I have spent all morning scouring:
The Java client documentation; and
The Javadocs; and
The RabbitMQ full reference guide; and
The AMQP specification
In all these docs, I cannot find clear definitions (besides some vague explanation of what priority, contentEncoding and deliveryMode are) of what each of these fields are, and what their valid values are. Does anybody know? More importantly, does anybody know where these are even documented? Thanks in advance!
Usually I use very simple approach to memorize something. I will provide all details below, but here is a simple picture of BasicProperties field and values. I've also tried to properly highlight queue/server and application context.
If you want me to enhance it a bit - just drop a small comment. What I really want is to provide some visual key and simplify understanding.
High-level description (source 1, source 2):
Please note Clust ID has been deprecated, so I will exclude it.
Application ID - Identifier of the application that produced the message.
Context: application use
Value: Can be any string.
Content Encoding - Message content encoding
Context: application use
Value: MIME content encoding (e.g. gzip)
Content Type - Message content type
Context: application use
Value: MIME content type (e.g. application/json)
Correlation ID - Message correlated to this one, e.g. what request this message is a reply to. Applications are encouraged to use this attribute instead of putting this information into the message payload.
Context: application use
Value: any value
Delivery mode - Should the message be persisted to disk?
Context: queue implementation use
Value: non-persistent (1) or persistent (2)
Expiration - Expiration time after which the message will be deleted. The value of the expiration field describes the TTL period in milliseconds. Please see details below.
Context: queue implementation use
Headers - Arbitrary application-specific message headers.
Context: application use
Message ID - Message identifier as a string. If applications need to identify messages, it is recommended that they use this attribute instead of putting it into the message payload.
Context: application use
Value: any value
Priority - Message priority.
Context: queue implementation use
Values: 0 to 9
ReplyTo - Queue name other apps should send the response to. Commonly used to name a reply queue (or any other identifier that helps a consumer application to direct its response). Applications are encouraged to use this attribute instead of putting this information into the message payload.
Context: application use
Value: any value
Time-stamp - Timestamp of the moment when message was sent.
Context: application use
Value: Seconds since the Epoch.
Type - Message type, e.g. what type of event or command this message represents. Recommended to be used by applications instead of including this information into the message payload.
Context: application use
Value: Can be any string.
User ID - Optional user ID. Verified by RabbitMQ against the actual connection username.
Context: queue implementation use
Value: Should be authenticated user.
BTW, I've finally managed to review latest sever code (rabbitmq-server-3.1.5), there is an example in rabbit_stomp_test_util.erl:
content_type = <<"text/plain">>,
content_encoding = <<"UTF-8">>,
delivery_mode = 2,
priority = 1,
correlation_id = <<"123">>,
reply_to = <<"something">>,
expiration = <<"my-expiration">>,
message_id = <<"M123">>,
timestamp = 123456,
type = <<"freshly-squeezed">>,
user_id = <<"joe">>,
app_id = <<"joe's app">>,
headers = [{<<"str">>, longstr, <<"foo">>},
{<<"int">>, longstr, <<"123">>}]
Good to know somebody wants to know all the details. Because it is much better to use well-known message attributes when possible instead of placing information in the message body. BTW, basic message properties are far from being clear and useful. I would say it is better to use a custom one.
Good example (source)
Update - Expiration field
Important note: expiration belongs to queue context. So message might be dropped by the servers.
README says the following:
expiration is a shortstr; since RabbitMQ will expect this to be
an encoded string, we translate a ttl to the string representation
of its integer value.
Sources:
Additional source 1
Additional source 2
At time of writing:
The latest AMQP standard is AMQP 1.0 OASIS Standard.
The latest version of RabbitMQ is 3.1.5 (server and client), which claims to support AMQP 0.9.1 (pdf and XML schemas zipped).
RabbitMQ provides it's own description of the protocol as XML schema including extensions (i.e. non-standard), plus XML schema without extensions (which is identical to the schema linked via (2)) and pdf doc.
In this answer:
links in (3) are the primary source of detail
(2) pdf doc is used as secondary detail if (3) is inadequate
The source code (java client, erlang server) is used as tertiary detail if (2) is inadequate.
(1) is generally not used - the protocol and schema have been (fairly) significantly evolved for/by OASIS and should apply to future versions of RabbitMQ, but do not apply now. The two exceptions where (1) was used was for textual descriptions of contentType and contentEncoding - which is safe, because these are standard fields with good descriptions in AMQP 1.0.
The following text is paraphrased from these sources by me to make a little more concise or clear.
content-type (AMQP XML type="shortstr"; java type="String"): Optional. The RFC-2046 MIME type for the message’s application-data section (body). Can contain a charset parameter defining the character encoding used: e.g., ’text/plain; charset=“utf-8”’. Where the content type is unknown the content-type SHOULD NOT be set, allowing the recipient to determine the actual type. Where the section is known to be truly opaque binary data, the content-type SHOULD be set to application/octet-stream.
content-encoding (AMQP XML type="shortstr"; java type="String"): Optional. When present, describes additional content encodings applied to the application-data, and thus what decoding mechanisms need to be applied in order to obtain the media-type referenced by the content-type header field. Primarily used to allow a document to be compressed without losing the identity of its underlying content type. A modifier to the content-type, interpreted as per section 3.5 of RFC 2616. Valid content-encodings are registered at IANA. Implementations SHOULD NOT use the compress encoding, except as to remain compatible with messages originally sent with other protocols, e.g. HTTP or SMTP. Implementations SHOULD NOT specify multiple content-encoding values except as to be compatible with messages originally sent with other protocols, e.g. HTTP or SMTP.
headers (AMQP XML type="table"; java type="Map"): Optional. An application-specified list of header parameters and their values. These may be setup for application-only use. Additionally, it is possible to create queues with "Header Exchange Type" - when the queue is created, it is given a series of header property names to match, each with optional values to be matched, so that routing to this queue occurs via header-matching.
deliveryMode (RabbitMQ XML type="octet"; java type="Integer"): 1 (non-persistent) or 2 (persistent). Only works for queues that implement persistence. A persistent message is held securely on disk and guaranteed to be delivered
even if there is a serious network failure, server crash, overflow etc.
priority (AMQP XML type="octet"; java type="Integer"): The relative message priority (0 to 9). A high priority message is [MAY BE?? - GB] sent ahead of lower priority messages waiting in the same message queue. When messages must be discarded in order to maintain a specific service quality level the server will first discard low-priority messages. Only works for queues that implement priorities.
correlation-id (AMQP XML type="octet"; java type="String"): Optional. For application use, no formal (RabbitMQ) behaviour. A client-specific id that can be used to mark or identify messages between clients.
replyTo (AMQP XML type="shortstr"; java type="String"): Optional. For application use, no formal (RabbitMQ) behaviour but may hold the name of a private response queue, when used in request messages. The address of the node to send replies to.
expiration (AMQP XML type="shortstr"; java type="String"): Optional. RabbitMQ AMQP 0.9.1 schema from (3) states "For implementation use, no formal behaviour". The AMQP 0.9.1 schema pdf from (2) states an absolute time when this message is considered to be expired. However, both these descriptions must be ignored because this TTL link and the client/server code indicate the following is true. From the client, expiration is only be populated via custom application initialisation of BasicProperties. At the server, this is used to determine TTL from the point the message is received at the server, prior to queuing. The server selects TTL as the minimum of (1) message TTL (client BasicProperties expiration as a relative time in milliseconds) and (2) queue TTL (configured x-message-ttl in milliseconds). Format: string quoted integer representing number of milliseconds; time of expiry from message being received at server.
message-id (AMQP XML type="shortstr"; java type="String"): Optional. For application use, no formal (RabbitMQ) behaviour. If set, the message producer should set it to a globally unique value. In future (AMQP 1.0), a broker MAY discard a message as a duplicate if the value of the message-id matches that of a previously received message sent to the same node.
timestamp (AMQP XML type="timestamp"; java type="java.util.Date"): Optional. For application use, no formal (RabbitMQ) behaviour. An absolute time when this message was created.
type (AMQP XML type="shortstr"; java type="String"): Optional. For application use, no formal (RabbitMQ) behaviour. [Describes the message as being of / belonging to an application-specific "type" or "form" or "business transaction" - GB]
userId (AMQP XML type="shortstr"; java type="String"): Optional. XML Schema states "For application use, no formal (RabbitMQ) behaviour" - but I believe this has changed in the latest release (read on). If set, the client sets this value as identity of the user responsible for producing the message. From RabbitMQ: If this property is set by a publisher, its value must be equal to the name of the user used to open the connection (i.e. validation occurs to ensure it is the connected/authenticated user). If the user-id property is not set, the publisher's identity remains private.
appId (RabbitMQ XML type="shortstr"; java type="String"): Optional. For application use, no formal (RabbitMQ) behaviour. The creating application id. Can be populated by producers and read by consumers. (Looking at R-MQ server code, this is not used at all by the server, although the "webmachine-wrapper" plugin provides a script and matching templates to create a webmachine - where an admin can provide an appId to the script.)
cluster Id (RabbitMQ XML type="N/A"; java type="String"): Deprecated in AMQP 0.9.1 - i.e. not used. In previous versions, was the intra-cluster routing identifier, for use by cluster applications, which should not be used by client applications (i.e. not populated). However, this has been deprecated and removed from the current schema and is not used by R-MQ server code.
As you can see above, the vast majority of these properties do not have enumerated / constrained / recommended values because they are "application use only" and are not used by RabbitMQ. So you have an easy job. You're free to write/read values that are useful to your application - as long as they match datatype and compile :). ContentType and contentEncoding are as per standard HTTP use. DeliveryMode and priority are constrained numbers.
Note: Useful, but simple constants for AMQP.BasicProperties are available in class MessageProperties.
Cheers :)
UPDATE TO POST:
With many thanks to Renat (see comments), have looked at erlang server code in rabbit_amqqueue_process.erl and documentation at RabbitMQ TTL Extensions to AMQP. Message expiration (time-to-live) can be specified
per queue via:
Map<String, Object> args = new HashMap<String, Object>();
args.put("x-message-ttl", 60000);
channel.queueDeclare("myqueue", false, false, false, args);
or per message via:
byte[] messageBodyBytes = "Hello, world!".getBytes();
AMQP.BasicProperties properties = new AMQP.BasicProperties();
properties.setExpiration("60000");
channel.basicPublish("my-exchange", "routing-key", properties, messageBodyBytes);
Here, the ttl/expiration is in millisecs, so 60 sec in each case.
Have updated above definition of expiration to reflect this.
The AMQP spec defines a generic, extensible model for properties.
AMQP properties are somewhat similar in concept to HTTP headers, in that they represent metadata about the messages in question. Just as in HTTP, they are framed separately to the message payload. But they are basically a key/value map.
Some brokers like RabbitMQ will interpret certain message properties like expiration to add extra vendor-specific value (in that case, enforcing a TTL).
But in the end, AMQP properties are just a big bunch of key/value pairs that get safely sent along with each message, should you choose to do so. Your AMQP broker's documentation will tell you which ones they interpret specially and how to send your own ones.
All that being said, if you're asking this question in the first place then you probably don't need to worry about them at all. You will be successfully able to send messages without having to worry about setting any message properties at all.