We Keep Coding

Zipkin, using existing libraries to handle tracing in microservices connected with Apache Kafka

I would like to implement tracing in my microservices architecture. I am using Apache Kafka as message broker and I am not using Spring Framework. Tracing is a new concept for me. At first I wanted to create my own implementation, but now I would like to use existing libraries. Brave looks like the one I will want to use. I would like to know if there are some guides, examples or docs on how to do this. Documentation on Github page is minimal, and I find it hard to start using Brave. Or maybe there is better library with proper documentation, that is easier to use. I will be looking at Apache HTrace because it looks promising. Some getting started guides will be nice.

There are a bunch of ways to answer this, but I'll answer it from the "one-way" perspective. The short answer though, is I think you have to roll your own right now!
While Kafka can be used in many ways, it can be used as a transport for unidirectional single producer single consumer messages. This action is similar to normal one-way RPC, where you have a request, but no response.
In Zipkin, an RPC span is usually request-response. For example, you see timing of the client sending to the server, and also the way back to the client. One-way is where you leave out the other side. The span starts with a "cs" (client send) and ends with a "sr" (server received).
Mapping this to Kafka, you would mark client sent when you produce the message and server received when the consumer receives it.
The trick to Kafka is that there is no nice place to stuff the trace context. That's because unlike a lot of messaging systems, there are no headers in a Kafka message. Without a trace context, you don't know which trace (or span for that matter) you are completing!
The "hack" approach is to stuff trace identifiers as the message key. A less hacky way would be to coordinate a body wrapper which you can nest the trace context into.
Here's an example of the former:
https://gist.github.com/adriancole/76d94054b77e3be338bd75424ca8ba30

I meet the same problem too.Here is my solution, a less hacky way as above said.
ServerSpan serverSpan = brave.serverSpanThreadBinder().getCurrentServerSpan();
TraceHeader traceHeader = convert(serverSpan);
//in kafka producer,user KafkaTemplete to send
String wrapMsg = "wrap traceHeader with originMsg ";
kafkaTemplate.send(topic, wrapMsg).get(10, TimeUnit.SECONDS);// use synchronization
//then in kafka consumer
ConsumerRecords<String, String> records = consumer.poll(5000);
// for loop
for (ConsumerRecord<String, String> record : records) {
String topic = record.topic();
int partition = record.partition();
long offset = record.offset();
String val = record.value();
//parse val to json
Object warpObj = JSON.parseObject(val);
TraceHeader traceHeader = warpObj.getTraceHeader();
//then you can do something like this
MyRequest myRequest = new MyRequest(traceHeader, "/esb/consumer", "POST");
brave.serverRequestInterceptor().handle(new HttpServerRequestAdapter(new MyHttpServerRequest(myRequest), new DefaultSpanNameProvider()));
//then some httprequest within brave-apache-http-interceptors
//http.post(url,content)
}
you must implements MyHttpServerRequest and MyRequest.It is easy,you just return something a span need,such as uri,header,method.
This is a rough and ugly code example,just offer an idea.

Using Apache Camel with multiple websocket endpoints

I have a Jetty endpoint receiving push-messages of different types from several data-providers - let´s call them type1, type2.
I also provide set of websocket-endpoints - one for each type - where clients can subscribe to push-updates of the different message-types (/ws_type1, /ws_type2).
The dataproviders and the clients are not the same.
Based on the type of incoming message, I am trying to get the camel route to push updates only to the clients subscribing to the specific messagetype.
My code is something like this:
from("jetty:http://0.0.0.0:8888/incoming_1?matchOnUriPrefix=true")
.log("incoming_1")
.to("websocket://ws_type1?sendToAll=true");
from("jetty:http://0.0.0.0:8888/incoming_2?matchOnUriPrefix=true")
.log("incoming_2")
.to("websocket://ws_type2?sendToAll=true");
from("jetty:http://0.0.0.0:8080/incoming?matchOnUriPrefix=true")
.log("incoming")
.to("websocket://woot?sendToAll=true");
from("websocket://ws_type1")
.log("ws_type1")
.to("websocket://ws_type1")
;
from("websocket://ws_type2")
.log("ws_type2")
.to("websocket://ws_type2")
;
The result of this is that ALL messages of ALL types are sent to ALL the connected clients - regardless if they are connected to the /ws_type1 or /ws_type2 endpoints.
Is it possible to filter the updates to different sets of clients? I.e. messages of type1 are pushed to the clients connected to /ws_type1, and messages of type2 are pushed to the clients connected to /ws_type2.
EDIT:
Using the code above, I am able to connect websockets to ws-endpoints "ws_type1", "ws_type2", and "woot". However - when I post data to the "/incoming_1" jetty-endpoint ALL three websockets receive the data, but only "incoming_1" is logged.
Am I missing something?
EDIT 2:
I looked at the camel-websocket source, and it seems like this is by design. There is no filtering on which websockets the message-contents are sent to, and the content is written directly to the socket. This also means that the camel-routes "ws_type1" and "ws_type2" in the example above will not be called when the server pushes data.
....
Collection<DefaultWebsocket> websockets = store.getAll();
Exception exception = null;
for (DefaultWebsocket websocket : websockets) {
try {
sendMessage(websocket, message);
...
Full source

Steps for publishing message to Solace

I have heard about Solace and I'm reading about it but I don't know much yet about it. I have a HashMap:
{Swaps_snaptime=2016-04-26T07:00:00.000Z, Swaps_20Y=2036-04-29 0.004588, Swaps_15Y=2031-04-29}
And I want to publish this to Solace. How do I do that? What are the steps I need to follow? Please help!

You need to execute the following tasks to send a message:
Connect to the Solace Message Router
Create a producer
Create the message object
Send the message
There's a simple example of how to do the above at the Solace website.
Note that the example makes use of TextMessages.
In order to send your map, you can do the following:
Serialize your map, and then send it as part of a BytesMessage.
Place the contents of your map into a MapMessage.
Place the contents of your map into a StreamMessage
Use any message type, and place the contents of your map into the header portion of the message. Refer to How to send the header along with payload in Solace.

Parse byteBuffer in Websphere MQ Exit

I'm trying to build a custom mq exit to archive messages that hit a queue. I have the following code.
class MyMqExits implements WMQSendExit, WMQReceiveExit{
#Override
public ByteBuffer channelReceiveExit(MQCXP arg0, MQCD arg1, ByteBuffer arg2) {
// TODO Auto-generated method stub
if ( arg2){
def _bytes = arg2.array()
def results = new String(_bytes)
println results;
}
return arg2;
}
...
The content of the message (header/body) is in the byte buffer, along with some unreadable binary information. How can I parse the message (including the body and the queue name) from arg2? We've gone through IBM's documentation, but haven't found an object or anything that makes this easy.

Assuming the following two points:
1) Your sender application has not hard coded the queue name where it puts messages. So you can change the application configuration to send messages to a different object.
2) MessageId of the archived message is not important, only message body is important.
Then one alternative I can think of is to create an Alias queue that resolves to a Topic and use two subscribers to receive messages.
1) Subscriber 1: An administratively defined durable subscriber with a queue provided to receive messages. Provide the same queue name from which your existing consumer application is receiving messages.
2) Subscriber 2: Another administratively defined durable subscriber with queue provided. You can write a simple java application to get messages from this queue and archive.
3) Both subscribers subscribe to the same topic.
Here are steps:
// Create a topic
define topic(ANY.TOPIC) TOPICSTR('/ANY_TOPIC')
// Create an alias queue that points to above created topic
define qalias(QA.APP) target(ANY.TOPIC) targtype(TOPIC)
// Create a queue for your application that does business logic. If one is available already then no need to create.
define ql(Q.BUSLOGIC)
// Create a durable subscription with destination queue as created in previous step.
define sub(SB.BUSLOGIC) topicstr('/ANY_TOPIC') dest(Q.BUSLOGIC)
// Create a queue for application that archives messages.
define ql(Q.ARCHIVE)
// Create another subscription with destination queue as created in previous step.
define sub(SB.ARCHIVE) topicstr('/ANY_TOPIC') dest(Q.ARCHIVE)
Write a simple MQ Java/JMS application to get messages from Q.ARCHIVE and archive messages.

A receive exit is not going to give you the whole message. Send and receive exits operate on the transmission buffers sent/received by channels. These will contain various protocol flows which are not documented because the protocol is not public, and part of those protocol flows will be chunks of the messages broken down to fit into 32Kb chunks.
You don't give enough information in your question for me to know what type of channel you are using, but I'm guessing it's on the client side since you are writing it in Java and that is the only environment where that is applicable.
Writing the exit at the client side, you'll need to be careful you deal with the cases where the message is not successfully put to the target queue, and you'll need to manage syncpoints etc.
If you were using QMgr-QMgr channels, you should use a message exit to capture the MQXR_MSG invocations where the whole message is given to you. If you put any further messages in a channel message exit, the messages you put are included in the channel's Syncpoint and so committed if the original messages were committed.
Since you are using client-QMgr channels, you could look at an API Exit on the QMgr end (currently client side API Exits are only supported for C clients) and catch all the MQPUT calls. This exit would also give you the MQPUT return codes so you could code your exit to look out for, and deal with failed puts.
Of course, writing an exit is a complicated task, so it may be worth finding out if there are any pre-written tools that could do this for you instead of starting from scratch.

I fully agree with Morag & Shashi, wrong approach. There is an open source project called Message Multiplexer (MMX) that will get a message from a queue and output it to one or more queues. Context information is maintained across the message put(s). For more info on MMX go to: http://www.capitalware.com/mmx_overview.html
If you cannot change the source or target queues to insert MMX into the mix then an API Exit may do the trick. Here is a blog posting about message replication via an API Exit: http://www.capitalware.com/rl_blog/?p=3304

This is quite an old question but it's worth replying with an update that's relevant to MQ 9.2.3 or later. There is a new feature called Streaming Queues (see https://www.ibm.com/docs/en/ibm-mq/9.2?topic=scenarios-streaming-queues) and one of the use-cases it is designed to support is putting a copy of every message sent to a given queue, to an alternative queue. Another application can then consume the duplicate messages and archive them separately to the application that is processing the original messages.

Work around for MessageNotReadableException in Java

I am building a small api around the JMS API for a project of mine. Essentially, we are building code that will handle the connection logic, and will simplify publishing messages by providing a method like Client.send(String message).
One of the ideas being discussed right now is that we provide a means for the users to attach interceptors to this client. We will apply the interceptors after preparing the JMS message and before publishing it.
For example, if we want to timestamp a message and wrote an interceptor for that, then this is how we would apply that
...some code ...
Message message = session.createMessage()
..do all the current processing on the message and set the body
for(interceptor:listOfInterceptors){
interceptor.apply(message)
}
One of the intrerceptors we though of was to compress the message body. But when we try to read the body of the message in the interceptor, we are getting a MessageNotReadableException. In the past, I normally compressed the content before setting it as the body of the message - so never had to worry about this exception.
Is there any way of getting around this exception?

It looks like your JMS client attempts to read a write-only message. Your interceptor cannot work this way, please elaborate how you were compressing message earlier.