We Keep Coding

How to ensure messages reach kafka broker?

I have have a message producer on my local machine and a broker on remote host (aws).
After sending a message from the producer,
I wait and call the console consumer on the remote host and
see excessive logs.
Without the value from producer.
The producer flushes the data after calling the send method.
Everything is configured correctly.
How can I check to see that the broker received the message from the producer and to see if the producer received the answer?

The Send method asynchronously sends the message to the topic and
returns a Future of RecordMetadata.
java.util.concurrent.Future<RecordMetadata> send(ProducerRecord<K,V> record)
Asynchronously sends a record to a topic
After the flush call,
check to see that the Future has completed by calling the isDone method.
(for example, Future.isDone() == true)
Invoking this method makes all buffered records immediately available to send (even if linger.ms is greater than 0) and blocks on the completion of the requests associated with these records. The post-condition of flush() is that any previously sent record will have completed (e.g. Future.isDone() == true). A request is considered completed when it is successfully acknowledged according to the acks configuration you have specified or else it results in an error.
The RecordMetadata contains the offset and the partition
public int partition()
The partition the record was sent to
public long offset()
the offset of the record, or -1 if {hasOffset()} returns false.
Or you can also use Callback function to ensure messages was sent to topic or not
Fully non-blocking usage can make use of the Callback parameter to provide a callback that will be invoked when the request is complete.
here is clear example in docs
ProducerRecord<byte[],byte[]> record = new ProducerRecord<byte[],byte[]>("the-topic", key, value);
producer.send(myRecord,
new Callback() {
public void onCompletion(RecordMetadata metadata, Exception e) {
if(e != null) {
e.printStackTrace();
} else {
System.out.println("The offset of the record we just sent is: " + metadata.offset());
}
}
});

You can try get() API of send , which will return the Future of RecordMetadata
ProducerRecord<String, String> record =
new ProducerRecord<>("SampleTopic", "SampleKey", "SampleValue");
try {
producer.send(record).get();
} catch (Exception e) {
e.printStackTrace();
}

Use exactly-once-delivery and you won't need to worry about whether your message reached or not: https://www.baeldung.com/kafka-exactly-once, https://www.confluent.io/blog/exactly-once-semantics-are-possible-heres-how-apache-kafka-does-it/

How to set message Id for IBM MQ using java program

I am able to set correlation id for IBM mq but unable to set message id for the message the message id I am setting is being overridden by the MQ how to set this message id below one is the code I am trying please help me on this task. Is there any thing I need do in the code???
public static void main(String args[])
{
try{
MQQueueConnectionFactory cf = new MQQueueConnectionFactory();
cf.setHostName("xxx");
cf.setPort(4444);
cf.setTransportType(1);
cf.setQueueManager("xxxx");
cf.setChannel("CLIENT.xyZ");
MQQueueConnection connection = (MQQueueConnection) cf.createQueueConnection();
MQQueueSession session = (MQQueueSession) connection.createQueueSession(false, Session.AUTO_ACKNOWLEDGE);
MQQueue queue = (MQQueue) session.createQueue("WW.ESB.ENTRY.SERVICE.IN");
queue.setBooleanProperty(WMQConstants.WMQ_MQMD_WRITE_ENABLED, true);
queue.setIntProperty(WMQConstants.WMQ_MQMD_MESSAGE_CONTEXT, WMQConstants.WMQ_MDCTX_SET_IDENTITY_CONTEXT);
MQQueueSender sender = (MQQueueSender) session.createSender(queue);
true);
File f=new File("C:/InputPayloads/Payloads/test4.xml");
JMSTextMessage message = (JMSTextMessage) session.createTextMessage(FileUtils.readFileToString(f));
message.setStringProperty("JMS_IBM_MQMD_UserIdentifier", "avada2");
// Hex-string 010203040506070801020304050607080102030405060708
byte[] customMessageId = new byte[24];
for (int i = 0; i < 24; i++) {
customMessageId[i] = (byte) ((i % 8) + 1);
}
message.setObjectProperty(WMQConstants.JMS_IBM_MQMD_MSGID, customMessageId);
message.setStringProperty("xxx", "SH_TEST04");
message.setStringProperty("yyy", "JP");
message.setStringProperty("zzz", "1");
connection.start();
System.out.println("before Sent message:\\n" + message);
sender.send(message);
System.out.println("Sent message:\\n" + message);
sender.close();
session.close();
connection.close();
}catch(Exception e)
{
System.out.println(e);
}
}
}
I am getting below error
com.ibm.msg.client.jms.DetailedJMSSecurityException: JMSWMQ2008: Failed to open MQ queue 'WW.zzz.xxx.yyy.zz'.
JMS attempted to perform an MQOPEN, but IBM MQ reported an error.
Use the linked exception to determine the cause of this error. Check that the specified queue and queue manager are defined correctly.
due to this line

The JMS Spec indicates that the message ID must be set by the JMS provider and that it must either be unique or null, i.e. you can't set it yourself.
However, you can use an IBM MQ specific extension to set the Message ID yourself, bearing in mind that you are now breaking the JMS Spec.
To do so, you need to set JMS_IBM_MQMD_MsgId, whose value is then copied into JMSMessageID (i.e. you can't set it directly).
Now you know the name of the attribute to set, see this other question for more details and a code example in an answer from an IBM MQ JMS expert (#Calanais).
Further reading
JMS message object properties
Reading and writing the message descriptor from a WebSphere MQ classes for JMS application

Remove message from a queue in ActiveMq

I have an application with several activeMq queues. I would like to list the messages in them and remove any of them from any of the queues based on the id of the message.
Here is my code so far.
public void killMessage(String id) {
try {
ActiveMQConnection activeMqConnection = (ActiveMQConnection) connectionFactory.createConnection();
activeMqConnection.start();
DestinationSource destinationSource = activeMqConnection.getDestinationSource();
Set<ActiveMQQueue> queues = destinationSource.getQueues();
QueueSession queueSession = activeMqConnection.createQueueSession(true, Session.CLIENT_ACKNOWLEDGE);
for(ActiveMQQueue queue : queues) {
QueueBrowser browser = queueSession.createBrowser(queue);
Enumeration<?> messagesInQueue = browser.getEnumeration();
while (messagesInQueue.hasMoreElements()) {
Message message = (Message) messagesInQueue.nextElement();
System.out.println("Current id: " + message.getJMSMessageID());
if(message.getJMSMessageID().equals(id)){
System.out.println("-----message id found-------");
}
}
}
activeMqConnection.close();
} catch (JMSException e) {
e.printStackTrace();
}
}
I iterate through all the queues, then I iterate through all messages in each queue. I even find the message I want to delete, but I cannot find a way to remove it from the queue.
Edit:
I also created a consumer. I am not sure how the consumer should make the messages disappear from the queue. My attempt at it, which have no effect at all, messages remain in the queue, and I get no error message and no exception is thrown which could indicate the consumer did not match a message:
if(message.getJMSMessageID().equals(id)){
System.out.println("-----message id found-------");
MessageConsumer consumer = queueSession.createConsumer(queue, "JMSMessageID='" + id + "'");
consumer.receive();
consumer.close();
}

If you want to use the JMS API to do this then you'll have to create a consumer and use a selector to consume the message with the ID that you want. A queue browser cannot consume messages; it can only browse them.
In the code you pasted you're creating a transacted session which means when you consume the message you'll need to commit the session otherwise the message will never be acknowledged. That said, you're probably better off creating a non-transacted session with AUTO_ACKNOWLEDGE instead.
Also, you probably want to call receive(int) (i.e. with a timeout) so that if the selector can't find the message for some reason your application doesn't just sit there forever waiting on the method to return.

rabbitmq exchange routing key matching unexpectedly

I am working on a POC for RabbitMQ for an M2M solution. I have a large number of physical devices that will be publishing data (simulating clients using the Java client for now - eventually over MQTT). I want to:
subscribe and journal all raw data to the database
subscribe to sub-sets of the data by data type so I can scale
solutions for those types of data independently
publish new events through the exchange (e.g. take a raw event, make
it more useful and resubmit it through the system)
Each message has a routing key like key:value.key:value.key:value.messageType:1 and data from the devices has an extra key of FROMDEVICE.MESSAGETYPE:1.key:value... etc. The subscriber that saves the raw data from the device builds a queue from the exchange with the routing key #.FROMDEVICE.# (case #1 above). The subscriber that takes a specific message type and value-adds it builds a queue with the routing key #.MESSAGETYPE:1.# (case #2 above) and submits a new message to the same exchange removing FROMDEVICE from the routing key and replacing .MESSAGETYPE:1 with .MESSAGETYPE:101 (case #3 above). There is then an independent subscriber/queue for the new message type.
Everything is fine except my subscriber that should only receive the data from the devices is also getting the value added data (MESSAGETYPE:101) even though the routingKey it should be searching for does not exist in the re-published/value-added message.
FROMDEVICE.MESSAGETYPE:1 ->
should match routing key #.FROMDEVICE.#
should match #.MESSAGETYPE:1.#
MESSAGETYPE:101
should match routing key #.MESSAGETYPE:101.#
should NOT match #.FROMDEVICE.# (but does)
Code to subscribe to data from devices only:
public class HandlerWriteEverythingFromDevice {
private final static String EXCHANGE_NAME = "logsTopicDurable";
private final static String QUEUE_NAME = "fromDevice";
/**
* Writes all data from device to a data store.
*/
public static void main(String[] args) throws java.io.IOException, java.lang.InterruptedException {
ConnectionFactory factory = new ConnectionFactory();
factory.setHost("192.168.56.101");
Connection connection = factory.newConnection();
Channel channel = connection.createChannel();
channel.exchangeDeclare(EXCHANGE_NAME, "topic", true);
channel.queueDeclare(QUEUE_NAME, true, false, false, null);
System.out.println(" [*] listens for messages from devices - durable!");
channel.basicQos(1);
String routingKey = "#.fromDevice.#".toUpperCase();
channel.queueBind(QUEUE_NAME, EXCHANGE_NAME, routingKey); //bind to all selected messages
System.out.println(" [*] subscribing to: " + routingKey);
System.out.println(" [*] Waiting for messages. To exit press CTRL_C");
QueueingConsumer consumer = new QueueingConsumer(channel);
boolean autoAck = false; //ack back when done
channel.basicConsume(QUEUE_NAME, autoAck, consumer);
int msgCount = 0;
while (true) {
QueueingConsumer.Delivery delivery = consumer.nextDelivery();
String message = new String(delivery.getBody());
System.out.println(" [x] Message Count: " + ++msgCount + " ROUTINGKEY: '" + delivery.getEnvelope().getRoutingKey() + "\n MESSAGE: '" + message + "'");
Thread.sleep(250); //simulate some time to insert into the db.
channel.basicAck(delivery.getEnvelope().getDeliveryTag(), false);
}
}
}
Code to subscribe only to messageType:1 and re-publish messageType:101
private final static String EXCHANGE_NAME = "logsTopicDurable";
private final static String QUEUE_NAME = "messageType1";
/**
* Handler for messageType:1
*/
public static void main(String[] args) throws java.io.IOException, java.lang.InterruptedException {
ConnectionFactory factory = new ConnectionFactory();
factory.setHost("192.168.56.101");
Connection connection = factory.newConnection();
Channel channel = connection.createChannel();
channel.exchangeDeclare(EXCHANGE_NAME, "topic", true);
channel.queueDeclare(QUEUE_NAME, true, false, false, null);
System.out.println(" [*] listens for messageType:1 and submits messageType:101");
channel.basicQos(1);
String routingKey = "#.messageType:1.#".toUpperCase();
channel.queueBind(QUEUE_NAME, EXCHANGE_NAME, routingKey); //bind to all selected messages
System.out.println(" [*] subscribing to: " + routingKey);
System.out.println(" [*] Waiting for messages. To exit press CTRL_C");
QueueingConsumer consumer = new QueueingConsumer(channel);
boolean autoAck = false; //ack back when done
channel.basicConsume(QUEUE_NAME, autoAck, consumer);
int msgCount = 0;
while (true) {
QueueingConsumer.Delivery delivery = consumer.nextDelivery();
String message = new String(delivery.getBody());
System.out.println(" [x] Message Count: " + ++msgCount + " ROUTINGKEY: '" + delivery.getEnvelope().getRoutingKey() + "\n MESSAGE: '" + message + "'");
channel.basicPublish(EXCHANGE_NAME,
delivery.getEnvelope().
getRoutingKey().
replaceAll("messageType:1", "messageType:101").
replaceAll(".FROMDEVICE", "").
replaceAll("FROMDEVICE.", "").trim(),
true,
MessageProperties.PERSISTENT_BASIC,
message.getBytes());
channel.basicAck(delivery.getEnvelope().getDeliveryTag(), false);
}
}
There is publisher code and subscriber code for messageType:101 but I don't think they are needed for this discussion. I've wondered if publishing to a channel that has a queue bound to it might be the cause, but I tried creating two channels (same connection object) and had the same result and a lot uglier code.

I would suggest that you are being a bit to liberal with your binding keys. To make things a little more clear you should use the term binding key and routing key differently. The routing key is what is sent by the producer. The binding key is what you use to bind the queue to the topic exchange.
As I cannot be sure which you are referring to when you say
"should match routing key #.MESSAGETYPE:101.#"
are you sending a message with a routing key #.MESSAGETYPE:101.# because that would be a bad idea. I presume not, but if you are don't!
Lets assume then that this is your binding key. I am not sure as I haven't done any testing of this specifically but the # before and after maybe causing some problems. You should think about a specification for your routing keys. Some format that they must conform to. It may be extendable but not completely free. That way you can have much more specific binding keys using * instead of # which will give a little more control.

Turning onMessage() method into an atomic action

I've encounter the problem that if my method below fails or it's an exception I still consume the msg. I would want the functionality to do a rollback during the catch and place the msg back on the queue/topic.
public void onMessage(Message message)
{
String messageId = null;
Date messagePublished = null;
try
{
messageId = message.getJMSMessageID();
messagePublished = new Date(message.getJMSTimestamp());
LOGGER.info("JMS Message id =" + messageId + " JMS Timestamp= " + messagePublished);
process(message);
LOGGER.info(" returning from onMessage() successfully =" + messageId + " JMS Timestamp= " + messagePublished);
}
catch(Throwable t)
{
LOGGER.error("Exception:",t);
LOGGER.error(t.getStackTrace() + "\n Exception is unrecoverable.");
throw new RuntimeException("Failed to handle message.",t);
}
}

You can look at the different acknowledge modes that exist within JMS for this. See this article http://www.javaworld.com/javaworld/jw-02-2002/jw-0315-jms.html.
The appropriate mode for you would be Client mode.
So basically, the client needs to acknowledge when they are happy they have processed the message.
You could call the acknowledge after the call to process(message), if an exception occurs in the proccess(message) method, the message will not have been dequeued as you didnt acknowledge it. We used this approach before with Oracle AQ and it works very well.
This approach means you dont have to worry about transactions for the messages on the queue (Database transactions are another story). The only thing you need to ensure is that your app can handle a call to process(message) with potential duplicate messages

you should be able to just make your onMessage method transacted.