I am working on a POC for RabbitMQ for an M2M solution. I have a large number of physical devices that will be publishing data (simulating clients using the Java client for now - eventually over MQTT). I want to:
subscribe and journal all raw data to the database
subscribe to sub-sets of the data by data type so I can scale
solutions for those types of data independently
publish new events through the exchange (e.g. take a raw event, make
it more useful and resubmit it through the system)
Each message has a routing key like key:value.key:value.key:value.messageType:1 and data from the devices has an extra key of FROMDEVICE.MESSAGETYPE:1.key:value... etc. The subscriber that saves the raw data from the device builds a queue from the exchange with the routing key #.FROMDEVICE.# (case #1 above). The subscriber that takes a specific message type and value-adds it builds a queue with the routing key #.MESSAGETYPE:1.# (case #2 above) and submits a new message to the same exchange removing FROMDEVICE from the routing key and replacing .MESSAGETYPE:1 with .MESSAGETYPE:101 (case #3 above). There is then an independent subscriber/queue for the new message type.
Everything is fine except my subscriber that should only receive the data from the devices is also getting the value added data (MESSAGETYPE:101) even though the routingKey it should be searching for does not exist in the re-published/value-added message.
FROMDEVICE.MESSAGETYPE:1 ->
should match routing key #.FROMDEVICE.#
should match #.MESSAGETYPE:1.#
MESSAGETYPE:101
should match routing key #.MESSAGETYPE:101.#
should NOT match #.FROMDEVICE.# (but does)
Code to subscribe to data from devices only:
public class HandlerWriteEverythingFromDevice {
private final static String EXCHANGE_NAME = "logsTopicDurable";
private final static String QUEUE_NAME = "fromDevice";
/**
* Writes all data from device to a data store.
*/
public static void main(String[] args) throws java.io.IOException, java.lang.InterruptedException {
ConnectionFactory factory = new ConnectionFactory();
factory.setHost("192.168.56.101");
Connection connection = factory.newConnection();
Channel channel = connection.createChannel();
channel.exchangeDeclare(EXCHANGE_NAME, "topic", true);
channel.queueDeclare(QUEUE_NAME, true, false, false, null);
System.out.println(" [*] listens for messages from devices - durable!");
channel.basicQos(1);
String routingKey = "#.fromDevice.#".toUpperCase();
channel.queueBind(QUEUE_NAME, EXCHANGE_NAME, routingKey); //bind to all selected messages
System.out.println(" [*] subscribing to: " + routingKey);
System.out.println(" [*] Waiting for messages. To exit press CTRL_C");
QueueingConsumer consumer = new QueueingConsumer(channel);
boolean autoAck = false; //ack back when done
channel.basicConsume(QUEUE_NAME, autoAck, consumer);
int msgCount = 0;
while (true) {
QueueingConsumer.Delivery delivery = consumer.nextDelivery();
String message = new String(delivery.getBody());
System.out.println(" [x] Message Count: " + ++msgCount + " ROUTINGKEY: '" + delivery.getEnvelope().getRoutingKey() + "\n MESSAGE: '" + message + "'");
Thread.sleep(250); //simulate some time to insert into the db.
channel.basicAck(delivery.getEnvelope().getDeliveryTag(), false);
}
}
}
Code to subscribe only to messageType:1 and re-publish messageType:101
private final static String EXCHANGE_NAME = "logsTopicDurable";
private final static String QUEUE_NAME = "messageType1";
/**
* Handler for messageType:1
*/
public static void main(String[] args) throws java.io.IOException, java.lang.InterruptedException {
ConnectionFactory factory = new ConnectionFactory();
factory.setHost("192.168.56.101");
Connection connection = factory.newConnection();
Channel channel = connection.createChannel();
channel.exchangeDeclare(EXCHANGE_NAME, "topic", true);
channel.queueDeclare(QUEUE_NAME, true, false, false, null);
System.out.println(" [*] listens for messageType:1 and submits messageType:101");
channel.basicQos(1);
String routingKey = "#.messageType:1.#".toUpperCase();
channel.queueBind(QUEUE_NAME, EXCHANGE_NAME, routingKey); //bind to all selected messages
System.out.println(" [*] subscribing to: " + routingKey);
System.out.println(" [*] Waiting for messages. To exit press CTRL_C");
QueueingConsumer consumer = new QueueingConsumer(channel);
boolean autoAck = false; //ack back when done
channel.basicConsume(QUEUE_NAME, autoAck, consumer);
int msgCount = 0;
while (true) {
QueueingConsumer.Delivery delivery = consumer.nextDelivery();
String message = new String(delivery.getBody());
System.out.println(" [x] Message Count: " + ++msgCount + " ROUTINGKEY: '" + delivery.getEnvelope().getRoutingKey() + "\n MESSAGE: '" + message + "'");
channel.basicPublish(EXCHANGE_NAME,
delivery.getEnvelope().
getRoutingKey().
replaceAll("messageType:1", "messageType:101").
replaceAll(".FROMDEVICE", "").
replaceAll("FROMDEVICE.", "").trim(),
true,
MessageProperties.PERSISTENT_BASIC,
message.getBytes());
channel.basicAck(delivery.getEnvelope().getDeliveryTag(), false);
}
}
There is publisher code and subscriber code for messageType:101 but I don't think they are needed for this discussion. I've wondered if publishing to a channel that has a queue bound to it might be the cause, but I tried creating two channels (same connection object) and had the same result and a lot uglier code.
I would suggest that you are being a bit to liberal with your binding keys. To make things a little more clear you should use the term binding key and routing key differently. The routing key is what is sent by the producer. The binding key is what you use to bind the queue to the topic exchange.
As I cannot be sure which you are referring to when you say
"should match routing key #.MESSAGETYPE:101.#"
are you sending a message with a routing key #.MESSAGETYPE:101.# because that would be a bad idea. I presume not, but if you are don't!
Lets assume then that this is your binding key. I am not sure as I haven't done any testing of this specifically but the # before and after maybe causing some problems. You should think about a specification for your routing keys. Some format that they must conform to. It may be extendable but not completely free. That way you can have much more specific binding keys using * instead of # which will give a little more control.
Related
I'm writing code in java (using Azure SDK for Java), I have a Service bus queue that contains sessionful messages. I want to receive those messages and process them to another place.
I make a connection to the Queue by using QueueClient, and then I use registerSessionHandler to process through the messages (code below).
The problem is that whenever a message is received, I can print all details about it including the content, but it is printed 10 times and after each time it prints an Exception.
(printing 10 times: I understand that this is because there is a 10 times retry policy before it throws the message to the Dead letter queue and goes to the next message.)
The Exception says
> USERCALLBACK-Receiver not created. Registering a MessageHandler creates a receiver.
The output with the Exception
But I'm sure that the SessionHandler does the same thing as MessageHandler but includes support for sessions, so it should create a receiver since it receives messages. I have tried to use MessageHandler but it won't even work and stops the whole program because it doesn't support sessionful messages, and the ones I receive have sessions.
My problem is understanding what the Exception wants me to do, and how can I fix the code so it won't give me any exceptions? Does anyone have suggestions on how to improve the code? or other methods that do the same thing?
QueueClient qc = new QueueClient(
new ConnectionStringBuilder(connectionString),
ReceiveMode.PEEKLOCK);
qc.registerSessionHandler(
new ISessionHandler() {
#Override
public CompletableFuture<Void> onMessageAsync(IMessageSession messageSession, IMessage message) {
System.out.printf(
"\nMessage received: " +
"\n --> MessageId = %s " +
"\n --> SessionId = %s" +
"\n --> Content Type = %s" +
"\n --> Content = \n\t\t %s",
message.getMessageId(),
messageSession.getSessionId(),
message.getContentType(),
getMessageContent(message)
);
return qc.completeAsync(message.getLockToken());
}
#Override
public CompletableFuture<Void> OnCloseSessionAsync(IMessageSession iMessageSession) {
return CompletableFuture.completedFuture(null);
}
#Override
public void notifyException(Throwable throwable, ExceptionPhase exceptionPhase) {
System.out.println("\n Exception " + exceptionPhase + "-" + throwable.getMessage());
}
},
new SessionHandlerOptions(1, true, Duration.ofMinutes(1)),
Executors.newSingleThreadExecutor()
);
(The getMessageContent(message) method is a separate method, for those interested:)
public String getMessageContent(IMessage message){
List<byte[]> content = message.getMessageBody().getBinaryData();
StringBuilder sb = new StringBuilder();
for (byte[] b : content) {
sb.append(new String(b)
);
}
return sb.toString();
}
For those who wonder, I managed to solve the problem!
It was simply done by using Azure Functions ServiceBusQueueTrigger, it will then listen to the Service bus Queue and process the messages. By setting isSessionsEnabled to true, it will accept sessionful messages as I wanted :)
So instead of writing more than 100 lines of code, the code looks like this now:
public class Function {
#FunctionName("QueueFunction")
public void run(
#ServiceBusQueueTrigger(
name = "TriggerName", //Any name you choose
queueName = "queueName", //QueueName from the portal
connection = "ConnectionString", //ConnectionString from the portal
isSessionsEnabled = true
) String message,
ExecutionContext context
) {
// Write the code you want to do with the message here
// Using the variable messsage which contains the messageContent, messageId, sessionId etc.
}
}
I am trying to send rabbit mq message on a exchange - exchange-X to a message queue - queque-X with routing key -mc, its being received well on my local rabbit mq but on production rabbit mq the message does not appear. The exchange and the queue is binded with the specified routing key. In the below message isSent is true always but actually message does not reach at the queue only on prod rabbitmq env. Is routing key mc case sensitive ?
public void sendMessageCenterNotification(Map<String, Object> headerMap,String correlationId,String message) {
boolean isSent = false;
try {
isSent = rabbitMQ.messageSender(message, headerMap, "mc", correlationId);
} catch (Exception e) {
logger.error(correlationId + " - Exception occured in sendMessageCenterNotification:", e);
} finally {
logger.info(correlationId
+ "-inside sendMessageCenterNotification message sending to message center was "+(isSent?"successfull":"failed")+", message:"
+ message);
}
}
From the documentation:
A fanout exchange routes messages to all of the queues that are bound
to it and the routing key is ignored.
You probably want a different type of exchange for what you are trying to accomplish.
As a side note, the protocol (0.9) reference does not appear to mention case sensitivity, but in the client implementations routing keys are case-sensitive.
I am able to set correlation id for IBM mq but unable to set message id for the message the message id I am setting is being overridden by the MQ how to set this message id below one is the code I am trying please help me on this task. Is there any thing I need do in the code???
public static void main(String args[])
{
try{
MQQueueConnectionFactory cf = new MQQueueConnectionFactory();
cf.setHostName("xxx");
cf.setPort(4444);
cf.setTransportType(1);
cf.setQueueManager("xxxx");
cf.setChannel("CLIENT.xyZ");
MQQueueConnection connection = (MQQueueConnection) cf.createQueueConnection();
MQQueueSession session = (MQQueueSession) connection.createQueueSession(false, Session.AUTO_ACKNOWLEDGE);
MQQueue queue = (MQQueue) session.createQueue("WW.ESB.ENTRY.SERVICE.IN");
queue.setBooleanProperty(WMQConstants.WMQ_MQMD_WRITE_ENABLED, true);
queue.setIntProperty(WMQConstants.WMQ_MQMD_MESSAGE_CONTEXT, WMQConstants.WMQ_MDCTX_SET_IDENTITY_CONTEXT);
MQQueueSender sender = (MQQueueSender) session.createSender(queue);
true);
File f=new File("C:/InputPayloads/Payloads/test4.xml");
JMSTextMessage message = (JMSTextMessage) session.createTextMessage(FileUtils.readFileToString(f));
message.setStringProperty("JMS_IBM_MQMD_UserIdentifier", "avada2");
// Hex-string 010203040506070801020304050607080102030405060708
byte[] customMessageId = new byte[24];
for (int i = 0; i < 24; i++) {
customMessageId[i] = (byte) ((i % 8) + 1);
}
message.setObjectProperty(WMQConstants.JMS_IBM_MQMD_MSGID, customMessageId);
message.setStringProperty("xxx", "SH_TEST04");
message.setStringProperty("yyy", "JP");
message.setStringProperty("zzz", "1");
connection.start();
System.out.println("before Sent message:\\n" + message);
sender.send(message);
System.out.println("Sent message:\\n" + message);
sender.close();
session.close();
connection.close();
}catch(Exception e)
{
System.out.println(e);
}
}
}
I am getting below error
com.ibm.msg.client.jms.DetailedJMSSecurityException: JMSWMQ2008: Failed to open MQ queue 'WW.zzz.xxx.yyy.zz'.
JMS attempted to perform an MQOPEN, but IBM MQ reported an error.
Use the linked exception to determine the cause of this error. Check that the specified queue and queue manager are defined correctly.
due to this line
The JMS Spec indicates that the message ID must be set by the JMS provider and that it must either be unique or null, i.e. you can't set it yourself.
However, you can use an IBM MQ specific extension to set the Message ID yourself, bearing in mind that you are now breaking the JMS Spec.
To do so, you need to set JMS_IBM_MQMD_MsgId, whose value is then copied into JMSMessageID (i.e. you can't set it directly).
Now you know the name of the attribute to set, see this other question for more details and a code example in an answer from an IBM MQ JMS expert (#Calanais).
Further reading
JMS message object properties
Reading and writing the message descriptor from a WebSphere MQ classes for JMS application
I need to store SMS (when they arrive to the GSM modem) in MySQL.
I have read the docs, but it says to use the smsserver and the conffile.
Then I read that I should use database.java.
Can someone guide me on this?
I am already using the ReadMessages.java and I have achieved storing in a MySQL database, but I have managed to store only the ones that are in the SIM card memory, not the inbound messages.
The inbound messages are just printed by the console, I read that this can do with threads.
This is my code:
public void doIt() throws Exception
{
// Define a list which will hold the read messages.
List<InboundMessage> msgList;
// Create the notification callback method for inbound & status report
// messages.
InboundNotification inboundNotification = new InboundNotification();
// Create the notification callback method for inbound voice calls.
CallNotification callNotification = new CallNotification();
//Create the notification callback method for gateway statuses.
GatewayStatusNotification statusNotification = new GatewayStatusNotification();
OrphanedMessageNotification orphanedMessageNotification = new OrphanedMessageNotification();
try
{
//System.out.println("Example: Read messages from a serial gsm modem.");
//System.out.println(Library.getLibraryDescription());
//System.out.println("Version: " + Library.getLibraryVersion());
// Create the Gateway representing the serial GSM modem.
SerialModemGateway gateway = new SerialModemGateway("modem.com1", "COM4", 115200, "Huawei", "E160");
// Set the modem protocol to PDU (alternative is TEXT). PDU is the default, anyway...
gateway.setProtocol(Protocols.PDU);
// Do we want the Gateway to be used for Inbound messages?
gateway.setInbound(true);
// Do we want the Gateway to be used for Outbound messages?
gateway.setOutbound(true);
// Let SMSLib know which is the SIM PIN.
gateway.setSimPin("0000");
// Set up the notification methods.
Service.getInstance().setInboundMessageNotification(inboundNotification);
Service.getInstance().setCallNotification(callNotification);
Service.getInstance().setGatewayStatusNotification(statusNotification);
Service.getInstance().setOrphanedMessageNotification(orphanedMessageNotification);
// Add the Gateway to the Service object.
Service.getInstance().addGateway(gateway);
// Similarly, you may define as many Gateway objects, representing
// various GSM modems, add them in the Service object and control all of them.
// Start! (i.e. connect to all defined Gateways)
Service.getInstance().startService();
// Printout some general information about the modem.
System.out.println();
System.out.println("Informacion del modem:");
System.out.println(" Fabricante: " + gateway.getManufacturer());
System.out.println(" Modelo: " + gateway.getModel());
System.out.println(" Serial No: " + gateway.getSerialNo());
System.out.println(" SIM IMSI: " + gateway.getImsi());
//System.out.println(" Signal Level: " + gateway.getSignalLevel() + " dBm");
//System.out.println(" Battery Level: " + gateway.getBatteryLevel() + "%");
System.out.println();
// In case you work with encrypted messages, its a good time to declare your keys.
// Create a new AES Key with a known key value.
// Register it in KeyManager in order to keep it active. SMSLib will then automatically
// encrypt / decrypt all messages send to / received from this number.
Service.getInstance().getKeyManager().registerKey("+306948494037", new AESKey(new SecretKeySpec("0011223344556677".getBytes(), "AES")));
// Read Messages. The reading is done via the Service object and
// affects all Gateway objects defined. This can also be more directed to a specific
// Gateway - look the JavaDocs for information on the Service method calls.
ConexionMySQL mysql = new ConexionMySQL();
Connection cnn = mysql.Conectar();
String sms= "";
String originator="";
String sql ="";
int n = 0;
String success="Mensajee satisfactorio";
msgList = new ArrayList<InboundMessage>();
Service.getInstance().readMessages(msgList, MessageClasses.ALL);
for (InboundMessage msg : msgList){
//System.out.println(msg);
sms = msg.getText().toString();
remitente = msg.getOriginator().toString();
//fecha = msg.getDate().getTime();
sql ="INSERT INTO message(message, originator) VALUES (?,?)";
PreparedStatement post = (PreparedStatement) cnn.prepareStatement(sql);
post.setString(1, message);
post.setString(2, originator);
//post.setDate(3, fecha); public void doIt() throws Exception
{
// Define a list which will hold the read messages.
List<InboundMessage> msgList;
// Create the notification callback method for inbound & status report
// messages.
InboundNotification inboundNotification = new InboundNotification();
// Create the notification callback method for inbound voice calls.
CallNotification callNotification = new CallNotification();
//Create the notification callback method for gateway statuses.
GatewayStatusNotification statusNotification = new GatewayStatusNotification();
OrphanedMessageNotification orphanedMessageNotification = new OrphanedMessageNotification();
try
{
//System.out.println("Example: Read messages from a serial gsm modem.");
//System.out.println(Library.getLibraryDescription());
//System.out.println("Version: " + Library.getLibraryVersion());
// Create the Gateway representing the serial GSM modem.
SerialModemGateway gateway = new SerialModemGateway("modem.com1", "COM4", 115200, "Huawei", "E160");
// Set the modem protocol to PDU (alternative is TEXT). PDU is the default, anyway...
gateway.setProtocol(Protocols.PDU);
// Do we want the Gateway to be used for Inbound messages?
gateway.setInbound(true);
// Do we want the Gateway to be used for Outbound messages?
gateway.setOutbound(true);
// Let SMSLib know which is the SIM PIN.
gateway.setSimPin("0000");
// Set up the notification methods.
Service.getInstance().setInboundMessageNotification(inboundNotification);
Service.getInstance().setCallNotification(callNotification);
Service.getInstance().setGatewayStatusNotification(statusNotification);
Service.getInstance().setOrphanedMessageNotification(orphanedMessageNotification);
// Add the Gateway to the Service object.
Service.getInstance().addGateway(gateway);
// Similarly, you may define as many Gateway objects, representing
// various GSM modems, add them in the Service object and control all of them.
// Start! (i.e. connect to all defined Gateways)
Service.getInstance().startService();
// Printout some general information about the modem.
System.out.println();
System.out.println("Informacion del modem:");
System.out.println(" Fabricante: " + gateway.getManufacturer());
System.out.println(" Modelo: " + gateway.getModel());
System.out.println(" Serial No: " + gateway.getSerialNo());
System.out.println(" SIM IMSI: " + gateway.getImsi());
//System.out.println(" Signal Level: " + gateway.getSignalLevel() + " dBm");
//System.out.println(" Battery Level: " + gateway.getBatteryLevel() + "%");
System.out.println();
// In case you work with encrypted messages, its a good time to declare your keys.
// Create a new AES Key with a known key value.
// Register it in KeyManager in order to keep it active. SMSLib will then automatically
// encrypt / decrypt all messages send to / received from this number.
Service.getInstance().getKeyManager().registerKey("+306948494037", new AESKey(new SecretKeySpec("0011223344556677".getBytes(), "AES")));
// Read Messages. The reading is done via the Service object and
// affects all Gateway objects defined. This can also be more directed to a specific
// Gateway - look the JavaDocs for information on the Service method calls.
ConexionMySQL mysql = new ConexionMySQL();
Connection cnn = mysql.Conectar();
String sms= "";
String remitente="";
//Date fecha;
String sql ="";
int n = 0;
String mensaje="Mensajee satisfactorio";
msgList = new ArrayList<InboundMessage>();
Service.getInstance().readMessages(msgList, MessageClasses.ALL);
for (InboundMessage msg : msgList){
//System.out.println(msg);
sms = msg.getText().toString();
remitente = msg.getOriginator().toString();
//fecha = msg.getDate().getTime();
sql ="INSERT INTO mensaje(mensaje, remitente) VALUES (?,?)";
PreparedStatement post = (PreparedStatement) cnn.prepareStatement(sql);
post.setString(1, sms);
post.setString(2, originator);
n = post.executeUpdate();
if(n >0){
JOptionPane.showMessageDialog(null, mensaje);
Service.getInstance().deleteMessage(msg);
}
// Sleep now. Emulate real world situation and give a chance to the notifications
// methods to be called in the event of message or voice call reception.
System.out.println("Now Sleeping - Hit <enter> to stop service.");
System.in.read();
System.in.read();
}
}
catch (Exception e)
{
e.printStackTrace();
}
finally
{
Service.getInstance().stopService();
}
}
n = post.executeUpdate();
if(n >0){
JOptionPane.showMessageDialog(null, mensaje);
Service.getInstance().deleteMessage(msg);
}
// Sleep now. Emulate real world situation and give a chance to the notifications
// methods to be called in the event of message or voice call reception.
System.out.println("Now Sleeping - Hit <enter> to stop service.");
System.in.read();
System.in.read();
}
}
catch (Exception e)
{
e.printStackTrace();
}
finally
{
Service.getInstance().stopService();
}
}
You should check where you've put the InboundNotification class.
The messages you're seeing from the console with inbound sms comes from there
public class InboundNotification implements IInboundMessageNotification
{
public void process(AGateway gateway, MessageTypes msgType, InboundMessage msg)
{
if (msgType == MessageTypes.INBOUND) System.out.println(">>> New Inbound message detected from Gateway: " + gateway.getGatewayId());
else if (msgType == MessageTypes.STATUSREPORT) System.out.println(">>> New Inbound Status Report message detected from Gateway: " + gateway.getGatewayId());
System.out.println(msg);
}
}
I've encounter the problem that if my method below fails or it's an exception I still consume the msg. I would want the functionality to do a rollback during the catch and place the msg back on the queue/topic.
public void onMessage(Message message)
{
String messageId = null;
Date messagePublished = null;
try
{
messageId = message.getJMSMessageID();
messagePublished = new Date(message.getJMSTimestamp());
LOGGER.info("JMS Message id =" + messageId + " JMS Timestamp= " + messagePublished);
process(message);
LOGGER.info(" returning from onMessage() successfully =" + messageId + " JMS Timestamp= " + messagePublished);
}
catch(Throwable t)
{
LOGGER.error("Exception:",t);
LOGGER.error(t.getStackTrace() + "\n Exception is unrecoverable.");
throw new RuntimeException("Failed to handle message.",t);
}
}
You can look at the different acknowledge modes that exist within JMS for this. See this article http://www.javaworld.com/javaworld/jw-02-2002/jw-0315-jms.html.
The appropriate mode for you would be Client mode.
So basically, the client needs to acknowledge when they are happy they have processed the message.
You could call the acknowledge after the call to process(message), if an exception occurs in the proccess(message) method, the message will not have been dequeued as you didnt acknowledge it. We used this approach before with Oracle AQ and it works very well.
This approach means you dont have to worry about transactions for the messages on the queue (Database transactions are another story). The only thing you need to ensure is that your app can handle a call to process(message) with potential duplicate messages
you should be able to just make your onMessage method transacted.