We Keep Coding

Browse Jms Queue in a multiThreading way

I have a spring-batch which browses Queue messages, this queue supposed to contain a huge number of messages. then it takes a lot of time to treat all of them. therefore i thought about multi-Threading to deal with the issue, but it's not clear for me yet.
Here is an example of browsing a Queue without multithreading:
import java.net.URISyntaxException;
import java.util.Enumeration;
import javax.jms.Connection;
import javax.jms.ConnectionFactory;
import javax.jms.Message;
import javax.jms.MessageConsumer;
import javax.jms.MessageProducer;
import javax.jms.Queue;
import javax.jms.QueueBrowser;
import javax.jms.Session;
import javax.jms.TextMessage;
import org.apache.activemq.ActiveMQConnectionFactory;
public class JmsQueueBrowseExample {
public static void main(String[] args) throws URISyntaxException, Exception {
Connection connection = null;
try {
// Producer
ConnectionFactory connectionFactory = new ActiveMQConnectionFactory(
"tcp://localhost:61616");
connection = connectionFactory.createConnection();
Session session = connection.createSession(false,
Session.AUTO_ACKNOWLEDGE);
Queue queue = session.createQueue("browseQueue");
MessageConsumer consumer = session.createConsumer(queue);
connection.start();
System.out.println("Browse through the elements in queue");
QueueBrowser browser = session.createBrowser(queue);
Enumeration e = browser.getEnumeration();
//Multithreading here
while (e.hasMoreElements()) {
TextMessage message = (TextMessage) e.nextElement();
System.out.println("Browse [" + message.getText() + "]");
}
System.out.println("Done");
browser.close();
session.close();
} finally {
if (connection != null) {
connection.close();
}
}
}
}
Thank you

Other than the close methods the JMS API Resources such as Session, MessageConsumer, QueueBrowser etc are not meant to be used by more than a single thread of control and as such trying to concurrently iterate over the messages returned from a QueueBrowser enumeration is likely to result is errors.
The JMS specification adds some insight into concurrency with session resources.
There are no restrictions on the number of threads that can use a Session object or those it creates. The restriction is that the resources of a Session should not be used concurrently by multiple threads. It is up to the user to insure that this concurrency restriction is met. The simplest way to do this is to use one thread. In the case of asynchronous delivery, use one thread for setup in stopped mode and then start asynchronous delivery. In more complex cases the user must provide explicit synchronization.

Unable to listen the SQS messages in java SqsListener produced from the python using boto SDK

I am sending the message using the boto SDK to Amazon-SQS. Whatever the message I send it has to listen using the SqsListener written in java.When I send message I am not able to get the exact message what I have send from the producer class written in python. In SQSListener class I just tried to print the message what I have got. I am getting the encrypted form of message instead of actual message.
Actual message:
Sample message
Received Message
U2FtcGxlIG1lc3NhZ2U=
The producer class written in the python is:
from boto import sqs
import json
from boto.sqs.message import Message
conn = sqs.connect_to_region('XXXXX', aws_access_key_id='XXXXXXXXXXXX', aws_secret_access_key='XXXXXXXXXXXXXXXXXXXXXXX')
queueInstance = conn.get_queue('sample')
try:
message = Message()
message.set_body("testing sample")
queueInstance.write(message)
except Exception, error:
print error
The Receiver SQSListener class is:
import javax.jms.JMSException;
import javax.jms.Message;
import javax.jms.MessageListener;
import javax.jms.TextMessage;
public class SQSMessageLisner implements MessageListener {
public void onMessage(Message message) {
try {
System.out.println("Inside ON MESSAGE");
//System.out.println(message.get);
String stringMessage = ((TextMessage) message).getText();
System.out.println(stringMessage);
} catch (JMSException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}
}
The sending the receiving of messages are written in same SDK's are working fine. Coming to sending from python and listening in java leads to this problem. Please help me out of this. Thanks in advance.

You get a base64 encoded message in your receiver. You need to decode it, as described here Decode Base64 data in Java
Your other option is to use boto.sqs.message.RawMessage in python instead of TextMessage, which will not be encoded.

Sending message to multiple consumer in ActiveMQ

I am new to ActiveMQ. I have tried to implement producer-consumer (sender-receiver) in activemq. In my code, I am easily send & receive the messages from single producer to single consumer via ActiveMQ. But, the problem is, I can't send the message to multiple consumers from the same producer.
Here is my producer & consumer class.
MsgProducer.java
package jms_service;
import javax.jms.JMSException;
import javax.jms.Session;
import javax.jms.TextMessage;
import org.apache.activemq.ActiveMQConnectionFactory;
public class MsgProducer {
private static String url = "failover://tcp://localhost:61616";
public static javax.jms.ConnectionFactory connFactory;
public static javax.jms.Connection connection;
public static javax.jms.Session mqSession;
public static javax.jms.Topic topic;
public static javax.jms.MessageProducer producer;
public static void main(String[] args) throws JMSException {
connFactory = new ActiveMQConnectionFactory(url);
connection = connFactory.createConnection("system","manager");
connection.start();
mqSession = connection.createSession(false,Session.AUTO_ACKNOWLEDGE);
topic = mqSession.createTopic("RealTimeData");
producer = mqSession.createProducer(topic);
producer.setTimeToLive(30000);
TextMessage message = mqSession.createTextMessage();
int seq_id =1;
while(true)
{
message.setText("Hello world | " +"seq_id #"+seq_id);
producer.send(message);
seq_id++;
System.out.println("sent_msg =>> "+ message.getText());
// if(seq_id>100000) break;
try {
Thread.sleep(1000);
}
catch (InterruptedException e) { e.printStackTrace();}
}
}
}
MsgConsumer.java
package jms_service;
import java.text.SimpleDateFormat;
import java.util.Calendar;
import javax.jms.JMSException;
import javax.jms.Message;
import javax.jms.MessageListener;
import javax.jms.Session;
import javax.jms.TextMessage;
import org.apache.activemq.ActiveMQConnectionFactory;
public class MsgConsumer {
private static String url = "failover://tcp://localhost:61616";
public static javax.jms.ConnectionFactory connFactory;
public static javax.jms.Connection connection;
public static javax.jms.Session mqSession;
public static javax.jms.Topic topic;
public static javax.jms.MessageConsumer consumer;
public static void main(String[] args) throws JMSException, InterruptedException {
connFactory = new ActiveMQConnectionFactory(url);
connection = connFactory.createConnection("system", "manager");
connection.setClientID("0002");
//connection.start();
mqSession = connection.createSession(true, Session.CLIENT_ACKNOWLEDGE);
topic = mqSession.createTopic("RealTimeData");
consumer = mqSession.createDurableSubscriber(topic, "SUBS01");
connection.start();
MessageListener listner = new MessageListener() {
public void onMessage(Message message) {
try {
if (message instanceof TextMessage) {
TextMessage txtmsg = (TextMessage) message;
Calendar cal = Calendar.getInstance();
//cal.getTime();
SimpleDateFormat sdf = new SimpleDateFormat("HH:mm:ss");
String time = sdf.format(cal.getTime());
String msg="received_message =>> "+ txtmsg.getText() + " | received_at :: "+time;
System.out.println(msg);
//consumer.sendData(msg);
}
} catch (JMSException e) {
System.out.println("Caught:" + e);
e.printStackTrace();
}
}
};
consumer.setMessageListener(listner);
}
}
Can anyone help to figure out the way for sending message to multiple consumers.
Thanks in advance.

Queue semantics deliver a message once-and-only-once across all consumers. This is per the JMS spec (a great read to understand the basics).
Topic semantics deliver a message to every consumer. So, a Topic may be the answer to your needs.

Assuming your question is
Can anyone help to figure out the way for sending message to multiple consumers
and without reading through your complete code, an approach could be to put your clients in a collection
static Vector<consumer> vecConsumer;
where you put in every new client and give a reference to all existing clients.
The broadcasting is just like sending to a single client, encapsulated in, for an example, a foreach loop
for(consumer cons : vecConsumer)
{
//send stuff or put in sending queue
}

Topics is the best route. One producer to many consumers or one publisher to many subscribers. With Queues you have go write a loop to to get all the possible consumers and use different destinations to send the messages. Your motive would also determine whether to use Queues or Topics.
If u you think your consumers can be offline or have network issues then choose queues. In this case when they come back on they will receive the pending messages
With topics there is no way they will receive the message when there is a disconnection unless u explicitly persist the message however new messages would overwrite them

Java on ServiceBus for Windows Server- Options?

What are the options available to develop Java applications using Service Bus for Windows?
Java Message Broker API - This need ACS to work with, which SB for Win doesnt support.
AMQP - This doesnt seem to work on SB for Windows, I keep getting error
org.apache.qpid.amqp_1_0.client.Sender$SenderCreationException: Peer did not create remote endpoint for link, target:
While the same code works with Azure SB. So AMQP on SB for Windows seems to be not fully working?
Correct me if I have missed something?
Update
To test AMQP on local machine, this is what I did
Installed Service bus 1.1 on my local machine
Took the sample mentioned here http://www.windowsazure.com/en-us/develop/java/how-to-guides/service-bus-amqp/
Created a new namespace on my local machine
Specified the following connection string in servicebus.properties (which is correctly referred in the code
connectionfactory.SBCF = amqps://<username>:<password>#<MachineName>:5671/StringAnalyzerNS/
queue.QUEUE = queue1
Code is updated with certificates.
At runtime I get this error
javax.jms.JMSException: Peer did not create remote endpoint for link, target: queue1
at org.apache.qpid.amqp_1_0.jms.impl.MessageProducerImpl.<init>(MessageProducerImpl.java:77)
at org.apache.qpid.amqp_1_0.jms.impl.SessionImpl.createProducer(SessionImpl.java:348)
at org.apache.qpid.amqp_1_0.jms.impl.SessionImpl.createProducer(SessionImpl.java:63)
at com.stringcompany.Analyzer.SimpleSenderReceiver.<init>(SimpleSenderReceiver.java:70)
at com.stringcompany.Analyzer.SimpleSenderReceiver.main(SimpleSenderReceiver.java:95)
Caused by: org.apache.qpid.amqp_1_0.client.Sender$SenderCreationException: Peer did not create remote endpoint for link, target: queue1
at org.apache.qpid.amqp_1_0.client.Sender.<init>(Sender.java:171)
at org.apache.qpid.amqp_1_0.client.Sender.<init>(Sender.java:104)
at org.apache.qpid.amqp_1_0.client.Sender.<init>(Sender.java:97)
at org.apache.qpid.amqp_1_0.client.Sender.<init>(Sender.java:83)
at org.apache.qpid.amqp_1_0.client.Sender.<init>(Sender.java:69)
at org.apache.qpid.amqp_1_0.client.Sender.<init>(Sender.java:63)
at org.apache.qpid.amqp_1_0.client.Session.createSender(Session.java:74)
at org.apache.qpid.amqp_1_0.client.Session.createSender(Session.java:66)
at org.apache.qpid.amqp_1_0.jms.impl.MessageProducerImpl.<init>(MessageProducerImpl.java:72)
... 4 more
javax.jms.JMSException: Session remotely closed
With the same code If I point to Azure service bus by setting the SB namespace and queue like below
connectionfactory.SBCF = amqps://<Policy name>:<Sec. Key>#<ns>.servicebus.windows.net
queue.QUEUE = testq
This works, messages are exchanged.
Here is the code if someone wants to try it
package com.stringcompany.Analyzer;
//SimpleSenderReceiver.java
import java.io.BufferedReader;
import java.io.InputStreamReader;
import java.util.Hashtable;
import java.util.Random;
import javax.jms.Connection;
import javax.jms.ConnectionFactory;
import javax.jms.Destination;
import javax.jms.ExceptionListener;
import javax.jms.JMSException;
import javax.jms.Message;
import javax.jms.MessageConsumer;
import javax.jms.MessageListener;
import javax.jms.MessageProducer;
import javax.jms.Queue;
import javax.jms.Session;
import javax.jms.TextMessage;
import javax.naming.Context;
import javax.naming.InitialContext;
public class SimpleSenderReceiver implements MessageListener {
private static boolean runReceiver = true;
private Connection connection;
private Session sendSession;
private Session receiveSession;
private MessageProducer sender;
private MessageConsumer receiver;
private static Random randomGenerator = new Random();
public SimpleSenderReceiver() throws Exception {
// Configure JNDI environment
Hashtable<String, String> env = new Hashtable<String, String>();
env.put(Context.INITIAL_CONTEXT_FACTORY,
"org.apache.qpid.amqp_1_0.jms.jndi.PropertiesFileInitialContextFactory");
env.put(Context.PROVIDER_URL, "D:\\Java\\Azure\\workspace\\Analyzer\\src\\main\\resources\\servicebus.properties");
Context context = new InitialContext(env);
// Lookup ConnectionFactory and Queue
ConnectionFactory cf = (ConnectionFactory) context.lookup("SBCF");
System.out.println("cf:"+cf);
// Create Connection
connection = cf.createConnection();
System.out.println("connection :"+connection);
connection.setExceptionListener(new ExceptionListener() {
public void onException(JMSException arg0) {
System.err.println(arg0);
}
});
connection.start();
// Create sender-side Session and MessageProducer
sendSession = connection.createSession(false, Session.AUTO_ACKNOWLEDGE);
System.out.println("Session open");
Destination queue = (Destination) context.lookup("QUEUE");
System.out.println("queue:"+queue);
sender = sendSession.createProducer(queue);
Queue q=(Queue) queue;
System.out.println(sender.getDestination());
System.out.println("sender:"+sender);
if (runReceiver) {
System.out.println("Waitng for new message");
// Create receiver-side Session, MessageConsumer,and MessageListener
receiveSession = connection.createSession(false,
Session.CLIENT_ACKNOWLEDGE);
receiver = receiveSession.createConsumer(queue);
receiver.setMessageListener(this);
connection.start();
}
}
public static void main(String[] args) {
try {
if ((args.length > 0) && args[0].equalsIgnoreCase("sendonly")) {
runReceiver = false;
}
//System.setProperty("javax.net.debug","ssl");
System.setProperty("javax.net.ssl.trustStore","D:\\Java\\Azure\\workspace\\Analyzer\\src\\main\\resources\\SBKeystore.keystore");
System.setProperty("log4j.configuration","D:\\Java\\Azure\\workspace\\Analyzer\\src\\main\\resources\\log4j.properties");
SimpleSenderReceiver simpleSenderReceiver = new SimpleSenderReceiver();
System.out
.println("Press [enter] to send a message. Type 'exit' + [enter] to quit.");
BufferedReader commandLine = new java.io.BufferedReader(
new InputStreamReader(System.in));
while (true) {
String s = "Message";//commandLine.readLine();
if (s.equalsIgnoreCase("exit")) {
simpleSenderReceiver.close();
System.exit(0);
} else {
simpleSenderReceiver.sendMessage();
}
}
} catch (Exception e) {
e.printStackTrace();
}
}
private void sendMessage() throws JMSException {
TextMessage message = sendSession.createTextMessage();
message.setText("Test AMQP message from JMS");
long randomMessageID = randomGenerator.nextLong() >>> 1;
message.setJMSMessageID("ID:" + randomMessageID);
sender.send(message);
System.out.println("Sent message with JMSMessageID = "
+ message.getJMSMessageID());
}
public void close() throws JMSException {
connection.close();
}
public void onMessage(Message message) {
try {
System.out.println("Received message with JMSMessageID = "
+ message.getJMSMessageID());
message.acknowledge();
} catch (Exception e) {
e.printStackTrace();
}
}
}

Hi we had the same problems and thankfully MS updated their documentation to show how to do this correctly. :
http://msdn.microsoft.com/en-us/library/dn574799.aspx

The simplest answer to the question is as you should URL Encode the SASPolicyKey.
connectionfactory.SBCF = amqps://[SASPolicyName]:[SASPolicyKey]#[namespace].servicebus.windows.net
Where SASPolicyKey should be URL-Encoded.

AMQP 1.0 is supported with Service Bus 1.1 for windows server. Basically there are two differences between the cloud and on-prem usage of AMQP in ServiceBus:
1. Addressing: You will need to build an AMQP connection strings (and will need DNS in case you're looking for HA)
2. Authentication: You will need to use domain joined accounts as ACS is not there on-prem. You will also need to distribute your SB certificate to your clients.

Ok, I have sorted the first issue (Java Message Broker API not supporting SAS endpoint), by writing a wrapper which will seamlessly work with existing API. You can get the library from this GitHub repository. With this, I can develop/test my Java application on local service bus environment and host it on Azure / On-Premise Service Bus farm.
https://github.com/Dhana-Krishnasamy/ServiceBusForWindows-SASWrapper

The sender and receiver Queues you will have to configure differently. Here is an example of my working configuration (servicebus.properties):
connectionfactory.SBCF = amqps://$PolicyName:$UrlEncodedKey#$Your-EventHub-NamespaceName.servicebus.windows.net
queue.EventHubSender=$YourEventHubName
queue.EventHubReceiver=$YourEventHubName/ConsumerGroups/$YourConsumerGroupName/Partitions/1
Replace appropriately your own '$' items in there.
The Shared Policy Key has to be URL encoded.
Make sure that your sender will reference the 'EventHubSender' defined in this config and the receiver will reference the 'EventHubReciever'.

Grab the Azure Java SDK from http://www.windowsazure.com/en-us/develop/java/ and then follow this guide: http://www.windowsazure.com/en-us/develop/java/how-to-guides/service-bus-queues/

What are the possible causes that let ActiveMQ application silently quit?

I have made a simple ActiveMQ application.
It listens to a queue. If a message comes, print out the dataId
Here is the code:
import javax.jms.Connection;
import javax.jms.ConnectionFactory;
import javax.jms.Destination;
import javax.jms.ExceptionListener;
import javax.jms.JMSException;
import javax.jms.MapMessage;
import javax.jms.Message;
import javax.jms.MessageConsumer;
import javax.jms.MessageListener;
import javax.jms.Session;
import org.apache.activemq.ActiveMQConnectionFactory;
public class MQ implements MessageListener {
private Connection connection = null;
private Session session = null;
private Destination destination = null;
private void errorOnConnection(JMSException e) {
System.out.println("MQ is having problems. Exception::"+ e);
}
private void init() throws JMSException {
String BROKER_URL = "failover:(tcp://myQueue001:61616,tcp://myQueue002:61616)?randomize=false";
ConnectionFactory connectionFactory = new ActiveMQConnectionFactory(BROKER_URL);
connection = connectionFactory.createConnection("user", "password");
connection.setExceptionListener(
new ExceptionListener() {
#Override public void onException(JMSException e) {
errorOnConnection(e);
}
});
session = connection.createSession(false, Session.AUTO_ACKNOWLEDGE);
destination = session.createQueue("myQueue");
MessageConsumer consumer = session.createConsumer(destination);
consumer.setMessageListener(this);
}
public boolean start() {
try {
if(connection==null )
init();
connection.start();
} catch (Exception e) {
System.out.println("MQListener cannot be started, exception: " + e);
}
return true;
}
#Override
public void onMessage(Message msg) {
try {
if(msg instanceof MapMessage){
MapMessage m = (MapMessage)msg;
int dataId = m.getIntProperty("dataId");
System.out.println(dataId);
}
} catch (JMSException e) {
System.out.println("Got an exception: " + e);
}
}
public static void main(String[] args) {
MQ mq = new MQ();
mq.start();
}
}
It works fine and does what it is meant to accomplish.
However, the problem is that it can run only for several days. After several days, it just quits silently without any exceptions or error.
The queue I am listening to is from 3rd party. From a guy there, the queue sometimes will be closed or restarted or interrupted.
But I think even if that happen, the default ActiveMQ settings will handle it by consistently reconnect to it, right? (according to http://activemq.apache.org/cms/configuring.html)
So any other possible causes which lead my code to quitting silently?

Depends on bit on your version. Since you are not doing anything yourself to keep the application running but instead depending on the ActiveMQ code to keep at least one non-deamon thread running. In some ActiveMQ versions the client wasn't always doing this so your application could quite while a failover was occurring. Best bet is to switch to v5.8.0 which I believe had some fixes for this.
You could add some polling code in main to read something from console or what not to ensure that the client stays up until you are sure you want it to go down.