Thanks for this opportunity to ask questions here.
Summary
I have a spring boot application which runs on Tomcat. I'm subscribing ActiveMQ topic with StompClient. After connected i get messages via StompHandler's handleFrame method.
In that method i create MessageUtils which implements Runnable interface. In MessageUtils run method, creating new thread with ExecutorService and do the task generateMessage which send messages to kafka topic.
Problem
When I shutdown the tomcat, threads are still alive. In catalina.out;
A web application appears to have started a thread named [foo] but has failed to stop it. This is very likely to create a memory leak
So the tomcat cannot shutdown properly.
Code Samples
#Service
StompService.class
#EventListener(ApplicationReadyEvent.class)
public void start() {
logI("Service run Client Methods");
List<String> topics = Arrays.asList(topicListString.split(","));
for (String topic : topics) {
StompClient client = new StompClient(topic, username, password, url, topic, bootstrapAddress);
try {
client.run();
runMap.put(topic, client);
boolean connected = client.getSession().isConnected();
logI("Topic: " + topic + " is connected: " + connected);
} catch (InterruptedException e) {
logE("InterruptedException during start of stomp client: ", e);
} catch (TimeoutException e) {
logE("TimeoutException during start of stomp client: ", e);
} catch (ExecutionException e) {
logE("ExecutionException during start of stomp client: ", e);
} catch (Exception e) {
logE("Unexpected exception during start of stomp client: ", e);
}
}
}
StompClient.class
public void run() throws ExecutionException, TimeoutException, InterruptedException {
WebSocketClient client = new StandardWebSocketClient();
WebSocketStompClient stompClient = new WebSocketStompClient(client);
ThreadPoolTaskScheduler taskScheduler = new ThreadPoolTaskScheduler();
taskScheduler.afterPropertiesSet();
StompHeaders connectHeaders = new StompHeaders();
connectHeaders.add("login", this.userName);
connectHeaders.add("passcode", this.password);
stompClient.getDefaultHeartbeat();
stompClient.setTaskScheduler(taskScheduler);
stompClient.setMessageConverter(new StringMessageConverter());
stompClient.setAutoStartup(true);
StompSessionHandler sessionHandler = new StompHandler(this.topic, this.bootstrapAddress);
StompSession stompSession = null;
try {
stompSession = stompClient.connect(url, new WebSocketHttpHeaders(), connectHeaders, sessionHandler)
.get(5, TimeUnit.SECONDS);
} catch (Exception e) {
logE("Cannot connect with stomp client." , e);
}
this.setSession(stompSession);
}
StompHandler.class which extends StompSessionHandlerAdapter
#Override
public void handleFrame(StompHeaders headers, Object payload) {
String msg = (String) payload;
MessageUtils message = new MessageUtils();
message.setHeaders(headers);
message.setTopic(topic);
message.setMsg(msg);
message.setBootstrapAddress(bootstrapAddress);
message.run();
}
MessageUtils.class
#Override
public void run() {
ExecutorService executorService = Executors.newSingleThreadExecutor();
executorService.execute(new Runnable() {
public void run() {
generateMessage(getMsg().toString());
}
});
executorService.shutdown();
try {
executorService.awaitTermination(200, TimeUnit.MILLISECONDS);
} catch (InterruptedException e) {
logI("InterruptedException during await termination", e);
}
}
Messages published and response received are on two different subjects. Right now I have following set of functionality in my java class. Class is implementing TibrvMsgCallback interface.
How can i make sure that whatever the message is published i am receiving exactly its response?
public class TibcoRVUtility implements TibrvMsgCallback {
public void onMsg(TibrvListener listener, TibrvMsg msg) {
try {
_log.info("Request and Response found");
msgReceived = true;
} catch (final TibrvException ex) {
_log.error("Exception#" + this.getClass().getName() + ".onMsg", ex);
}
}
private void sendMessage(String messageString, final String soType,
final String responseSubject) {
try {
Tibrv.open(Tibrv.IMPL_NATIVE);
TibrvTransport transport = new TibrvRvdTransport(tibcoSetting.getService(), tibcoSetting.getNetwork(),
tibcoSetting.getDaemon());
String inboxName = transport.createInbox();
TibrvMsg msg = new TibrvMsg();
msg.setSendSubject("PUBLISH_SUBJECT");
msg.add("DATA", "DUMMY_MESSAGE");
TibrvListener listener = new TibrvListener(Tibrv.defaultQueue(), this, transport, responseSubject, null);
transport.send(msg);
_log.info("msg" + msg.toString());
_log.info("message successfully sent.");
while (!msgReceived) {
try {
Tibrv.defaultQueue().dispatch();
} catch (InterruptedException ex) {
_log.error("Exception#" + this.getClass().getName() + ".sendMessage", ex);
break;
} catch (TibrvException ex) {
_log.error("Exception#" + this.getClass().getName() + ".sendMessage", ex);
break;
}
}
listener.destroy();
transport.destroy();
} catch (TibrvException e) {
_log.error("Exception#" + this.getClass().getName() + ".sendMessage", e);
}
}
}
When you send a message, add another field
var correlation_id = Guid.NewGuid().ToString();
msg.add("CORRELATION_ID", correlation_id);
and then stash that correlation ID somewhere, in a hash set perhaps.
Have the publisher read the CORRELATION_ID off of the request and add it to the response.
When you receive a message, only process it if it has the ID that you are expecting in the CORRELATION_ID field.
This is the context:
A Java application subscribes to the Topic "ActiveMQ.Advisory.Connection" from an ActiveMQ 5.9.1 via MQTT (Paho 0.4.0):
public class SupervisorMqttClient implements MqttCallback {
private MqttClient client = null;
private MemoryPersistence persistence = null;
private MqttConnectOptions connOpts = null;
private final int STATUS_OK = 0;
private final int STATUS_ERROR = 1;
private String mqttServer = null;
private String clientId = null;
private int status = STATUS_OK;
public SupervisorMqttClient() {
try {
this.init();
} catch (MqttException e) {
Logger.error(e.getLocalizedMessage());
Logger.debug(e);
}
}
private void init() throws MqttException {
Properties props = PropertiesManager.getInstance("supervisor");
mqttServer = props.getProperty("supervisor.mqtt.server");
String supervisorID = props.getProperty("supervisor.mqtt.client.number");
clientId = Supervisor.APP_NAME+"-"+supervisorID;
connOpts = new MqttConnectOptions();
connOpts.setKeepAliveInterval(30);
connOpts.setCleanSession(true); // important non-durable
persistence = new MemoryPersistence();
client = new MqttClient(mqttServer, clientId, persistence);
connectAndSubscribe();
}
private void connectAndSubscribe() throws MqttSecurityException, MqttException {
try {
client.connect(connOpts);
client.setCallback(this);
client.subscribe("ActiveMQ/Advisory/Connection");
} catch (MqttSecurityException e) {
Logger.error(e.getLocalizedMessage());
Logger.debug(e);
} catch (MqttException e) {
Logger.error(e.getLocalizedMessage());
Logger.debug(e);
processError(e);
}
}
public void publish(String orderType, JSONObject jsonExtraData) {
if (status == STATUS_ERROR) {
connectAndSubscribe();
}
if (status == STATUS_OK) {
// some code here
}
}
#Override
public void connectionLost(Throwable err) {
Logger.info("Connection lost");
}
#Override
public void deliveryComplete(IMqttDeliveryToken arg0) {
Logger.info("deliveryComplete");
}
#Override
public void messageArrived(String topic, MqttMessage msg) throws Exception {
System.out.println("MQTT Mesage Arrived[" + topic + "] Msg[" + msg.toString() + "]");
}
private void processError(MqttException e) {
status = STATUS_ERROR;
try {
if (client.isConnected()) {
Logger.error("disconnecting");
client.disconnect();
}
} catch (MqttException ex) {
Logger.error(ex.getLocalizedMessage());
Logger.debug(ex);
}
}
}
The connection with ActiveMQ is established fine. This topic offers information about the connections (open/close) in the ActiveMQ, but my problem is that messages I catch are empty:
MQTT Mesage Arrived[ActiveMQ/Advisory/Connection] Msg[]
Is there any way to catch them using MQTT? or I should use JMS for that?
Thanks,
Jon Ander.
The question would be what do you want the MQTT client to receive on the Advisory topic as the message body. The advisories generally include much of the information as message properties however those cannot be mapped to MQTT as MQTT messages don't have properties. The body of the Connection advisory is a copy of the ConnectionInfo object that was used to create the connection. On the MQTT side there is not much you could do with that as all you would receive would be the serialized bytes of that object which you wouldn't be able to do anything with.
I having a problem with the RabbitMQ Work Queue implementation. im current running it in Tomcat, and i have the following class constantly listerning to new task in the queue. But after a day or two, sudden it behaving strangely, where by the object DeliveryOK return by channel.queueDeclare(taskQueueName, isDurable, false, false, null); is always zero. (i print out this in the log below mentioning "Current poolSize").
But in Rabbit admin (./rabbitmqadmin list queues or the RabbitMq Admin portal) it always return a number greater than zero (say 1267 messages in the queue). And it will not reduce to zero until i restart the tomcat, the class below only able to detect that there are actually some messages in the queue.
Initially i thought that this class was terminated somehow, but it is able to consume those messages that newly arrive. It will not consume those 1267 messages that is left hanging inside the queue. For example messages 1267 in the queue, will not be consume until i restart tomcat.
From the code below, is it because buggy implementation or is there a better way to implement a queue consumer specifically for RabbitMQ? i have read a related stack post(Producer/Consumer threads using a Queue), but im not sure if it helps.
Also, is it true that this consumer implementation below will not survive a RunTimeException?
MqConsumer Class:
#Service
public class MqConsumer implements Runnable{
private static final Logger logger = LoggerFactory.getLogger(MqConsumer.class);
private final int MAX_ALERT_THRESHOLD = 10000;
#Autowired
private AsynchSystemConnections asynchSystemConnections;
public MqConsumer(){
}
#PostConstruct
private void init() {
(new Thread(new MqConsumer(asynchSystemConnections))).start();
}
public MqConsumer(AsynchSystemConnections asynchSystemConnections){
this.asynchSystemConnections = asynchSystemConnections;
}
#Override
public void run() {
logger.info("Execute Consumer instance...");
while (true) { // infinite loop until it die due server restart
boolean toSleep = consume(asynchSystemConnections);
if (toSleep){
logger.error("Sleeping for 1 second...");
try {
Thread.sleep(1000);
} catch (InterruptedException e) {
logger.error("", e);
}
}
}
}
private boolean consume(AsynchSystemConnections asynchSystemConnections) {
com.rabbitmq.client.Connection mqConnection = null;
Channel mqChannel = null;
DatasiftMq dMq = null;
try {
ConnectionFactory factory = new ConnectionFactory();
factory.setHost(asynchSystemConnections.getMqServerHost());
mqConnection = factory.newConnection();
mqChannel = mqConnection.createChannel();
//consumePushInteractionJob method will forward to AsynchTwService.consume(connection, channel, AsynchTwService.PUSH_INTERACTION_QUEUE )
dMq = asynchSystemConnections.getAsynchService().consumePushInteractionJob(mqConnection, mqChannel);
int poolSize = asynchSystemConnections.getAsynchService().getPushInteractionQueueSize();
logger.info("Current poolSize: " + poolSize);
} catch(NullPointerException e) {
logger.error("", e);
if (dMq != null) {
try {
logger.error("Removing JSON with" + dMq.getLogHeader(dMq));
asynchSystemConnections.getAsynchService().ack(mqChannel, dMq.getDelivery());
logger.error("Removed JSON with" + dMq.getLogHeader(dMq));
} catch (IOException e1) {
logger.error("Remove JSON Failed: ", e);
}
}
return true;
} catch (IOException e) {
logger.error("Unable to create new MQ Connection from factory.", e);
return true;
} catch (InterruptedException e) {
logger.error("", e);
return true;
} catch (ClassNotFoundException e) {
logger.error("", e);
return true;
} catch (Exception e) {
logger.error("Big problem, better solve this fast!!", e);
asynchSystemConnections.getNotificationService().notifySystemException(null, e);
return true;
} finally {
try {
asynchSystemConnections.getAsynchService().ack(mqChannel, dMq.getDelivery());
asynchSystemConnections.getAsynchService().disconnect(mqConnection, mqChannel);
} catch (IOException e) {
logger.error("", e);
}
}
return false;
}
AsynchTwService Class:
#Service("asynchTwService")
public class AsynchTwService implements AsynchService {
static final String FAVOURITE_COUNT_QUEUE = "favourite_count_queue";
static final String FRIENDS_FOLLOWERS_QUEUE = "friends_followers_queue";
static final String DIRECT_MESSAGE_RECEIVE_QUEUE = "direct_message_receive_queue";
static final String PUSH_INTERACTION_QUEUE = "push_interaction_queue";
private static String mqServerHost;
private static final Logger logger = LoggerFactory.getLogger(AsynchTwService.class);
private static final boolean isDurable = true;
private boolean autoAck = false;
private ConcurrentHashMap<String, Integer> currentQueueSize = new ConcurrentHashMap<String, Integer>();
#Override
public Connection getConnection() throws IOException{
ConnectionFactory factory = new ConnectionFactory();
factory.setHost(mqServerHost);
return factory.newConnection();
}
#Override
public void produce(Connection connection, Channel channel, Object object, String taskQueueName) throws IOException {
sendToQueue(connection, channel, object, taskQueueName);
}
#Override
public QueueItem consume(Connection connection, Channel channel, String taskQueueName) throws IOException, InterruptedException, ClassNotFoundException{
Serializer serializer = new Serializer();
try {
Delivery delivery = listenFromQueue(connection, channel, taskQueueName);
Object messageObj = serializer.toObject(delivery.getBody());
QueueItem queueItem = (QueueItem)messageObj;
queueItem.setDelivery(delivery);
return queueItem;
} catch (InterruptedException e) {
throw e;
} catch (ClassNotFoundException e) {
logger.error("Unable to serialize the message to QueueItem object", e);
throw e;
}
}
#Override
public int getQueueSize(String taskQueueName){
return this.currentQueueSize.get(taskQueueName);
}
private Delivery listenFromQueue(Connection connection, Channel channel, String taskQueueName) throws IOException, InterruptedException, ClassNotFoundException{
try {
DeclareOk ok = channel.queueDeclare(taskQueueName, isDurable, false, false, null);
currentQueueSize.put(taskQueueName, ok.getMessageCount());
logger.info("Queue ("+ taskQueueName + ") has items: " +ok.getMessageCount());
} catch (IOException e) {
// TODO Auto-generated catch block
}
logger.info(" [*] Consuming "+taskQueueName+" message...");
QueueingConsumer consumer = new QueueingConsumer(channel);
try {
channel.basicConsume(taskQueueName, autoAck, consumer);
} catch (IOException e) {
logger.error("", e);
}
try {
QueueingConsumer.Delivery delivery = consumer.nextDelivery();
return delivery;
} catch (ShutdownSignalException e) {
logger.error("Unable to retrieve message from Queue", e);
throw e;
} catch (ConsumerCancelledException e) {
logger.error("Unable to retrieve message from Queue", e);
throw e;
} catch (InterruptedException e) {
logger.error("Unable to retrieve message from Queue", e);
throw e;
}
}
private void sendToQueue(Connection connection, Channel channel, Object object, String taskQueueName) throws IOException{
//Initialization, create Message Queue broker connection
try{
channel.queueDeclare(taskQueueName, isDurable, false, false, null);
}catch(IOException e) {
logger.error(e.getMessage());
logger.error("Error create Message Queue connection for queue name:" + taskQueueName, e);
throw e;
}
//send message to broker
try {
long start = System.currentTimeMillis();
Serializer serializer = new Serializer();
logger.info("Sending Twitter QueueItem to Message Queue...");
channel.basicPublish("", taskQueueName, MessageProperties.PERSISTENT_TEXT_PLAIN,
serializer.toBytes(object));
logger.info("Queue successfully sent, process took: " + (System.currentTimeMillis()-start)+ "ms");
} catch (IOException e) {
logger.error("Error while sending object to queue : " + taskQueueName, e);
throw e;
}
}
public static String getMqServerHost() {
return mqServerHost;
}
public static void setMqServerHost(String mqServerHost) {
AsynchTwService.mqServerHost = mqServerHost;
}
#Override
public void disconnect(Connection connection, Channel channel) throws IOException{
try {
if (channel != null){
if (channel.isOpen()){
channel.close();
}
}
if (connection != null){
if (connection.isOpen()){
connection.close();
}
}
logger.debug("MQ Channel Disconnected");
} catch (IOException e) {
throw e;
}
}
#Override
public void ack(Channel channel, QueueingConsumer.Delivery delivery) throws IOException {
// this is made as another method call is to avoid Ack too fast un intentionally
try {
channel.basicAck(delivery.getEnvelope().getDeliveryTag(), false);
logger.info("[x] acked" );
} catch (IOException e) {
logger.error("Unable Acknowledge Queue Message", e);
throw e;
}
}
#Override
public DatasiftMq consumeDatasiftInteraction(Connection connection, Channel channel,
String taskQueueName) throws IOException, InterruptedException, ClassNotFoundException {
Serializer serializer = new Serializer();
try {
Delivery delivery = listenFromQueue(connection, channel, taskQueueName);
Object messageObj = serializer.toObject(delivery.getBody());
DatasiftMq dto = (DatasiftMq)messageObj;
dto.setDelivery(delivery);
return dto;
} catch (InterruptedException e) {
throw e;
} catch (ClassNotFoundException e) {
logger.error("Unable to serialize the message to DatasiftDTO object", e);
throw e;
}
}
#Override
public void reQueue(Channel channel, Delivery delivery) throws IOException {
try {
channel.basicNack(delivery.getEnvelope().getDeliveryTag(), false, true);
logger.info("[x] Nacked" );
} catch (IOException e) {
logger.error("Unable Acknowledge Queue Message", e);
throw e;
}
}
}
Seems like you are missing some basics here.
Taken from here and some code of mine.
Setting up the connection outside of the consumer thread:
//executed once
ConnectionFactory factory = new ConnectionFactory();
factory.setHost("someHost");
factory.setUsername("user");
factory.setPassword("pass");
Connection connection = factory.newConnection();
What you have to do inside your thread:
//Consumer - executed in a Thread
QueueingConsumer consumer = new QueueingConsumer(connection.createChannel());
boolean autoAck = false;
channel.basicConsume("hello", autoAck, consumer);
while (!Thread.current().isInterrupted())) {
QueueingConsumer.Delivery delivery = consumer.nextDelivery();
//...
channel.basicAck(delivery.getEnvelope().getDeliveryTag(), false);
}
In general I do still recommand you check out the spring-amqp library it integrates perfectly.
Hey how can i display every response from the xmpp server?
I try to send messages but a lot of them get lost on the way so i want to check the response of the server. I am using smack 3.3.1 sending to the facebook xmpp port.
DeliveryReceiptManager and/or MessageEventManager wont show anything so i would like to see everything the server is responding!
ConnectionConfiguration config = new ConnectionConfiguration("chat.facebook.com",5222);
config.setSASLAuthenticationEnabled(true);
XMPPConnection connection = new XMPPConnection(config);
try {
//ESTA LINEA HACE QUE NO DE TIMEOUT
SmackConfiguration.setPacketReplyTimeout(15000);
XMPPConnection.DEBUG_ENABLED = true;
SASLAuthentication.registerSASLMechanism("X-FACEBOOK-PLATFORM", SASLXFacebookPlatformMechanism.class);
SASLAuthentication.supportSASLMechanism("X-FACEBOOK-PLATFORM", 0);
connection.connect();
String apiKey = "1234567";
String accessToken = "";
connection.login(apiKey, accessToken);
}catch (XMPPException e){
e.printStackTrace();
}
try {
DeliveryReceiptManager deliveryReceiptManager = DeliveryReceiptManager.getInstanceFor(connection);
deliveryReceiptManager.addReceiptReceivedListener(new ReceiptReceivedListener() {
#Override
public void onReceiptReceived(String s, String s2, String s3) {
System.out.println("REVEIVED RESPONCE");
System.out.println(s);
System.out.println(s2);
System.out.println(s3);
}
});
Chat chat = connection.getChatManager().createChat("1234567890#chat.facebook.com", new MessageListener() {
#Override
public void processMessage(Chat chat, Message message) {
if(message.getType() == Message.Type.chat)
System.out.println(chat.getParticipant() + " says: " + message.getBody());
}
});
Message msg = new Message();
msg.setSubject("Invite");
msg.setBody("blablabla");
DeliveryReceiptManager.addDeliveryReceiptRequest(msg);
//MessageEventManager.addNotificationsRequests(msg, true, true, true, true);
chat.sendMessage(msg);
try {
Thread.sleep(10000);
} catch (InterruptedException e) {
e.printStackTrace();
}
}catch (XMPPException e){
e.printStackTrace();
}
try {
Thread.sleep(10000);
}catch (InterruptedException e) {
e.printStackTrace();
}
Connection.DEBUG_ENABLED = true;