We Keep Coding

Increase Spring Boot JMS message consumption rate

I'm trying to identify how to increase the rate of message consumption of a JMS Springboot application. I tried testing the rate of message consumption of the app and it took 1.5 hrs to consume and process 2000 waiting/pending messages in QUEUE.
In other words, problem is, it took 1.5hrs for the springboot app to empty the QUEUE it's consuming from.
public class MyMessageListener implements MessageListener {
#Autowired
private MyMessageService messageService;
#Override
public void onMessage(Message message) {
String messageContent = null;
try {
if (message instanceof BytesMessage) {
BytesMessage bytesMessage = (BytesMessage) message;
long length = bytesMessage.getBodyLength();
byte[] content = new byte[(int) length];
bytesMessage.readBytes(content);
messageContent = new String(content, StandardCharsets.UTF_8);
} else if (message instanceof TextMessage) {
TextMessage textMessage = (TextMessage) message;
messageContent = textMessage.getText();
}
if (messageContent != null) {
FileIOHelper.writeInboundXmlToFile(messageContent); //write message to file
String accountNumber = XmlUtil.extractAccountNumber(messageContent);
final String xmlMessageTransformed = messageService.transformXmlMessageToOldSchema(messageContent);
if (!xmlMessageTransformed.isEmpty()) {
FileIOHelper.writeTransformedXmlToFile(accountNumber, xmlMessageTransformed); //write message to file
Map<String, String> outboundHeaderProperties = messageService.createJMSHeaderProperties(message);
messageService.publishMessageToOutboundTopic(xmlMessageTransformed, outboundHeaderProperties);
} else {
FileIOHelper.writeUnprocessedXmlToFile(messageContent); //write message to file
log.error(
String.format("Failed transformation of message account# ", accountNumber));
}
message.acknowledge(); // acknowledge ALL inbound messages from inbound QUEUE
}
} catch (Exception e) {
log.error(e.getMessage());
}
}
}
As you can see, part of the processing of message involves writing a copy of inbound and outbound messages received to a file. I suspect this is what's causing the slow consumption/processing rate of messages from QUEUE.
In my JMS Configuration class, I have the following :
#Bean
public DefaultMessageListenerContainer listenerContainer(MessageListenerAdapter messageListener,
#Qualifier("sourceConnection") ConnectionFactory listenerConnectionFactory) {
DefaultMessageListenerContainer container = new DefaultMessageListenerContainer();
container.setConnectionFactory(listenerConnectionFactory);
container.setDestinationName(jmsSourceQueue);
container.setMessageListener(messageListener);
container.setSessionTransacted(true);
container.setSessionAcknowledgeMode(Session.CLIENT_ACKNOWLEDGE);
container.setRecoveryInterval(30000); //reconnect every 30 seconds if disconnected.
// container.setConcurrentConsumers(1); // do I need to add this line?
// container.setMaxConcurrentConsumers(5); //Or, add this line?
return container;
}
I searched SO and learned about setConcurrentConsumers() and setMaxConcurrentConsumers() I'm not sure if that's how I can solve the slow message consumption rate.
The requirement for our JMS application is to be able to consume messages in just a few minutes. In my example above, it took 1.5 hrs to consume all 2000 messages.
Can you suggest a way or approach to solve this without removing the write-to-file step?
Thank you!

Google PubSub resent messages aren't being processed

I've used the subscriber example from the google documentation for Google PubSub
the only modification I've made is commenting out the acknowledgement of the messages.
The subscriber doesn't add messages to the queue anymore while messages should be resent according to the interval set in the google cloud console.
Why is this happening or am I missing something?
public class SubscriberExample {
use the default project id
private static final String PROJECT_ID = ServiceOptions.getDefaultProjectId();
private static final BlockingQueue<PubsubMessage> messages = new LinkedBlockingDeque<>();
static class MessageReceiverExample implements MessageReceiver {
#Override
public void receiveMessage(PubsubMessage message, AckReplyConsumer consumer) {
messages.offer(message);
//consumer.ack();
}
}
/** Receive messages over a subscription. */
public static void main(String[] args) throws Exception {
// set subscriber id, eg. my-sub
String subscriptionId = args[0];
ProjectSubscriptionName subscriptionName = ProjectSubscriptionName.of(
PROJECT_ID, subscriptionId);
Subscriber subscriber = null;
try {
// create a subscriber bound to the asynchronous message receiver
subscriber = Subscriber.newBuilder(subscriptionName, new MessageReceiverExample()).build();
subscriber.startAsync().awaitRunning();
// Continue to listen to messages
while (true) {
PubsubMessage message = messages.take();
System.out.println("Message Id: " + message.getMessageId());
System.out.println("Data: " + message.getData().toStringUtf8());
}
} finally {
if (subscriber != null) {
subscriber.stopAsync();
}
}
}
}

When you do not acknowledge a messages, the Java client library calls modifyAckDeadline on the message until maxAckExtensionPeriod passes. By default, this value is one hour. Therefore, if you don't ack/nack the message or change this value, it is likely the message will not be redelivered for an hour. If you want to change the max ack extension period, set it on the builder:
subscriber = Subscriber.newBuilder(subscriptionName, new MessageReceiverExample())
.setMaxAckExtensionPeriod(Duration.ofSeconds(60))
.build();
It is also worth noting that when you don't ack or nack messages, then flow control may prevent the delivery of more messages. By default, the Java client library allows up to 1000 messages to be outstanding, i.e., waiting for ack or nack or for the max ack extension period to pass.

How to receive message from wildfly jms queue using consumer

I encountered a knotty problem when receiving message from WildFly JMS queue. My code is below:
Session produceSession = connectionFactory.createConnection().createSession(false, Session
.CLIENT_ACKNOWLEDGE);
Session consumerSession = connectionFactory.createConnection().createSession(false, Session
.CLIENT_ACKNOWLEDGE);
ApsSchedule apsSchedule = new ApsSchedule();
boolean success;
MessageProducer messageProducer = produceSession.createProducer(outQueueMaxusOrder);
success = apsSchedule.sendD90Order(produceSession,messageProducer, d90OrderAps);
if (!success) {
logger.error("Can't send APS schedule msg ");
} else {
MessageConsumer consumer = consumerSession.createConsumer(inQueueDeliveryDate);
data = apsSchedule.receiveD90Result(consumerSession,consumer);
}
then getting into the receiveD90Result():
public DeliveryData receiveD90Result(Session session, MessageConsumer consumer) {
DeliveryData data = null;
try {
Message message = consumer.receive(10000);
if (message == null) {
return null;
}
TextMessage msg = (TextMessage) message;
String text = msg.getText();
logger.debug("Receive APS d90 result: {}", text);
ObjectMapper mapper = new ObjectMapper();
data = mapper.readValue(text, DeliveryData.class);
} catch (JMSException je) {
logger.error("Can't receive APS d90 order result: {}", je.getMessage());
} catch (Exception e) {
e.printStackTrace();
} finally {
try {
consumer.close();
} catch (JMSException e) {
e.printStackTrace();
}
}
return data;
}
But when implementing the consumer.receive(10000), the project can't get a message from queue. If I use asynchronous way of MDB to listen the queue, I can get the message from queue. How to resolve it?

There are multiple modes you can choose to get a message from the queue. Message Queues are by default asynchronous in usage. There are however cases when you want to read it synchronously , for example sending a message with account number and using another queue to read the response and match it with a message id or a message correlation id. When you do a receive , the program is waiting for a message to arrive within that polling interval specified in receive.
The code snippet you have , as i see it uses the psuedo synchronous approach. If you have to use it as an MDB , you will have to implement message driven bean (EJB Resource) or message listener.
The way that MDB/Message Listener works is more event based , instead of a poll with a timeout (like the receive) , you implement a callback called onMessage() that is invoked every time there is a message. Instead of a synchronous call , this becomes asynchronous. Your application may require some changes both in terms of design.

I don't see where you're calling javax.jms.Connection.start(). In fact, it doesn't look like you even have a reference to the javax.jms.Connection instance used for your javax.jms.MessageConsumer. If you don't have a reference to the javax.jms.Connection then you can't invoke start() and you can't invoke close() when you're done so you'll be leaking connections.
Furthermore, connections are "heavy" objects and are meant to be re-used. You should create a single connection for both the producer and consumer. Also, if your application is not going to use the javax.jms.Session from multiple threads then you don't need multiple sessions either.

Unable to receive message sent to acivemq by C# in Java

What I want to do is to send messages via Apache Activemq between C# app and Java app.
C#:
using (IConnection connection = factory.CreateConnection())
using (ISession session = connection.CreateSession())
{
IDestination destination = SessionUtil.GetDestination(session, "queue://ISI");
// Create a consumer and producer
using (IMessageProducer producer = session.CreateProducer(destination))
{
// Start the connection so that messages will be processed.
connection.Start();
ITextMessage request = session.CreateTextMessage(JsonConvert.SerializeObject(obj));
/*request.NMSCorrelationID = "abc";
request.Properties["NMSXGroupID"] = "cheese";
request.Properties["myHeader"] = "Cheddar";*/
producer.Send(request);
return request;
}
}
Java:
ActiveMQConnectionFactory connectionFactory = new ActiveMQConnectionFactory(isiProperties.getMqUrl());
connection = connectionFactory.createConnection();
connection.start();
Session session = connection.createSession(false, Session.AUTO_ACKNOWLEDGE);
Destination destination = session.createQueue("ISI");
MessageConsumer consumer = session.createConsumer(destination);
Message message = consumer.receive();
if(message instanceof TextMessage) {
try {
String text = ((TextMessage) message).getText();
ObjectMapper mapper = new ObjectMapper();
StatusChangeMessage obj = mapper.readValue(text, StatusChangeMessage.class);
if (obj instanceof StatusChangeMessage) {
StatusChangeMessage received = (StatusChangeMessage) obj;
Order order = orderRepository.findOne(received.getOrderId());
order.setStatus(received.getStatus());
orderRepository.saveAndFlush(order);
}
} catch(JMSException e) {
} catch(IOException e) {
}
}
The C# app correctly sends messages (it is visible in activemq admin interface) but there are no active subscribers (Java app should do that). Do you see anything wrong here?
Basically, breakpoint on if(message instanceof TextMessage) { does not get executed.

I have finally found a solutioin. It was two-steps problem. Firstly, Windows firewall influenced activemq. Secondly, client library probably didn't fully match to the server. Problem finally gone after downgrading server to 5.8.0.

Cannot create a subscriber on the durable subscription

I'm working on a small project for a Systems Integration subject, and I'm using JMS (JBOSS). We have to use durable topics, and that part is quite easy. The thing is, let's say I use the following code:
TopicConnectionFactory topicConnectionFactory = InitialContext.doLookup("jms/RemoteConnectionFactory");
try(JMSContext jmsContext = topicConnectionFactory.createContext(<username>,<password>)) {
Topic topic = InitialContext.doLookup(<topic>);
JMSConsumer jmsConsumer = jmsContext.createDurableConsumer(topic, <client-id>);
Message message = jmsConsumer.receive();
if(message != null) {
result = message.getBody(ArrayList.class);
}
}
This try-with-resources is useful, since it destroys the connection when the block ends. But let's say I interrupt the program while the JMSConsumer waits for the message. When I restart the program, it will throw:
javax.jms.IllegalStateRuntimeException: Cannot create a subscriber on the durable subscription since it already has subscriber(s)
Is there a way to close the connection/unsubscribe/something when the program is interrupted?

If you need to do some cleanup but not swallow the exception, you can catch the exception, do some cleanup, then rethrow the original exception:
try(JMSContext jmsContext = topicConnectionFactory.createContext(<username>,<password>)) {
// ...
} catch (InterruptedException e) {
// Do some cleanup.
throw e;
}
(I'm assuming that it's an InterruptedException, because you said "say I interrupt the program" - but maybe it is some other type: same idea applies)

Basically, I used the following code:
TopicConnectionFactory topicConnectionFactory = InitialContext.doLookup("jms/RemoteConnectionFactory");
try(JMSContext jmsContext = topicConnectionFactory.createContext(<username>,<password>)) {
Topic topic = InitialContext.doLookup(<topic>);
JMSConsumer jmsConsumer = jmsContext.createDurableConsumer(topic, <client-id>);
Runtime.getRuntime().addShutdownHook(new Thread() {
public void run() {
jmsConsumer.close();
this.interrupt();
}
});
Message message = jmsConsumer.receive();
if(message != null) {
result = message.getBody(ArrayList.class);
}
}
I was trying to close the connection using jmsContext.stop(), I think. Anyhow, it wasn't working, now it is. Yay me.