What would be the best way to consume "topic-ed" message batches from RabbitMQ in parallel and in order.
We have a server that processes data for many customers. Every time a customer's data is processed, a bunch of messages is sent to RMQ. On the other side we have a process that consumes the data and stores it in a database.
The consumption process is slow and we want to parallelize it and make it scalable. The problem is that data for a single customer cannot be processed by two consumers at the same time.
The producer runs every now and then and can add messages to the queue even for a customer that already has messages in the queue.
One of the suggestions was to create a new DB table that will indicated for each customer if it's data is being processed. A consumer will only ask for messages for customers that are not being processed by other consumers and will register itself in the DB for that customer.
I'm reluctant to use that solution because it requires connecting to a database and it holds runtime state in a database.
I was hoping to find a solution that could be handled within the scope of our consumer/producer code and RMQ.
A suggestion was made to have messages written to RMQ under customer "topics" and have the consumers read a single "topic". A message will be added to a separate queue (or "topic") for each batch or customer messages. A consumer will consumer a "customers" message and use it's data to select a "topic" from the main queue.
The problem is what happens when the producer wants to add new data to the main queue for a customer that already has data in the main queue that is currently being processed.
How can we sync the consumption an production over RMQ?
I think you could have a look at rabbitmq tutorials available.
http://www.rabbitmq.com/tutorials/tutorial-five-java.html
Sample code:
import com.rabbitmq.client.*;
import java.io.IOException;
public class ReceiveLogsTopic {
private static final String EXCHANGE_NAME = "topic_logs";
public static void main(String[] argv) throws Exception {
ConnectionFactory factory = new ConnectionFactory();
factory.setHost("localhost");
Connection connection = factory.newConnection();
Channel channel = connection.createChannel();
channel.exchangeDeclare(EXCHANGE_NAME, "topic");
String queueName = channel.queueDeclare().getQueue();
if (argv.length < 1) {
System.err.println("Usage: ReceiveLogsTopic [binding_key]...");
System.exit(1);
}
for (String bindingKey : argv) {
channel.queueBind(queueName, EXCHANGE_NAME, bindingKey);
}
System.out.println(" [*] Waiting for messages. To exit press CTRL+C");
Consumer consumer = new DefaultConsumer(channel) {
#Override
public void handleDelivery(String consumerTag, Envelope envelope,
AMQP.BasicProperties properties, byte[] body) throws IOException {
String message = new String(body, "UTF-8");
System.out.println(" [x] Received '" + envelope.getRoutingKey() + "':'" + message + "'");
}
};
channel.basicConsume(queueName, true, consumer);
}
}
Method to execute the code and saving in a file:
java -cp $CP ReceiveLogsTopic "#" > logfile.log
I hope it helps and give you a Idea.
Actually best way is to use DB but if you are not ok with it means you can give a try by having the messages in a file and track and reuse it.
That is you can save the details in file while executing and can track it as you needed in runtime.
Note : I have attached the sample code given in tutorial, because anyone can track details even though if link is getting changed in future.
Related
After I went to multiple sites and learned JMS I wrote a JMS standalone client to read messages from a database and send them one by one. I also want to receive message one by one message and then update the database.
I need to send a message to a queue and the other application using standard JMS which will consume a TextMessage and whose body will be read as an ISO-8859-1 string. Also they will similarly send reply as a TextMessage.
I wrote a for loop to read the message one by one from the DB.
I am new to JMS so could you please correct me whether my below code works properly to read and send messages to a queue and receive and update the DB. Is there any thing I need to change in the JMS Type or any thing I need to correct. Does the for loop work fine?
/*MQ Configuration*/
MQQueueConnectionFactory mqQueueConnectionFactory = new MQQueueConnectionFactory();
mqQueueConnectionFactory.setHostName(url);
mqQueueConnectionFactory.setChannel(channel);//communications link
mqQueueConnectionFactory.setPort(port);
mqQueueConnectionFactory.setQueueManager(qmgr);//service provider
mqQueueConnectionFactory.setTransportType(JMSC.MQJMS_TP_CLIENT_MQ_TCPIP);
/*Create Connection */
QueueConnection queueConnection = mqQueueConnectionFactory.createQueueConnection();
queueConnection.start();
/*Create session */
QueueSession queueSession = queueConnection.createQueueSession(false, Session.AUTO_ACKNOWLEDGE);
/*Create response queue */
// Queue queue = queueSession.createQueue("QUEUE.RESPONSE");
int messageCount = 0;
Queue queue = queueSession.createQueue(replytoQueueName);
QueueSender queueSender = null;
QueueReceiver queueReceiver=null;
for (Testbean testBean : testbeanList) {
String testMessage = testBean.getMessage();
/*Create text message */
textMessage = queueSession.createTextMessage(testMessage);
logger.info("Text messages sent: " + messageCount);
textMessage.setJMSReplyTo(queue);
textMessage.setJMSType("mcd://xmlns");//message type
textMessage.setJMSExpiration(2*1000);//message expiration
textMessage.setJMSDeliveryMode(DeliveryMode.PERSISTENT); //message delivery mode either persistent or non-persistemnt
/*Create sender queue */
// QueueSender queueSender = queueSession.createSender(queueSession.createQueue("QUEUE.REQEST"));
queueSender = queueSession.createSender(queueSession.createQueue(outputQName));
queueSender.setTimeToLive(2*1000);
queueSender.send(textMessage);
/*After sending a message we get message id */
System.out.println("after sending a message we get message id "+ textMessage.getJMSMessageID());
String jmsCorrelationID = " JMSCorrelationID = '" + textMessage.getJMSMessageID() + "'";
/*Within the session we have to create queue reciver */
queueReceiver = queueSession.createReceiver(queue,jmsCorrelationID);
/*Receive the message from*/
Message message = queueReceiver.receive(60*1000);
// String responseMsg = ((TextMessage) message).getText();
byte[] by = ((TextMessage) message).getText().getBytes("ISO-8859-1");
logger.info(new String(by));
String responseMsg = new String(by,"UTF-8");
testDAO rmdao = new testDAO();
rmdao.updateTest(responseMsg, jmsCorrelationID);
messageCount += 1;
}
queueSender.close();
queueReceiver.close();
queueSession.close();
queueConnection.close();
Couple of things:
I would create your QueueSender and the Queue object its sending messages to outside the for loop since they don't appear to be changing.
Without the corresponding consumer code it's ultimately impossible to tell if the selector will work or not, but not invoking setCorrelationID() on the message you send looks a bit strange to me. Using the provider-assigned message ID may be a common pattern with IBM MQ request/reply applications, but the general pattern for using a correlation ID is to invoke setJMSCorrelationID() on the sent message. This makes the code more clear and also allows the application to directly control the uniqueness of the correlation ID. This is potentially important for application portability (e.g. if you migrated from IBM MQ to a different JMS provider) since different JMS providers use styles/formats of message ID specific to their particular implementation. Also, regarding the message ID the JMS spec states, "The exact scope of uniqueness is provider defined," which in my opinion is not a strong enough guarantee of uniqueness especially when using something like java.util.UUID.randomUUID().toString() is so simple.
You should ensure that you're using an XA transaction for both the JMS & database work so that they are atomic.
Close your JMS resources in a finally block.
I have a problem when reading messages from multiple JMS Queues in a single transaction using WebLogic JMS client (wlthin3client.jar) from WebLogic 11g (WebLogic Server 10.3.6.0). I am trying to read first one message from queue Q1 and then, if this message satisfy some requirements, read other message (if available at that time) from queue Q2.
I expect that after committing the transaction both messages should disappear from Q1 and Q2. In case of rollback - messages should remain in both Q1 and Q2.
My first approach was to use an asynchronous queue receiver to read from Q1 and then synchronously read from Q2 when it is needed:
void run() throws JMSException, NamingException {
QueueConnectionFactory cf = (QueueConnectionFactory) ctx.lookup(connectionFactory);
// create connection and session
conn = cf.createQueueConnection();
session = conn.createQueueSession(true, Session.SESSION_TRANSACTED);
Queue q1 = (Queue) ctx.lookup(queue1);
// setup async receiver for Q1
QueueReceiver q1Receiver = session.createReceiver(q1 );
q1Receiver.setMessageListener(this);
conn.start();
// ...
// after messages are processed
conn.close();
}
#Override
public void onMessage(Message q1msg) {
try {
QueueReceiver q2receiver = session.createReceiver(queue2);
if(shouldReadFromQ2(q1msg)){
// synchronous receive from Q2
Message q2msg = q2receiver.receiveNoWait();
process(q2msg);
}
session.commit();
} catch (JMSException e) {
e.printStackTrace();
} finally {
q2receiver.close();
}
}
Unfortunately even though I issue a session.commit() the message from Q1 remains uncommitted. It is in receive state until the connection or receiver is closed. Then is seems to be rolled back as it gets delayed state.
Other observations:
Q1 message is correctly committed if Q2 is empty and there is nothing to read from it.
The problem does not occur when I am using synchronous API in similar, nested way for both Q1 and Q2. So if I use q1Receiver.receiveNoWait() everything is fine.
If I use asynchronous API in similar, nested way for Q1 and Q2, then only Q1 message listener is called and commit works on Q1. But Q2 message listener is not called at all and Q2 is not committed (message stuck in receive / delayed).
Am I misusing the API somehow? Or is this a WLS JMS bug? How to combine reading from multiple queues with asynchronous API?
It turns out this is an WLS JMS bug 28637420.
The bug status says it is fixed, but I wouldn't rely on this - a WLS 11g patch with this fix doesn't work (see bug 29177370).
Oracle says that this happens because two different delivery mechanisms (synchronous messages vs asynchronous messages) were not designed to work together on the same session.
Simplest way to work around the problem is just use synchronous API (polling) for cases when you need to work on multiple queues in a single session. I decided on this approach.
Another option suggested by oracle is to to use UserTransactions with two different sessions, one session for the async consumer and another session for the synchronous consumer. I didn't test that though.
I am sending some messages to a JMS queue. What are the possible ways to search for a particular message in a queue to consume?
I tried out in the following way: I am setting the JMSCorrelationID while sending a message to the queue:
public void createDQueue(String queuename, String json, Integer userid) {
try {
QueueSession.AUTO_ACKNOWLEDGE );
Queue queue = session.createQueue(queuename);
ObjectMessage objectMessage = session.createObjectMessage();
objectMessage.setJMSCorrelationID(String.valueOf(userid));
objectMessage.setObject(json);
session.createSender(queue).send(objectMessage);
session.close();
connection.close();
}catch(Exception e){
e.printStackTrace();
}
}
In the consumer code I want to get that particular message based on the JMSCorrelationID. I am not able to get that particular message. Can you suggest a solution?
public void getSpecificMessage(String queuename, Integer userid) {
try {
QueueConnectionFactory connectionFactory = new ActiveMQConnectionFactory( "tcp://localhost:61616");
((ActiveMQConnectionFactory) connectionFactory).setUseAsyncSend(true);
QueueConnection connection = connectionFactory.createQueueConnection();
connection.start();
QueueSession session = connection.createQueueSession( false,
QueueSession.AUTO_ACKNOWLEDGE );
String id = String.valueOf(userid);
Queue queue = session.createQueue(queuename);
QueueReceiver receiver = session.createReceiver(queue, "JMSCorrelationID="+id);
Message message = receiver.receive();
} catch (JMSException e) {
e.printStackTrace();
}
}
Your first problem is that you are trying to think about the message broker as a database, you must always remember this sage piece of advice, "A message broker is not a database".
There are certain limits on how deep a consumer or Queue browser can go into a destination before the broker will not page in more messages from disk, so you need to check your depth and see if its large than you maxPageSize setting and adjust as needed, but remember that messages paged in remain in memory until consumed.
Just wrap the id value in single quotes
"JMSCorrelationID='"+id+"'"
This functionality is not recommended to be used , there are lot more complications as explained by Tim , but if you want to obsolutely work with it make the change
You can search messages using the MeessageID of a message. This would be fast as messaging providers index messages on message id. There are other way to search based on CorrelationId, meta data etc.
But please remember the primary objective of using a messaging provider is to connect applications in a time independent manner. The receiving application must get messages as soon as possible. If messages are piling up in a queue, it indicates a problem that must be addressed.
I have a server containing folders date wise and each folder further contains many files (size 200kb each) containing all the log for a particular day. I am new to RabbitMQ , while going through the documentation of RabbitMQ i found below code for Producer
Refer Link: https://github.com/rabbitmq/rabbitmq-tutorials/blob/master/java/Send.java
public class Send {
private final static String QUEUE_NAME = "hello";
public static void main(String[] argv) throws Exception {
ConnectionFactory factory = new ConnectionFactory();
factory.setHost("localhost");
Connection connection = factory.newConnection();
Channel channel = connection.createChannel();
channel.queueDeclare(QUEUE_NAME, false, false, false, null);
String message = "Hello World!";
channel.basicPublish("", QUEUE_NAME, null, message.getBytes());
System.out.println(" [x] Sent '" + message + "'");
channel.close();
connection.close();
}
}
on the above code i have added sample string "Hello World!" to publish. As stated above in the problem description that i have to read the log information from the server with different date stamp directory So do i need to write a simply an infinite loop(as logs are continuously updated) and recursively read all directory and files and Then for each line of File i can compose a message and then publish it to receiver ?
In this case our channel will never close and Connection will be always up is it an idle condition with RabbitMQ ?
Is it possible for RabbitMQ to mark the file which are read and don't read it again OR i need to manage it programmatically like renaming the file and folder with some different names. I was thinking this as might be our program gets terminated with some power failure or something while i am in mid of any file and then how can i guarantee that records would not be duplicated ?
Any other best way to achieve this would be great help for me. Thanks in advance.
I would enqueue a list of files to process to RabbitMQ and then have a separate set of processes picking up messages from that queue to do what you want with the data. Then try to make sure to subscribe to the queues in ack mode, so RabbitMQ will only delete the message from the queue once you ack it. With this setting, you should prevent sending the same information twice.
That would work on most situations. I say most, because if RabbitMQ sends a message to your consumer, then your consumer takes an action (like replicating the information, or placing an entry on a database) and then the connection to RabbitMQ dies before you sent the ack to RabbitMQ, then the broker has no way of telling that you already processed the message, so it will deliver it again later.
I'm using ActiveMQ on a simulation of overloading servers in Java. And mainly it goes ok, but when I get over 600 requests the thing just go WTF!
I think the bottleneck is my Master Server which is this guy below. I'm already reusing the connection and creating various sessions to consume messages from clients. Like I said, I'm using about 50-70 sessions per connection, reutilizing the connection and queue. Any idea of what I can reuse/optimize of my components/listener below?
The architecture is the follow:
* = various
Client ---> JMS MasterQueue ---> * Master ---> JMS SlavaQueue ---> * SlaveQueue
Mainly I'm creating a Temp Queue for each session of Master --> Slave communication, is that a big problem on performance?
/**
* This subclass implements the processing log of the Master JMS Server to
* propagate the message to the Server (Slave) JMS queue.
*
* #author Marcos Paulino Roriz Junior
*
*/
public class ReceiveRequests implements MessageListener {
public void onMessage(Message msg) {
try {
ObjectMessage objMsg = (ObjectMessage) msg;
// Saves the destination where the master should answer
Destination originReplyDestination = objMsg.getJMSReplyTo();
// Creates session and a sender to the slaves
BankQueue slaveQueue = getSlaveQueue();
QueueSession session = slaveQueue.getQueueConnection()
.createQueueSession(false, Session.AUTO_ACKNOWLEDGE);
QueueSender sender = session
.createSender(slaveQueue.getQueue());
// Creates a tempQueue for the slave tunnel the message to this
// master and also create a masterConsumer for this tempQueue.
TemporaryQueue tempDest = session.createTemporaryQueue();
MessageConsumer masterConsumer = session
.createConsumer(tempDest);
// Setting JMS Reply Destination to our tempQueue
msg.setJMSReplyTo(tempDest);
// Sending and waiting for answer
sender.send(msg);
Message msgReturned = masterConsumer.receive(getTimeout());
// Let's check if the timeout expired
while (msgReturned == null) {
sender.send(msg);
msgReturned = masterConsumer.receive(getTimeout());
}
// Sends answer to the client
MessageProducer producerToClient = session
.createProducer(originReplyDestination);
producerToClient.send(originReplyDestination, msgReturned);
} catch (JMSException e) {
logger.error("NO REPLY DESTINATION PROVIDED", e);
}
}
}
Well, After some reading I found out how to optimize.
We should reuse some session variables, such as the sender and tempqueue. Instead of creating new ones.
Another approach is put the stack size for thread in Java lower, following this link
ActiveMQ OutOfMemory Can't create more threads
It could have to do with the configuration of the listener thread pool. It could be that up to a certain threshold number of requests per second the listener is able to keep up and process the incoming requests in a timely way, but above that rate it starts to fall behind. it depends on the work done for each incoming request, the incoming request rate, the memory and CPU available to each listener, and the number of listeners allocated.
If this is true, you should be able to watch the queue and see when the number of incoming messages start to back up. That's the point at which you need to increase the resources and number of listeners to process efficiently.