I am writing a Java application, running in a LINUX environment, that does transactions on an MQ using SYNCPOINT. It uses Websphere MQ Java Classes to interact with the MQ service. What I am doing in my code is the following (pseudo):
MQGetMessageOptions gmo = new MQGetMessageOptions();
gmo.options = MQConstants.MQGMO_FAIL_IF_QUIESCING | MQConstants.MQGMO_SYNCPOINT;
MQMessage message = new Message();
queue.get(message, gmo);
// process the message, save to database
databaseConnection.commit();
queueManager.commit();
I basically grab the message, process it, persist to database, then call a commit on the queueManager. The process listens for a message on TIBRV in order to do a graceful shutdown.
I've been testing the process to make sure no messages are lost. I place 20k messages on a queue, then run the process. I perform a graceful shutdown call in the middle of processing. I then compare the amount of messages on the queue versus the amount of messages in database. When a graceful shutdown occurs via TIBRV message, the number of MQ messages + the number of DB messages = total messages originally on the queue.
However, when I do a kill or kill -9, I see that a message is lost. I always end up with a result of 19999 total messages.
Is there a way I can investigate how I am losing this message? Is there anything that occurs on the Websphere App Server that I would need to be aware of?
There is no reason to expect the numbers to reconcile when using single-phase commit. The program will always be between the WMQ and DB Commit call or else between the DB and WMQ Commit call when you kill it.
What you are asking for would require 2-Phase (XA) Commit. For WMQ, 2PC would require the application to be using bindings mode and for WMQ to be the resource coordinator. You would then call MQBEGIN, execute your WMQ and DB updates, then call MQCOMMIT. This way both the WMQ and DB transactions would succeed or fail together.
Are you connecting or MQ in bindings mode or client mode? In my experience, transactions didn't work out of the box in client mode, but they did in bindings mode.
Related
I have a conceptual question regarding MQ and Java application.
I have a batch application (java) that will read messages from MQ (set to persist messages) and I have two options, get it sync or assync...
The point is that I'm confused what happens when the async message is sent and the application cannot handle that in that moment. Is the message lost? Kept in any kind of queue (inside the application) or should I handle with exceptions to remake the request when application is available?
MQ > are there resources to process? (no more threads for example) > No > Application > Reject > ?
Was it clear enough?
Your Persistent message is held by MQ on a queue until your getter can get it.
But note this. If your app is getting Persistent messages, ensure it uses syncpoints (allowing you to commit or rollback). Then the app can rollback its get if it failed to process the message correctly, and the message will be available on the queue again for the next MQGET.
I have a situation where I need to read a(on going) messages from a topic and put them on another Queue . I have doubts do I need jms Queue or I can be satisfied with an in memory java Queue . I will do the reading from the Queue by other thread(s) in same jvm and will do client acknowledge of the message to the topic after reading the message from the (in memory) queue and process it as necessary (send it to remote IBM MQ) .So if my client crash the messages that were exist in the in memory queue will be lost but will still exist on topic and will be redeliver to me . Am I right ?
Some of this depends on how you have set up the queue/topic and the connection string you are using to read from IBM's MQ but if you are using the defaults you WILL lose messages if you're reading it to an in-memory queue.
I'd use ActiveMQ, either in the same JVM as a library so you have it taking care of receipt, delivery and persistence.
Also if you are listening to a topic you're not going to be sent missed messages after a crash even if you reconnect afterwards unless you've
configured your client as a durable subscriber
reconnect in the time (before the expireMessagesPeriod is reached)
The ActiveMQ library is not large and worth using if ensure delivery of every message is important, especially in an asynchronous environment.
Main difference is that in-memory loses data when the application goes down; JMS queue loses data when the server goes down IF the topic/queue is not persistent. The former is much more likely than the latter, so I'd also say go with JMS.
I'm using XMS (i.e IBM Message Services Client for .NET) to connect to IBM MQ and get the messages.
With that said, I wanted to know how to get count of messages on a Queue at any given point of time.
(Explored on IQueueBrowser.GetEnumerator, but it will download the messages onto client.)
XMS .NET is an implementation of JMS specification and JMS specific does not define a method or a property that retrieves count of messages in a queue. Hence XMS does not provide a way to do that.
Anything to do with queue attributes is a provider specific administrative job, so you have to use message provider specific APIs.
But why do you want to know the count of messages? Application should be coded to receive messages continuously. Your application can choose to quit receiving messages when there are no more messages in the queue or wait for further messages. If there are no messages, the receive call will return with null message object. Your application can check for this and decide to make further receive calls or quit.
HTH
I understand JMS as depicted by the following diagram:
(source: techhive.com)
Is there any way for me to access the underlying database using JMS or some other thing? Further, the JDBC connections that the JMS server maintains, can I add new connections in it so as to access other databases also and do CRUD operations on them? If yes, how?
Where did you get this from?
Normally JMS is used to send messages to queue (or topics). You have message producers that push messages in the queue and message consumers consume them and process it.
In your exemple it seems that you have multiple queues. One for the messages that need to be processed, and one for each client to retrieve the result the processing of its messages.
With JMS Server you don't necessarily have a database behind. Everything can stay in memory, or can be written to files. You will need database server behind only if you configure your JMS server to be persistent (and to assure that even if server/application crash your messages won't be lost). But in that case you will never have to interact with the database. Only the JMS server will and you will interact with the JMS server sending and consuming messages.
planning on moving a lot of our single threaded synchronous processing batch jobs to a more distributed architecture with workers. the thought is having a master process read records off the database, and send them off to a queue. then have a multiple workers read off the queue to process the records in parallel.
is there any well known java pattern for a simple CLI/batch job that constantly runs to poll/listen for messages on queues? would like to use that for all the workers. or is there a better way to do this? should the listener/worker be deployed in an app container or can it be just a standalone program?
thanks
edit: also to note, im not looking to use JavaEE/JMS, but more hosted solutions like SQS, a hosted RabbitMQ, or IronMQ
If you're using a JavaEE application server (and if not, you should), you don't have to program that logic by hand since the application server does it for you.
You then implement and deploy a message driven bean that listens to a queue and processes the message received. The application server will manage a connection pool to listen to queue messages and create a thread with an instance of your message driven bean which will receive the message and be able to process it.
The messages will be processed concurrently since the application server will have a connection pool and a thread pool available to listen to the queue.
All JavaEE-featured application servers like IBM Websphere or JBoss have configurations available in their admin consoles to create Message Queue listeners depending or the message queue implementation and then bind this message queue listeners to your Message Driven Bean.
I don't a lot about this, and I maybe don't really answer your question, but I tried something a few month ago that might interest you to deals with message queue.
You can have a look at this: http://www.rabbitmq.com/getstarted.html
I seems Work Queue could fix your requirements.