I have a situation where I need to read a(on going) messages from a topic and put them on another Queue . I have doubts do I need jms Queue or I can be satisfied with an in memory java Queue . I will do the reading from the Queue by other thread(s) in same jvm and will do client acknowledge of the message to the topic after reading the message from the (in memory) queue and process it as necessary (send it to remote IBM MQ) .So if my client crash the messages that were exist in the in memory queue will be lost but will still exist on topic and will be redeliver to me . Am I right ?
Some of this depends on how you have set up the queue/topic and the connection string you are using to read from IBM's MQ but if you are using the defaults you WILL lose messages if you're reading it to an in-memory queue.
I'd use ActiveMQ, either in the same JVM as a library so you have it taking care of receipt, delivery and persistence.
Also if you are listening to a topic you're not going to be sent missed messages after a crash even if you reconnect afterwards unless you've
configured your client as a durable subscriber
reconnect in the time (before the expireMessagesPeriod is reached)
The ActiveMQ library is not large and worth using if ensure delivery of every message is important, especially in an asynchronous environment.
Main difference is that in-memory loses data when the application goes down; JMS queue loses data when the server goes down IF the topic/queue is not persistent. The former is much more likely than the latter, so I'd also say go with JMS.
Related
I am using Java with ActiveMQ for sending messages from Embedded-Linux. My requirements are that messages should be persisted if the destination broker is not available. I chose KahaDB as the persistence adapter in order for the messages to survive a broker restart.
broker = new SslBrokerService();
KahaDBPersistenceAdapter kahaDBPersistenceAdapter=new KahaDBPersistenceAdapter();
kahaDBPersistenceAdapter.setJournalMaxFileLength(16 * 1024 * 1024);
broker.setPersistenceAdapter(kahaDBPersistenceAdapter);
/** Connector setup code */
broker.start();
Even though I have limited the maximum file size to 16MB i would like to ONLY persist messages if the destination broker/consumer is not available. The reason for this is to limit the CPU and disk usage. For example: Produce message 1, attempt to send message. If successful persistence should not occur since I am not interested in re-sending already sent data ( or should occur only in memory ) . Produce messages 2,3,4 but destination broker is unavailable, then persist messages on disk such that in the unlikely shutdown of borker messages will survive.
After going back and forth through the documentation of ActiveMQ I have not found any answer for my use case so I am asking this great community if what I am looking for is actually possible.
You would have to write a custom plugin to get that exact scenario. JMS does not describe a use case to persist if only the consumer is not available, b/c the reliability contract is also with the producer. I think a store-and-forward design would be what you are looking for. Configure embeddedBroker to use a static: networkConnector to send it to the destination broker. ActiveMQ will automatically push messages up to the destination server when available, and queue them up on the embedded server if the remote broker is unavailable.
I understand JMS as depicted by the following diagram:
(source: techhive.com)
Is there any way for me to access the underlying database using JMS or some other thing? Further, the JDBC connections that the JMS server maintains, can I add new connections in it so as to access other databases also and do CRUD operations on them? If yes, how?
Where did you get this from?
Normally JMS is used to send messages to queue (or topics). You have message producers that push messages in the queue and message consumers consume them and process it.
In your exemple it seems that you have multiple queues. One for the messages that need to be processed, and one for each client to retrieve the result the processing of its messages.
With JMS Server you don't necessarily have a database behind. Everything can stay in memory, or can be written to files. You will need database server behind only if you configure your JMS server to be persistent (and to assure that even if server/application crash your messages won't be lost). But in that case you will never have to interact with the database. Only the JMS server will and you will interact with the JMS server sending and consuming messages.
We're using a Tibco client implementation of the JMS API. We have a MessageListener with an onMessage() implementation.
Is there a way with the Tibco client to inspect past (received) messages in the queue? (I realise this totally ignores the logical concept of a queue - I wondered if the queue implementation provided this workaround.)
No. Not for "past" messages.
Messages acknowledged by the receiver are removed from the queue - as their "function" is already done.
You could have a Listener configured to persist your messages in some DB or file - but for future messages.
A client uses a QueueBrowser object to look at messages on a queue without removing them.
#hawkeye Its not possible to browse messages from the past... At any point of time , you can browse destinations only for the pending messages.
There is no way for you browse all the received messages as EMS server usually deletes the message once it has delivered ( acknowledged) for the given delivery mode.
One possible way is to a send copy of the messages to another queue (without any receivers) before actually confirming the messages.
Also it depends on your acknowledgement mode and logic involved.
We are running a high throughput system that utilizes tibco-ems JMS to pass large numbers of messages to and from our main server to our client connections. We've done some statistics and have determined that JMS is the causing a lot of latency. How can we make tibco JMS more performant? Are there any resources that give a good discussion on this topic.
Using non-persistent messages is one option if you don't need persistence.
Note that even if you do need persistence, sometimes it's better to use non persistent messages, and in case of a crash perform a different recovery action (like resending all messages)
This is relevant if:
crashes are rare (as the recovery takes time)
you can easily detect a crash
you can handle duplicate messages (you may not know exactly which messages were delivered before the crash
EMS also provides some mechanisms that are persistent, but less bullet proof then classic guaranteed delivery
these include:
instead of "exactly once" message delivery you can use "at least once" or "up to once" delivery.
you may use the pre-fetch mechanism which causes the client to fetch messages to memory before your application request them.
EMS should not be the bottle neck. I've done testing and we have gotten a shitload of throughput on our server.
You need to try to determine where the bottle neck is. Is the problem in the producer of the message or the consumer. Are messages piling up on the queue.
What type of scenario are you doing.
Pub/sup or request reply?
are you having temporary queue pile up. Too many temporary queues can cause performance issues. (Mostly when they linger because you didn't close something properly)
Are you publishing to a topic with durable subscribers if so. Try bridging the topic to queue and reading from those. Durable subscribers can cause a little hiccup in performance too since it needs to track who has copies of all messages.
Ensure that your sending process has one session and multiple calls through that session. Don't open a complete session for each operation. Re-use where possible. Do the same for the consumer.
make sure you CLOSE when you are done. EMS doesn't clear things up. So if you make a connection and just close your app the connection still is there and sucking up resources.
review your tolerance for lost messages in the even of a crash. If you are doing Client ack and it doesn't matter if you crash processing the message then switch to auto. Also I believe if you are using (TEMS - Tibco EMS for WCF) there's a problem with the session acknowledge. So a message is only when its processed on the whole message, we switched from Client ACK to the one that had Dups ok and it worked better)
My Java EE application sends JMS to queue continuously, but sometimes the JMS consumer application stopped receiving JMS. It causes the JMS queue very large even full, that collapses the server.
My server is JBoss or Websphere. Do the application servers provide strategy to remove "timeout" JMS messages?
What is strategy to handle large JMS queue? Thanks!
With any asynchronous messaging you must deal with the "fast producer/slow consumer" problem. There are a number of ways to deal with this.
Add consumers. With WebSphere MQ you can trigger a queue based on depth. Some shops use this to add new consumer instances as queue depth grows. Then as queue depth begins to decline, the extra consumers die off. In this way, consumers can be made to automatically scale to accommodate changing loads. Other brokers generally have similar functionality.
Make the queue and underlying file system really large. This method attempts to absorb peaks in workload entirely in the queue. This is after all what queuing was designed to do in the first place. Problem is, it doesn't scale well and you must allocate disk that 99% of the time will be almost empty.
Expire old messages. If the messages have an expiry set then you can cause them to be cleaned up. Some JMS brokers will do this automatically while on others you may need to browse the queue in order to cause the expired messages to be deleted. Problem with this is that not all messages lose their business value and become eligible for expiry. Most fire-and-forget messages (audit logs, etc.) fall into this category.
Throttle back the producer. When the queue fills, nothing can put new messages to it. In WebSphere MQ the producing application then receives a return code indicating that the queue is full. If the application distinguishes between fatal and transient errors, it can stop and retry.
The key to successfully implementing any of these is that your system be allowed to provide "soft" errors that the application will respond to. For example, many shops will raise the MAXDEPTH parameter of a queue the first time they get a QFULL condition. If the queue depth exceeds the size of the underlying file system the result is that instead of a "soft" error that impacts a single queue the file system fills and the entire node is affected. You are MUCH better off tuning the system so that the queue hits MAXDEPTH well before the file system fills but then also instrumenting the app or other processes to react to the full queue in some way.
But no matter what else you do, option #4 above is mandatory. No matter how much disk you allocate or how many consumer instances you deploy or how quickly you expire messages there is always a possibility that your consumer(s) won't keep up with message production. When this happens your producer app should throttle back, or raise an alarm and stop or do anything other than hang or die. Asynchronous messaging is only asynchronous up to the point that you run out of space to queue messages. After that your apps are synchronous and must gracefully handle that situation, even if that means to (gracefully) shut own.
Sure!
http://download.oracle.com/docs/cd/E17802_01/products/products/jms/javadoc-102a/index.html
Message#setJMSExpiration(long) does exactly what you want.