My Java EE application sends JMS to queue continuously, but sometimes the JMS consumer application stopped receiving JMS. It causes the JMS queue very large even full, that collapses the server.
My server is JBoss or Websphere. Do the application servers provide strategy to remove "timeout" JMS messages?
What is strategy to handle large JMS queue? Thanks!
With any asynchronous messaging you must deal with the "fast producer/slow consumer" problem. There are a number of ways to deal with this.
Add consumers. With WebSphere MQ you can trigger a queue based on depth. Some shops use this to add new consumer instances as queue depth grows. Then as queue depth begins to decline, the extra consumers die off. In this way, consumers can be made to automatically scale to accommodate changing loads. Other brokers generally have similar functionality.
Make the queue and underlying file system really large. This method attempts to absorb peaks in workload entirely in the queue. This is after all what queuing was designed to do in the first place. Problem is, it doesn't scale well and you must allocate disk that 99% of the time will be almost empty.
Expire old messages. If the messages have an expiry set then you can cause them to be cleaned up. Some JMS brokers will do this automatically while on others you may need to browse the queue in order to cause the expired messages to be deleted. Problem with this is that not all messages lose their business value and become eligible for expiry. Most fire-and-forget messages (audit logs, etc.) fall into this category.
Throttle back the producer. When the queue fills, nothing can put new messages to it. In WebSphere MQ the producing application then receives a return code indicating that the queue is full. If the application distinguishes between fatal and transient errors, it can stop and retry.
The key to successfully implementing any of these is that your system be allowed to provide "soft" errors that the application will respond to. For example, many shops will raise the MAXDEPTH parameter of a queue the first time they get a QFULL condition. If the queue depth exceeds the size of the underlying file system the result is that instead of a "soft" error that impacts a single queue the file system fills and the entire node is affected. You are MUCH better off tuning the system so that the queue hits MAXDEPTH well before the file system fills but then also instrumenting the app or other processes to react to the full queue in some way.
But no matter what else you do, option #4 above is mandatory. No matter how much disk you allocate or how many consumer instances you deploy or how quickly you expire messages there is always a possibility that your consumer(s) won't keep up with message production. When this happens your producer app should throttle back, or raise an alarm and stop or do anything other than hang or die. Asynchronous messaging is only asynchronous up to the point that you run out of space to queue messages. After that your apps are synchronous and must gracefully handle that situation, even if that means to (gracefully) shut own.
Sure!
http://download.oracle.com/docs/cd/E17802_01/products/products/jms/javadoc-102a/index.html
Message#setJMSExpiration(long) does exactly what you want.
Related
We have a use case that holds active jobs in a RabbitMQ job queue. The Clients, when they are free, pull jobs from this queue. Pretty normal. But in our case, we do not ACK the jobs. We allow them to stay in the Unacked state so if the Client dies, the job goes to a pre-defined DeadLetter queue. We then have a process that pulls messages from dead-letter, and decides to either requeue message back to original job-queue, or discard.
This has worked well for a long time. Now, we have upgraded to a newer version of RMQ, and found that we get disconnects with PRECONDITION_FAILED, because the default ack timeout of 30 minutes has expired.
Beyond removing this from the server, does anyone know a way to configure this on a per-message level?
While some might say just ACK the job, and use a handler to return to DEADLETTER if needed. Well, sorry, that will not work for us.
So, any thoughts?
No, there is not at this time. You should configure the default to be greater than the longest expected job duration. Please note that if you are using quorum queues this may cause disk usage growth because the log files can't be compacted while messages are outstanding.
We may make this timeout configurable in a more granular way, so please keep an eye on future RabbitMQ releases for that.
NOTE: the RabbitMQ team monitors the rabbitmq-users mailing list and only sometimes answers questions on StackOverflow.
We have an external system that pushes data to SQS, and the consumer is another java service, that listens and created the requests, distributes the tasks, then acknowledges the SQS.
But a memory heap issue occurred a time, and from the logs, it is understood that the java consumer consumed multiple times the Queue, but all times throws a memory exception, and seems saved all later.
So the Q didn't get acknowledged, so data was available for the consumer. The db commits happened after a certain while only.
The strategy to avoid this kind of issue, what approach we can take?
Exception Handling, SQS retry mechanism change, check exists kind of validation..?
I have a simple test case where I start a HornetQ server (V2.4.7.Final) as part of a Spring context. This works quite well and I have access to a queue via JMS, the HornetQ API and/or JMX.
Testcase
The test case is supposed to empty the queue at start, check that it is empty and then add 10 messages to the queue. As long as there are no consumers on this queue, this works using either the management queue or JMSQueueControl. Even doing some operation on the queue via JMX is working well.
Problem description
As soon as I add a message listener to this queue using Spring configuration - and the listener consumes the messages as expected - I cannot remove all messages from the queue. Neither method invocation via JMX, nor the management queue, nor JMSQueueControl is working, i.e. the methods are called without exception but they show no effect.
I thought that maybe I have to pause the queue before doing some modifications to its content but pausing does not work either. I can see that the queue is paused via JMX and the same is reported when using the API but the consumer still consumes messages from the very queue. Thus I think it has not been paused at all.
I know that it is difficult without the source code but from my point of view this is all pretty basic setup as you find it in many, many tutorials. Could anyone give advice what I am doing wrong. In case any source code is needed, please leave a comment and I will add the revelant parts.
HornetQ supports removal of messages which are in the queue on the broker side. Once the messages are dispatched to the consumer and buffered on the consumer, it is not possible to remove the messages from the consumer buffer using any management API.
One way to solve this (if you must) is to disable consumer buffering by setting the consumer-window-size to 0, but be aware of the potential performance degradation.
Otherwise, you need to handle it programmatically; by adding some validity checks before processing the message.
You can read more about HornetQ Flow control here https://docs.jboss.org/hornetq/2.2.5.Final/user-manual/en/html/flow-control.html
I have a situation where I need to read a(on going) messages from a topic and put them on another Queue . I have doubts do I need jms Queue or I can be satisfied with an in memory java Queue . I will do the reading from the Queue by other thread(s) in same jvm and will do client acknowledge of the message to the topic after reading the message from the (in memory) queue and process it as necessary (send it to remote IBM MQ) .So if my client crash the messages that were exist in the in memory queue will be lost but will still exist on topic and will be redeliver to me . Am I right ?
Some of this depends on how you have set up the queue/topic and the connection string you are using to read from IBM's MQ but if you are using the defaults you WILL lose messages if you're reading it to an in-memory queue.
I'd use ActiveMQ, either in the same JVM as a library so you have it taking care of receipt, delivery and persistence.
Also if you are listening to a topic you're not going to be sent missed messages after a crash even if you reconnect afterwards unless you've
configured your client as a durable subscriber
reconnect in the time (before the expireMessagesPeriod is reached)
The ActiveMQ library is not large and worth using if ensure delivery of every message is important, especially in an asynchronous environment.
Main difference is that in-memory loses data when the application goes down; JMS queue loses data when the server goes down IF the topic/queue is not persistent. The former is much more likely than the latter, so I'd also say go with JMS.
We are running a high throughput system that utilizes tibco-ems JMS to pass large numbers of messages to and from our main server to our client connections. We've done some statistics and have determined that JMS is the causing a lot of latency. How can we make tibco JMS more performant? Are there any resources that give a good discussion on this topic.
Using non-persistent messages is one option if you don't need persistence.
Note that even if you do need persistence, sometimes it's better to use non persistent messages, and in case of a crash perform a different recovery action (like resending all messages)
This is relevant if:
crashes are rare (as the recovery takes time)
you can easily detect a crash
you can handle duplicate messages (you may not know exactly which messages were delivered before the crash
EMS also provides some mechanisms that are persistent, but less bullet proof then classic guaranteed delivery
these include:
instead of "exactly once" message delivery you can use "at least once" or "up to once" delivery.
you may use the pre-fetch mechanism which causes the client to fetch messages to memory before your application request them.
EMS should not be the bottle neck. I've done testing and we have gotten a shitload of throughput on our server.
You need to try to determine where the bottle neck is. Is the problem in the producer of the message or the consumer. Are messages piling up on the queue.
What type of scenario are you doing.
Pub/sup or request reply?
are you having temporary queue pile up. Too many temporary queues can cause performance issues. (Mostly when they linger because you didn't close something properly)
Are you publishing to a topic with durable subscribers if so. Try bridging the topic to queue and reading from those. Durable subscribers can cause a little hiccup in performance too since it needs to track who has copies of all messages.
Ensure that your sending process has one session and multiple calls through that session. Don't open a complete session for each operation. Re-use where possible. Do the same for the consumer.
make sure you CLOSE when you are done. EMS doesn't clear things up. So if you make a connection and just close your app the connection still is there and sucking up resources.
review your tolerance for lost messages in the even of a crash. If you are doing Client ack and it doesn't matter if you crash processing the message then switch to auto. Also I believe if you are using (TEMS - Tibco EMS for WCF) there's a problem with the session acknowledge. So a message is only when its processed on the whole message, we switched from Client ACK to the one that had Dups ok and it worked better)