Recently we had to drop all the KahaDB files in order to get a production ActiveMQ server up and running. We waited for 45min but nothing showed up, so we were forced to drop all the info.
The pending messages were not an issue, but there were a number of scheduled jobs that requires rescheduling.
I was thinking, anticipating future similar situations, if it is possible to store that information in a differente storage, or maybe in a different KahaDB using mKahaDB?
Thanks
Carlos
Related
I am using a managed RabbitMQ cluster through AWS Amazon-MQ. If the consumers finish their work quickly then everything is working fine. However, depending on few scenarios few consumers are taking more than 30 mins to complete the processing.
In that scenarios, RabbitMQ deletes the consumer and makes the same messages visible again in the queue. Becasue of this another consumer picks it up and starts processing. It is happing in the loop. Therefore the same transaction is getting executed again and I am loosing the consumer as well.
I am not using any AcknowledgeMode so I believe it's AUTO by default and it has 30 mins limit.
Is there any way to increase the Delivery Acknowledgement Timeout for AUTO mode?
Or please let me know if anyone has any other solutions for this.
Reply From AWS Support:
Consumer timeout is now configurable but can be done only by the service team. The change will be permanent irrespective of any version.
So you may update RabbitMQ to latest, and no need to stick with 3.8.11. Provide your broker details and desired timeout, they should be able to do it for you.
This is the response from AWS support.
From my understanding, I see that your workload is currently affected by the consumer_timeout parameter that was introduced in v3.8.15.
We have had a number of reach outs due to this, unfortunately, the service team has confirmed that while they can manually edit the rabbitmq.conf, this will be overwritten on the next reboot or failover and thus is not a recommended solution. This will also mean that all security patching on the brokers where a manual change is applied, will have to be paused. Currently, the service does not support custom user configurations for RabbitMQ from this configuration file, but have confirmed they are looking to address this in future, however, is not able to an ETA on when this will available.
From the RabbitMQ github, it seems this was added for quorum queues in v3.8.15 (https://github.com/rabbitmq/rabbitmq-server/releases/tag/v3.8.15 ), but seems to apply to all consumers (https://github.com/rabbitmq/rabbitmq-server/pull/2990 ).
Unfortunately, RabbitMQ itself does not support downgrades (https://www.rabbitmq.com/upgrade.html )
Thus the recommended workaround and safest action form the service team, as of now is to create a new broker on an older version (3.8.11) and set auto minor version upgrade to false, so that it wont be upgraded.
Then export the configuration from the existing RabbitMQ instance and import it into new instance and use this instance going forward.
I have activemq which has a huge storeUsage ~100GB, sice it was about to exhaust I cnaged to value of storeUsage and triying to restart again but not activemq server is not staring.
What can be issue, nothing more in log in debug mode
Can I just backup kahaDb and delete all file inside kahaDb and restart it ?
Please suggest what to do, Its prod server and we have big issue.
Thanks,
Arvind
If you delete kahaDB the contents of your queues and topics will be gone. If you do not care about them - delete it. It will most likely render your broker usable again.
You should have tried to get rid of old messages in the first place. They use a lot of resources. A message broker is not a database.
You could also try to restore your kahadb on a much more powerful machine.
I assume monitoring of your production environment was faulty or non-existent. So you should invest in that area, too.
We have RV messaging systems publishing and receiving messages.Recently some underlying jars were upgraded - these are serialization jars used by all publishers and subscribers. However , it seems that some of the publishers are still referencing old versions of the serialization jars and therefore the receivers fail when trying to deserialize received messages.
Obviously restarting these publisher services should fix the problem. However , how do I identify all publishers using a particular topic to send messages to ? There must be some RV admin way of listing all the processes that are publishing to a given topic ?
I just gave a similar answer on another question:
There is a really great tool for this called Rai Insight
Basically what it can do is to sit on a box and silently listen all the multicast data and represent statistics even in real time. We used it to monitor traffic flow spikes with just few seconds delay.
It can give you traffic statistics braked down by multicast group, service number or even sending machine. Traffic flow peak/average, retransmission rate peak/average. All you can think of.
It will also give you per-service per-topic information.
we are using a set of Active MQ servers (three ) behind a load balancer .
These configured queues will Persist the Data to a disk (For helping in case of a crash )
My question is Does a developer or MQ admin will take care of these things
Thanks
If the messages are REALLY important, you might think about replication of them. Once persisted to the disk, replicate them on some other machine also. That is minimum what you should do - not keep messages on the same machine. You should be looking at distributed queues:
Distributed Queue
Who's responsibility it is? Well, you companies, the people who design and build the solution. It's everyone's. If you can do it (and I am sure you can try at least), then go ahead.
IMHO in your case the ActiveMQ part needs to be done by developer, and the replication on the Server side by an admin, not necessarily an MQ Admin, but the admin. May be set up a cron job to replicate the needed data?
Cheers,Eugene.
Your setup is as secure as the weakest element of safety. You can loose messages when one server crash (disks). You will not be able to recover messages so You should take care for safety in app.
ActiveMQ can be more safe (but slower). Replicated Message Stores
Look here http://activemq.apache.org/clustering.html
I have a production system that uses ActiveMQ (5.3.2) to send messages from server A to server B. A few weeks ago, the system inexplicably started taking 10+ second to send a message. After a reboot of the producer, the system worked fine.
After investigation, I'm pretty sure this is due to producer flow control. (I have a fairly standard activemq setup). The day before this happened (for other reasons) my consumer software had been acting erratically and had even stopped accepting connections for a while. So I'm guessing this triggered this. (It does puzzle me that the requests were still being throttled a day later).
Question -- how can I confirm that the requests were being throttled. I took a heap dump of the server -- is there data in memory I can look for?
Edit: I've found the following:
WireFormatNegotiator.tcpNoDelayEnabled=false for one of three WireFormatNegotiator instances in the memory. I'm trying to figure out what sets this.
And second (and more important), is there a way I can use JMX to tell if the messages are being throttled? I'd like to set up a Nagios alert to let me know if this happens in the future. What property should I check for with JMX?
you can configure your producer client to throw javax.jms.ResourceAllocationException exceptions which can then be detected/logged, etc. just set one of the following...
<systemUsage>
<systemUsage sendFailIfNoSpaceAfterTimeout="3000">
...OR...
<systemUsage sendFailIfNoSpace="true">