When my activemq goes down, how can i store the message that are on its way to activemq?If the answer is using persistance db , then how and when can i resend those messages that were stored in db back to activemq queue(assuming it is up and working)?
(To give you a complete background: whenever a row gets inserted into by db my db triggers http to my java app .this app puts the changes in db as messages into activemq(we have written this thing as we are not experts in java spring frame work))
any solutions or suggestions in this regard is much appreciated
What you are looking for is indeed persistency:
Persistent messaging (ensures the messages are stored in a datastore until the broker receives the acknowledgement that it has been delivered successfully to all consumers)
This will ensure the messages are re-sent (automatically) once the broker is back alive.
If you want redundancy, you should look for the master/slave topology
Related
I am working on application that writes to Kafka queue which is read by other application. When I am unable to send message on Kafka due to network or other reason, I need to write messages during Kafka down time to other place e.g Oracle or local file system, so that I don't loose messages generated during down time.Problem with oracle or other DB is it too can go down. Is there any recommendations about how could I achieve fail safe during Kafka down time.
Number of messages generated are approx 20-25 million per day. For messages stored during downtime I am planning to have batch job to re send them to destination application once target application is up again.
Thank you
You can push those messages into a cloud based messaging service like SQS. It supports around 3K messages per second.
There is also a connector that allows you to push back the messages into Kafka directly, with no other headaches.
If you can't export the data out of your local network, then maybe a cluster of RabbitMQ instances may help, although it wouldn't be a plug & play solution.
I have web service that handles requests and publishes messages to RabbitMQ (Written in spring boot).
The problem is when there is no connection I cannot detect it immediately and I am losing all my messages. How can I deal with this problem? I got AMQP Connection error after 30 seconds. In that interval I cannot handle this problem. I want to know when message delivered to RabbitMQ. If it is not delivered I need to store all messages and when rabbitmq is up resend all these messages. By the way, performance is important.
I have read documentations below. I think it could be solved with OperationsCallback but I dont know how..
https://docs.spring.io/spring-amqp/docs/current/reference/html/#scoped-operations
https://docs.spring.io/spring-amqp/docs/current/reference/html/#cf-pub-conf-ret
Thanks in Advance
its rather a rabbit MQ feature more than java or spring, which allows any message that is published to be acknowledged back to the publisher, see the below link for references on how:
https://www.rabbitmq.com/confirms.html
Does ZooKeeper have an "in-memory log"? I've some experience with ZooKeeper but I never seen anything like it before (from the client side), and after searching I haven't found anything related to an in-memory log. As far as I know, every operation (create, setData, delete) is made on disk.
However, in the paper Ravana: Controller Fault-Tolerance in Software-Defined Networking the authors
Event logging: The master saves each event in ZooKeeper’s distributed
in-memory log. Slaves monitor the log by registering a
trigger for it. When a new event is propagated to a slave’s log, the
trigger is activated so that the slave can read the newly arrived event
locally.
So, assuming that there is an in-memory log, how would a (java) client app use it? Or is it server side only?
I understand JMS as depicted by the following diagram:
(source: techhive.com)
Is there any way for me to access the underlying database using JMS or some other thing? Further, the JDBC connections that the JMS server maintains, can I add new connections in it so as to access other databases also and do CRUD operations on them? If yes, how?
Where did you get this from?
Normally JMS is used to send messages to queue (or topics). You have message producers that push messages in the queue and message consumers consume them and process it.
In your exemple it seems that you have multiple queues. One for the messages that need to be processed, and one for each client to retrieve the result the processing of its messages.
With JMS Server you don't necessarily have a database behind. Everything can stay in memory, or can be written to files. You will need database server behind only if you configure your JMS server to be persistent (and to assure that even if server/application crash your messages won't be lost). But in that case you will never have to interact with the database. Only the JMS server will and you will interact with the JMS server sending and consuming messages.
I will be publishing to a single Activemq Topic and I will have many subscribers consuming from this Activemq. Some of my subscribers may connect at a later date, but when they do I want them to receive ALL MESSAGES ever published to that Activemq topic. How do I do this and what is this pub-sub type called where you get a full picture on first subscribe?
It's typically a lot better to create a separate initial load service. New clients connecting and wanting years of missed updates can trigger some sync from the source application and receive these message through some other channel (a queue for instance). Once up to sync, you simply use durable subscribers on your topic to guarantee that you miss no further updates.
ActiveMQ is not really built to store huge amount of data in the middle for long term. Kahadb is not like a regular database (although you can back it with a JDBC data source if you wish). Storing messages long term in MOM software is actually an anti-pattern.