Using HornetQ (In JBoss AS 6.0) I would like to setup a JMS topic to which multiple clients can subscribe.
A producer periodically sends a message to this topic with a reply-to destination, to which all subscribers should reply.
The problem I'm having is that I'm not entirely sure how to check that all subscribers have indeed replied.
One solution could be that each subscriber first sends a message to the topic after subscription with its details (perhaps some GUID). The producer remembers these details and uses it to check later whether all subscribed clients have replied.
However, rather than inventing the wheel myself I would like to use something that already exists. This seems like a standard problem, but I could not find any existing solution.
You could use durable subscriptions, and then query the subscriptions and messages.
See http://hornetq.sourceforge.net/docs/hornetq-2.0.0.BETA5/user-manual/en/html/management.html#d0e5742
Note that usage of durable subscriptions and persistent messages will incur a performance penalty. You'll have to gauge the severity of the performance impact according to your specific needs.
JMS itself doesn't support this, it's too simple. If you didn't mind coupling your code to HornetQ, then you could use its native API to find out this stuff. Not ideal, but it's well written and has readable source code, so it wouldn't be too hard.
Related
I am totally new to spring framework. I am trying to create a java maven project where I can have the connectivity to the rabbitMq and I even before publish the message, I want to check if the queues are alive or not. Is this possible to ping the queue to see if it a alive or not.. I am totally new to this rabbitMQ.
Thanks for the answers
Checking for the availability of a queue is a bit of an anti-pattern with messaging systems.
The message producer should not care if there is something on the other end to receive / process the message. The producer only cares that the RabbitMQ instance is available, with the correct exchange.
If the message must be delivered to a consumer, guaranteed, then the consumer needs to configure the queue with durability in mind and the producer should send the message with the persistence flag to ensure it is written to disk.
...
re-reading your question, i'm wondering if you mean "rabbitmq server" when you say "queue". are you wanting to check if the rabbitmq server is available?
if that is the case, the proper thing to do is use a heartbeat in your RabbitMQ connection. the spring framework should know how to do this, and should respond with some kind of event or other code that executes when the connection dies. i'm not really familiar with spring, though, so i don't know the details of doing that with this framework.
You might check this post or this RabbitMQ page on handling this.
I am considering using Apache Kafka as a distributed message publisher to many subscribers. It is the perfect fit for me, since the solution has to scale easily.
The Kafka's documentation states that the message may be acknowledged thus ensuring the message delivery. However, today I came across this article which states that there are scenarios in which the messages may be lost. Then again, the article is only available in Google cache, so I do not know whether it is trustworthy...
So I have one doubt - is there any moment, any scenario, in which the message will be lost? In another words - my main requirement is that each message must reach its destination. Can it be met by using the Apache Kafka? Is it the right tool for this job?
The original of the article you are looking for is here: http://engineering.onlive.com/2013/12/12/didnt-use-kafka/
If you read the full article and the comments you'll see much of the concern is not about the guarantee of at least once delivery, but that it was delivered AND successfully processed by the client. The last couple of comments on the article, including by the original author, seem to indicate he's satisfied with the approach.
You might also find this article of interest - similar concerns:
https://www.mail-archive.com/users%40kafka.apache.org/msg04492.html
And from some of the documentation:
So effectively Kafka guarantees at-least-once delivery by default and
allows the user to implement at most once delivery by disabling
retries on the producer and committing its offset prior to processing
a batch of messages. Exactly-once delivery requires co-operation with
the destination storage system but Kafka provides the offset which
makes implementing this straight-forward.
Most of the conversations I've seen are not about the guarantee of at least once, but how to go from there to at most once or to exactly once.
Kafka does claim that
For a topic with replication factor N, we will tolerate up to N-1 server failures without losing any messages committed to the log.
It might worth reading the Message Delivery Semantics written in their doc page for a better understanding
I will be publishing to a single Activemq Topic and I will have many subscribers consuming from this Activemq. Some of my subscribers may connect at a later date, but when they do I want them to receive ALL MESSAGES ever published to that Activemq topic. How do I do this and what is this pub-sub type called where you get a full picture on first subscribe?
It's typically a lot better to create a separate initial load service. New clients connecting and wanting years of missed updates can trigger some sync from the source application and receive these message through some other channel (a queue for instance). Once up to sync, you simply use durable subscribers on your topic to guarantee that you miss no further updates.
ActiveMQ is not really built to store huge amount of data in the middle for long term. Kahadb is not like a regular database (although you can back it with a JDBC data source if you wish). Storing messages long term in MOM software is actually an anti-pattern.
We have RV messaging systems publishing and receiving messages.Recently some underlying jars were upgraded - these are serialization jars used by all publishers and subscribers. However , it seems that some of the publishers are still referencing old versions of the serialization jars and therefore the receivers fail when trying to deserialize received messages.
Obviously restarting these publisher services should fix the problem. However , how do I identify all publishers using a particular topic to send messages to ? There must be some RV admin way of listing all the processes that are publishing to a given topic ?
I just gave a similar answer on another question:
There is a really great tool for this called Rai Insight
Basically what it can do is to sit on a box and silently listen all the multicast data and represent statistics even in real time. We used it to monitor traffic flow spikes with just few seconds delay.
It can give you traffic statistics braked down by multicast group, service number or even sending machine. Traffic flow peak/average, retransmission rate peak/average. All you can think of.
It will also give you per-service per-topic information.
We are running a high throughput system that utilizes tibco-ems JMS to pass large numbers of messages to and from our main server to our client connections. We've done some statistics and have determined that JMS is the causing a lot of latency. How can we make tibco JMS more performant? Are there any resources that give a good discussion on this topic.
Using non-persistent messages is one option if you don't need persistence.
Note that even if you do need persistence, sometimes it's better to use non persistent messages, and in case of a crash perform a different recovery action (like resending all messages)
This is relevant if:
crashes are rare (as the recovery takes time)
you can easily detect a crash
you can handle duplicate messages (you may not know exactly which messages were delivered before the crash
EMS also provides some mechanisms that are persistent, but less bullet proof then classic guaranteed delivery
these include:
instead of "exactly once" message delivery you can use "at least once" or "up to once" delivery.
you may use the pre-fetch mechanism which causes the client to fetch messages to memory before your application request them.
EMS should not be the bottle neck. I've done testing and we have gotten a shitload of throughput on our server.
You need to try to determine where the bottle neck is. Is the problem in the producer of the message or the consumer. Are messages piling up on the queue.
What type of scenario are you doing.
Pub/sup or request reply?
are you having temporary queue pile up. Too many temporary queues can cause performance issues. (Mostly when they linger because you didn't close something properly)
Are you publishing to a topic with durable subscribers if so. Try bridging the topic to queue and reading from those. Durable subscribers can cause a little hiccup in performance too since it needs to track who has copies of all messages.
Ensure that your sending process has one session and multiple calls through that session. Don't open a complete session for each operation. Re-use where possible. Do the same for the consumer.
make sure you CLOSE when you are done. EMS doesn't clear things up. So if you make a connection and just close your app the connection still is there and sucking up resources.
review your tolerance for lost messages in the even of a crash. If you are doing Client ack and it doesn't matter if you crash processing the message then switch to auto. Also I believe if you are using (TEMS - Tibco EMS for WCF) there's a problem with the session acknowledge. So a message is only when its processed on the whole message, we switched from Client ACK to the one that had Dups ok and it worked better)