The question is already posted, Mqtt How a client can get to know that another client is connected or not
and How to Find Connected MQTT Client Details
In my case, if client X is already subscribed in a channel A, client Y can't subcribe to the channel A, until X unsubcribes. I can only have one client subscribed in the channel
Can I also use the idea of retained messages and LWT?
If yes, I don't know exactly from where should I start. It would be good to start with a simple example to see how the retained messages and LWT work. So far, I just have experience in publishing and subscribing but no more.
Could you please, tell me some advises may be some links or examples or any useful information so I can have a starting point.
MQTT is all about having multiple clients subscribing to the same topics, it's part of the whole pub/sub pattern and sharing information. So there is nothing baked into the protocol that will do what you want.
You may be able to implement something like the following:
If have a topic say foo/bar and you only want one subscriber you could publish a retained message with a payload of the client-id of the subscriber to lock/foo/bar. You could then publish a "free" to this lock topic when you disconnected and set up a LWT to do the same in case the client dies.
The problem with this is that everything is asynchronous so it opens up lots of timing windows for race conditions. e.g. say client-1 and client-2 both want to subscribe to foo/bar, they would both need to first subscribe to lock/foo/bar to check it's state. They both do this at very nearly the same time, they then have to wait for some time to see what message they get back ("free" or a client-id). They would both get "free" so would both assume that they can publish their client-ids. client-1 published first shortly followed by client-2 and then they both subscribe to foo/bar.
Related
I'm trying to get a basic implementation of a ZMQ publisher and subscriber working, but it's failing silently. I'm using JeroMQ 0.5.2 (the current version) and Java 8.
Consider the following official test file:
https://github.com/zeromq/jeromq/blob/master/src/test/java/org/zeromq/PubSubTest.java
I've copied the first test (testRaceConditionIssue322) in its entirety into a new main class and run it. The publisher binds to the port, and claims to send every message, but the receiver does not receive a single message. Adding logs indicates that the subscriber believes itself to be subscribed before the publisher sends messages.
I've tried this on two computers, as well as with different code, and it's the same net result each time. What gives?
Ok, I figured it out. Two things converged in an unfortunate way.
The test I linked was, possibly on purpose, starting the subscriber before the publisher. For some reason, the subscriber reported a successful connection, even though the publisher had not yet opened the port. It did not make a connection, and did not receive the messages subsequently sent. When I made sure the publisher was bound and listening for connections, and then the subscriber connected, and then the publisher published messages, it worked how I as expecting.
The OTHER code I was using, as a subscriber, had a line in it I didn't notice - socket.hasReceiveMore(). It was expecting two strings in one message, but I was sending two strings separately. This meant part of the receiver code never executed - it received the strings I was sending, but discarded them as partial messages. When I sent my first string with the flag publisher.send(msg, ZMQ.SNDMORE); (and the second without), it worked as I expected.
I'm setting up an ActiveMQ broker with MQTT that uses an external service for user authentication.
I figured out how to write a BrokerFilter and plug it into a broker, so the basics are covered.
I could even limit users in the topics they are allowed to subscribe on using the addConsumer() override. That method looks like this, and works:
override fun addConsumer(context: ConnectionContext?, info: ConsumerInfo?): Subscription {
// let connections from the local vm through
return if (isLocalConnection(context!!)) {
super.addConsumer(context, info)
} else {
val username = context?.userName ?: ""
val cameraCode = getTopicElementsFromDestination(info!!.destination).first()
assertUserHasPermissionForCamera(username, cameraCode)
super.addConsumer(context, info)
}
}
So I thought that restricting publishing would work pretty much the same with an override of addProducer(), but I stumbled over a few problems.
The first problem was that subscriptions also called addProducer(), which was surprising to say the least. But the real surprise was that ConsumerInfo::destination is always null when this method is called. I've searched far and wide, but I'm unable to find a way to extract the topic being published to from the information passed to addProducer(). Without knowing which topic the producer wants to publish to, I obviously can't limit it.
So I tried overriding messageDelivered() instead, figuring I could just discard the message when it is published to the wrong topic, achieving more or less the same effect. According to the documentation, this method should be called whenever the broker receives a message. But it doesn't get called when I send a message to the broker, so either I'm misunderstanding what "a message delivered to the broker" means or something is fishy.
I also tried addSession(), which also doesn't get called.
So... How can I intercept when a client publishes to a certain topic?
After some hunting through the source code and a lot of overriding methods and seeing when they get called and what they receive, I learned two things that helped me do what I wanted:
The topic being published to is encapsulated in the message. A Producer doesn't register to publish to a certain topic, it just registers to publish to any topic. The broker itself doesn't know to what topic, that's handled on a message-by-message basis. So my first attempts to limit topics in addProducer() were futile, because nobody knows the topic yet.
The earliest point in the publish chain where both username and topic are available is addDestination(). The topic can first be known in send(), but you don't have the user. It might be possible to do the authorisation there by persisting the username for the context, but I don't like state. So I put the authorisation into addDestination(), and it works.
There's a potential caveat here, though. According to the documentation, addDestination() only gets called if a destination doesn't exist yet. All tests I could do in the limited time confirmed that this is always the case on a publish, even if somebody is subscribed to the topic. But it might be different if the producer maintains a persistent connection, So use this solution with care.
I've seen a lot of code and examples on how to send a message from the server/producer to the client, but I'm lost as to how to send a message from the client to the server. Does it follow the same format?
For example, say my server has 5 topics: A, B, C, D, E. My client is listening to only topic C.
So my server creates a queue and sends messages to subscribers/clients.
My client gets messages that are on topic C. It performs some computation, and I want to send a message back to the publisher (say an int for example). What are the required steps?
Do I need to set a destination/topic for the producer to listen to?
Does my producer automatically listen to all clients (once they connect)?
I think you may need to do a bit of reading around the various communication models available e.g. this on JMS Models.
You have started by using Topics, but your statements hint at a One-to-One relationship between sender and receiver. For this queues may be more appropriate. Topics are generally for publishing information where the publisher doesn't know (nor care) about the subscribers - and there can be many.
On the other hand Queuing is intended to deliver to a specific target. With this you can still do "fire-and-forget" (where the sender puts the message out to a queue and doesn't wait for the expected consumer to process it)
Or you can add response handling to that - essentially by adding a "parallel" response queue (where the initial sender will processes the response messages back from the consumer - usually in a different thread).
ActiveMQ and JMS have "reply-to" functionality - where the sender indicates on send the queue name where it wants to see responses come back to - and the consumer complies with that. The reply-to queue can be fixed or a temporary one that the sender creates up-front.
All depends on what model you are going for.
I am trying to understand the best use of RabbitMQ to satisfy the following problem.
As context I'm not concerned with performance in this use case (my peak TPS for this flow is 2 TPS) but I am concerned about resilience.
I have RabbitMQ installed in a cluster and ignoring dead letter queues the basic flow is I have a service receive a request, creates a persistent message which it queues, in a transaction, to a durable queue (at this point I'm happy the request is secured to disk). I then have another process listening for a message, which it reads (not using auto ack), does a bunch of stuff, writes a new message to a different exchange queue in a transaction (again now happy this message is secured to disk). Assuming the transaction completes successfully it manually acks the message back to the original consumer.
At this point my only failure scenario is is I have a failure between the commit of the transaction to write to my second queue and the return of the ack. This will lead to a message being potentially processed twice. Is there anything else I can do to plug this gap or do I have to figure out a way of handling duplicate messages.
As a final bit of context the services are written in java so using the java client libs.
Paul Fitz.
First of all, I suggest you to look a this guide here which has a lot of valid information on your topic.
From the RabbitMQ guide:
At the Producer
When using confirms, producers recovering from a channel or connection
failure should retransmit any messages for which an acknowledgement
has not been received from the broker. There is a possibility of
message duplication here, because the broker might have sent a
confirmation that never reached the producer (due to network failures,
etc). Therefore consumer applications will need to perform
deduplication or handle incoming messages in an idempotent manner.
At the Consumer
In the event of network failure (or a node crashing), messages can be
duplicated, and consumers must be prepared to handle them. If
possible, the simplest way to handle this is to ensure that your
consumers handle messages in an idempotent way rather than explicitly
deal with deduplication.
So, the point is that is not possibile in any way at all to guarantee that this "failure" scenario of yours will not happen. You will always have to deal with network failure, disk failure, put something here failure etc.
What you have to do here is to lean on the messaging architecture and implement if possibile "idempotency" of your messages (which means that even if you process the message twice is not going to happen anything wrong, check this).
If you can't than you should provide some kind of "processed message" list (for example you can use a guid inside every message) and check this list every time you receive a message; you can simply discard them in this case.
To be more "theorical", this post from Brave New Geek is very interesting:
Within the context of a distributed system, you cannot have
exactly-once message delivery.
Hope it helps :)
i'm working on an Java client-server application.
The client send a message sequence (the messages can be different types, i,ve got header), and listens for the replies. I've got 2 thread, one for the transmission and one for the receipts.
So i need to handle the replies, in case of errors or in case of the replies doesn't arrive, for example i can try to send the message another time.
My question is.. is there any java patterns that can helps me?
i would like to handle the send and the relative repliy like a single transaction, but note that i don't need to have a synchronous communication. I send all the message in the sequence in the TX thread and wait for the replies on the RX thread.
I've thought to the mediator Pattern, but i don't know if it is the right way.
Thanks
If the question is purely about transmission protocol I would take a look at NAK. http://en.wikipedia.org/wiki/NAK_(protocol_message)
I have implemented a protocol I made up called JCast that sends files over multicast. The files are broken down into small fragments that are numbered. The receiving clients then respond back with any missing fragments that it did not get (these are the NAK's). The server then would resend only the NAK'd fragments.
EDIT: The benefit of NAK over ACK is that the server can send all the packets it needs to without having to wait for ACK's. Since networks are very much improved nowadays, most of the packets would arrive. The few that do not arrive would then just be resent.