I currently use ZeroMQ with Java binding. My program is in a PUB/SUB mode.
It seems reasonable to set a timeout, while a client can't receive a message from PUB-side.
But for the publish server, who sends messages without a fixed frequency, it's hard to decide a reasonable timeout.
On the other hand, if no timeout is set, then a program would probably stuck at function:
recv()
forever even publish server is dead.
If there a good solution to fix this issue?
Yes, there is a good solution or two:
A principally best solution is to use a Poller instance, for which a .poll() method tells your code, again with a help of an explicitly configurable aTimeOut in [ms], whether there is any incoming message to either attempt a non-blocking .recv() method or not even trying to do so, once there is NACK for any such message being present from the call to a .poll() method ( for details check the API ).
Another way is to use a non-blocking mode of a call to the .recv( aSockINST, ZMQ_DONTWAIT ) method. Here, the API + wrapper / binding, specify how it handles a state, where none such message was locally ready to get .recv()-ed, so that one may rely on the common language's available syntax-scaffolding - like { try: except: finally: } or { try: catch: } or { defer; panic(); recover() } - to handle either of the return states from the .recv( .., ZMQ_DONTWAIT ) call. Similar rules apply to an ( almost ) blocking call, with some moderately small .recv() timeout.
You can use Pollers:
poller = zmq.Poller()
poller.register(client_receiver, zmq.POLLIN);
for further reading:
http://learning-0mq-with-pyzmq.readthedocs.io/en/latest/pyzmq/multisocket/zmqpoller.html
hope it helps.
Related
Given 2 applications where
application A is using a publisher client to contentiously stream data to application B which has a sub server socket to accept that data, how can we configure pub client socket in application A such that when B is being unavailable (like its being redeployed, restarted) A buffers all the pending messages and when B becomes available buffered messages go trough and socket catches up with real time stream?
In a nutshell, how do we make PUB CLIENT socket buffer messages with some limit while SUB SERVER is unavailable?
The default behaviour for PUB client is to drop in mute state, but it would be great if we could change that to a limit sized buffer, is it possible with zmq? or do i need to do it on application level...
I've tried setting HWM and LINGER in my sockets, but if i'm not wrong they are only responsible for slow consumer case, where my publisher is connected to subscriber, but subscriber is so slow that publisher starts to buffer messages (hwm will limit number of those messages)...
I'm using jeromq since i'm targeting jvm platform.
First of all, welcome to the world of Zen-of-Zero, where latency matters most
PROLOGUE :
ZeroMQ was designed by a Pieter HINTJENS' team of ultimately experienced masters - Martin SUSTRIK to be named first. The design was professionally crafted so as to avoid any unnecessary latency. So asking about having a (limited) persistence? No, sir, not confirmed - PUB/SUB Scalable Formal Communication Pattern Archetype will not have it built-in, right because of the added problems and decreased performance and scalability ( add-on latency, add-on processing, add-on memory-management ).
If one needs a (limited) persistence (for absent remote-SUB-side agent(s)' connections ), feel free to implement it on the app-side, or one may design and implement a new ZMTP-compliant such behaviour-pattern Archetype, extending the ZeroMQ framework, if such work goes into stable and publicly accepted state, but do not request the high-performance, latency-shaved standard PUB/SUB having polished the almost linear scalability ad astra, to get modified in this direction. It is definitely not a way to go.
Solution ?
App-side may easily implement your added logic, using dual-pointer circular buffers, working in a sort-of (app-side-managed)-Persistence-PROXY, yet in-front-of the PUB-sender.
Your design may get successful in squeezing some additional sauce from the ZeroMQ internal details in case your design also enjoys to use the recently made available built-in ZeroMQ-socket_monitor-component to setup an additional control-layer and receive there a stream of events as seen from "inside" the PUB-side Context-instance, where some additional network and connection-management related events may bring more light into your (app-side-managed)-Persistence-PROXY
Yet, be warned that
The _zmq_socket_monitor()_ method supports only connection-oriented
transports, that is, TCP, IPC, and TIPC.
so one may straight forget about this in case any of the ultimately interesting transport-classes was planned to be used { inproc:// | norm:// | pgm:// | epgm:// | vmci:// }
Heads up !
There are inaccurate, if not wrong, pieces of information from our Community honorable member smac89, who tried his best to address your additional interest expressed in the comment:
"...zmq optimizes publishing on topics? like if you keep publishing on some 100char long topic rapidly, is it actually sending the topic every time or it maps to some int and sends the int subsequently...?"
telling you:
"It will always publish the topic. When I use the pub-sub pattern, I usually publish the topic first and then the actual message, so in the subscriber I just read the first frame and ignore it and then read the actual message"
ZeroMQ does not work this way. There is nothing as a "separate" <topic> followed by a <message-body>, but rather the opposite
The TOPIC and the mechanisation of topic-filtering works in a very different way.
1) you never know, who .connect()-s:i.e. one can be almost sure the version 2.x till version 4.2+ will handle the topic-filtering in different manner ( ZMTP:RFC defines intial capability-version handshaking, to let the Context-instance decide, which version of topic-filtering will have to be used: ver 2.x used to move all messages to all peers, and let all the SUB-sides ( of ver 2.x+ ) be delivered the message ( and let the SUB-side Context-instance process the local topic-list filter processing )whereasver 4.2+ are sure to perform the topic-list filter processing on **the PUB-side Context-instance (CPU-usage grows, network-transport the opposite ), so your SUB-side will never be delivered a byte of "useless" read "not-subscribed" to messages.
2) (you may, but) there is no need to separate a "topic" into a first-frame of a thus-implied multi-frame message. Perhaps just the opposite ( it is a rather anti-pattern to do this in high performance, low-latecy distributed system design.
Topic filtering process is defined and works byte-wise, from left-to-right, pattern matching for each of the topic-list member value agains the delivered message payload.
Adding extra data, extra frame-management processing just and only does increase the end-to-end latency and processing overhead. Never a good idea to do this instead of proper distributed-system design work.
EPILOGUE :
There are no easy wins nor any low-hanging fruit in professional distributed-systems design, the less if low-latency or ultra-low-latency are the design targets.
On the other hand, be sure that ZeroMQ framework was made with this in mind and these efforts were crowned with stable, ultimately performant well-balanced set of tools for smart (by design), fast (in operation) and scalable (as hell may envy) signaling/messaging services people love to use right because of this design wisdom.
Wish you live happy with ZeroMQ as it is and feel free to add any additional set of features "in front" of the ZeroMQ layer, inside your application suite of choice.
I'm posting a quick update since the other two answers (though very informative were actually wrong), and i dont want others to be misinformed from my accepted answer. Not only you can do this with zmq, it is actually the default behaviour.
The trick is that if you publisher client never connected to the subscriber server before it keeps dropping messages (and that is why i was thinking it does not buffer messages), but if your publisher connects to subscriber and you restart subscriber, publisher will buffer messages until HWM is reached which is exactly what i asked for... so in short publisher wants to know there is someone on the other end accepting messages only after that it will buffer messages...
Here is some sample code which demonstrates this (you might need to do some basic edits to compile it).
I used this dependency only org.zeromq:jeromq:0.5.1.
zmq-publisher.kt
fun main() {
val uri = "tcp://localhost:3006"
val context = ZContext(1)
val socket = context.createSocket(SocketType.PUB)
socket.hwm = 10000
socket.linger = 0
"connecting to $uri".log()
socket.connect(uri)
fun publish(path: String, msg: Msg) {
">> $path | ${msg.json()}".log()
socket.sendMore(path)
socket.send(msg.toByteArray())
}
var count = 0
while (notInterrupted()) {
val msg = telegramMessage("message : ${++count}")
publish("/some/feed", msg)
println()
sleepInterruptible(1.second)
}
}
and of course zmq-subscriber.kt
fun main() {
val uri = "tcp://localhost:3006"
val context = ZContext(1)
val socket = context.createSocket(SocketType.SUB)
socket.hwm = 10000
socket.receiveTimeOut = 250
"connecting to $uri".log()
socket.bind(uri)
socket.subscribe("/some/feed")
while (true) {
val path = socket.recvStr() ?: continue
val bytes = socket.recv()
val msg = Msg.parseFrom(bytes)
"<< $path | ${msg.json()}".log()
}
}
Try running publisher first without subscriber, then when you launch subscriber you missed all the messages so far... now without restarting publisher, stop subscriber wait for some time and start it again.
Here is an example of one of my services actually benefiting from this...
This is the structure [current service]sub:server <= pub:client[service being restarted]sub:server <=* pub:client[multiple publishers]
Because i restart the service in the middle, all the publishers start buffering their messages, the final service that was observing ~200 messages per second observes drop to 0 (those 1 or 2 are heartbeats) then sudden burst of 1000+ messages come in, because all publishers flushed their buffers (restart took about 5 seconds)... I am actually not loosing a single message here...
Note that you must have subscriber:server <= publisher:client pair (this way publisher knows "there is only one place i need to deliver these messages to" (you can try binding on publisher and connecting on subscriber but you will not see publisher buffering messages anymore simply because its questionable if subscriber that just disconnected did it because it no longer needs the data or because it failed)
As we've discussed in the comments there is no way for the publisher to buffer messages while having nothing connected to it, it will simply drop any new messages:
From the docs:
If a publisher has no connected subscribers, then it will simply drop all messages.
This means your buffer needs to be outside of zeromq's care. Your buffer could then be a list, or a database, or any other method of storage you choose, but you cannot use your publisher for doing that.
Now the next problem is dealing with how to detect that a subscriber has connected/disconnected. This is needed to tell us when we need to start reading from the buffer/filling the buffer.
I suggest using Socket.monitor and listening for the ZMQ_EVENT_CONNECTED and ZMQ_EVENT_DISCONNECTED, as these will tell you when a client has connected/disconnected and thus enable you to switching to filling your buffer of choice. Of course, there might be other ways of doing this that does not directly involve zeromq, but that's up to you to decide.
I'm setting up an ActiveMQ broker with MQTT that uses an external service for user authentication.
I figured out how to write a BrokerFilter and plug it into a broker, so the basics are covered.
I could even limit users in the topics they are allowed to subscribe on using the addConsumer() override. That method looks like this, and works:
override fun addConsumer(context: ConnectionContext?, info: ConsumerInfo?): Subscription {
// let connections from the local vm through
return if (isLocalConnection(context!!)) {
super.addConsumer(context, info)
} else {
val username = context?.userName ?: ""
val cameraCode = getTopicElementsFromDestination(info!!.destination).first()
assertUserHasPermissionForCamera(username, cameraCode)
super.addConsumer(context, info)
}
}
So I thought that restricting publishing would work pretty much the same with an override of addProducer(), but I stumbled over a few problems.
The first problem was that subscriptions also called addProducer(), which was surprising to say the least. But the real surprise was that ConsumerInfo::destination is always null when this method is called. I've searched far and wide, but I'm unable to find a way to extract the topic being published to from the information passed to addProducer(). Without knowing which topic the producer wants to publish to, I obviously can't limit it.
So I tried overriding messageDelivered() instead, figuring I could just discard the message when it is published to the wrong topic, achieving more or less the same effect. According to the documentation, this method should be called whenever the broker receives a message. But it doesn't get called when I send a message to the broker, so either I'm misunderstanding what "a message delivered to the broker" means or something is fishy.
I also tried addSession(), which also doesn't get called.
So... How can I intercept when a client publishes to a certain topic?
After some hunting through the source code and a lot of overriding methods and seeing when they get called and what they receive, I learned two things that helped me do what I wanted:
The topic being published to is encapsulated in the message. A Producer doesn't register to publish to a certain topic, it just registers to publish to any topic. The broker itself doesn't know to what topic, that's handled on a message-by-message basis. So my first attempts to limit topics in addProducer() were futile, because nobody knows the topic yet.
The earliest point in the publish chain where both username and topic are available is addDestination(). The topic can first be known in send(), but you don't have the user. It might be possible to do the authorisation there by persisting the username for the context, but I don't like state. So I put the authorisation into addDestination(), and it works.
There's a potential caveat here, though. According to the documentation, addDestination() only gets called if a destination doesn't exist yet. All tests I could do in the limited time confirmed that this is always the case on a publish, even if somebody is subscribed to the topic. But it might be different if the producer maintains a persistent connection, So use this solution with care.
In ZeroMQ PUB/SUB mode tutorial, there is a .poll() method:
ZMQ.Poller items = new ZMQ.Poller (1);
while (!Thread.currentThread ().isInterrupted ()) {
byte[] message;
items.poll();
if (items.pollin(0)) {
message = subscriber.recv(0);
System.out.println("received message:" + message);
}
}
This method lets you check the status of a connection. But it seems that the .poll() method is "another" .recv() without timeout!?
Both of them would stick thread/program until data is received.
Using a .poll() just lets a program stuck at .poll() instead of .recv()!?
What do I miss here?
ZeroMQ has introduced two, very different mechanics here:
.recv() method:
The .recv( <aSocketAccessPointINSTANCE>, <ZMQ_DONTWAIT_FLAG> ) is a method, that may, but need not block on an attempt to receive aSocketAccessPoint-delivered message into the hands of the kind user for further processing.
As syntax enforces, coder has to specify, from which ZeroMQ Scalable Formal Communication Archetype's Access Point the method ought try to pick a next FIFO message or a part of a multi-part message.
ZeroMQ ensures atomic delivery of messages: peers shall receive either all message parts of a message or none at all. The total number of message parts is unlimited except by available memory.
An application that processes multi-part messages must use and further investigate the ZMQ_RCVMORE in a zmq_getsockopt(3) option after calling zmq_recvmsg() to determine, if there are further parts to receive.
.poll() method is different ( while it may "shortcut" some logic ):
Before one can use a .poll() method, there are some duties to setup first the details for the .Poller() instance behaviour.
ZeroMQ provides this as a mechanism for multiplexing more and many input/output events over a set of Access Points ( containing both ZeroMQ smart-sockets and ( in more recent API versions ) also standard, plain O/S-sockets. This mechanism mirrors the standard sockets' poll() system call.
Poller() instance can be instructed to .poll() one or several "local" Access Points to ask them about their internal state and can receive { zero | one | more }-answers, depending on the call-setup, actual state of the queried resources and whether a timeout has run off, before any specified event has arrived onto the "local" side of the listed Access Points at all.
The ZeroMQ original API defines for this:
int zmq_poll ( zmq_pollitem_t *items, int nitems, long timeout );
whereas the respective language bindings may re-wrap this API into some sort of a higher level helper method ( not to have manually declare how many records nitems one tries to pass in *items once the sizing of the MUX-events object is known at the runtime, before the low-level API is going to be called -- so re-check the ZeroMQ binding documentation for the exact syntax, exposed to user-code ).
As noted in the O/P, given a .poll() was called with the value of timeout == -1, having just a single Access Point in the *items, the .poll() shall block indefinitely until a requested event has occurred on the specified zmq_pollitem_t set for { ZMQ_POLLIN }, so here, the blocking the user-code effectively mirrors what would .recv() do at that very place. Yet, the respective mechanics are way different.
recv() waits for messages from just 1 ZeroMQ socket, while poll() lets you wait for messages from many ZeroMQ sockets.
The Poller also lets you easily specify a timeout when waiting for messages.
Note, your code seems to miss the needed calls to
items.register( subscriber, ZMQ.Poller.POLLIN );
MessageConsumer.receive(long) allows specifying a timeout on the receive.
However, I have found that if you make the producer go away (e.g. by killing the process), the receive on the consumer doesn't always time out. In fact, sometimes it waits forever and the process never exits.
It turns out that I can add an ExceptionListener to the connection, but this has some drawbacks:
(1) I might get errors about things other than the operation I was trying to perform;
(2) receive() itself still won't necessarily complete;
(3) Apparently we used to use an ExceptionListener and it was causing some other issues which I don't know the specifics of.
One potential hack which looks viable for us (we're using ActiveMQ) is that when the stall occurs, the thread will be in wait() (this is almost certainly implementation-specific but it seems like it would be a common choice.) So if we got an exception, we could interrupt the thread doing the receive() calls.
Is this a sane solution? Is there a more reliable, well-known, commonly-implemented workaround for this issue?
Instead of using receive, you could setup a message listener. This seems like a bug in ActiveMQ, have you reported it as an issue in their JIRA?
In my program, I am essentially trying to connect to a publisher and get data. The basic functionality is there in these steps
I make the connection to the publisher with username and password etc
I make the request for data. Method exits
The publisher's API gives me a callback to a method onDataUpdate(Object theUpdate)
From there, I can print the data, or write it to a database or anything I need to do. That all works.
My problem is, I would now like to wrap the functionality in such a way that a calling program can say request the data and receive it as soon as I have it. Meaning, I want my exposed method to look like
public Object getData() {
subscribeForData();
// somehow wait
return theUpdate;
}
How can I make this happen? Is there some way I can use threads to wait/notify when I've received the update? I'm a newb to stackoverflow and also multithreaded programming, so any help and sample code would be much appreciated!! Thanks in advance.
In this case I would prefer to use CountDownLatch, where i'll initialize my lathch with count 1 as soon i subscribe for publisher i will call await() on latch and when i get the callback i'll countdown the latch.
Use a SynchronousQueue. Create it in getData, call put() in the callback method, then call take() in the original thread at the end of getData().
Check out CompletionService, especially ExecutorCompletionService. There is a nice example of a web page loader/renderer in the book Java Concurrency in Practice.
I'm not entirely certain about your question but I'll give it a shot - hope it helps :)
You could use a blockingqueue in java for this purpose (producer consumer message) - if you write to the queue when the callback gets invoked - from another thread, you could read from the queue. Blocking queues are thread safe (but may not fit your requirements).
You could also look into readwrite locks if you only have one thread writing to a collection and perhaps multiple readers (or even just on reader).
You could also look into the observer pattern - for reference: http://www.vogella.com/articles/DesignPatternObserver/article.html
If neither of those work, one could look into using a queue/topic from an in-VM messaging server such as ZeroMQ/ActiveMQ or perhaps something like Redis/HazelCast.
Hope it helps and good luck
Converting a asynchronous call to a synchronous one is an interesting exercise, I use it often in interviews (and the reverse, wrapping a synchronous call in asynchronous).
So there is a requestData method that is going to return immediately and it (or something else) will later call onDataUpdate in a different thread. You want to create a new method, say requestDataSynchronous that does not require the caller to use a callback but instead blocks till data is available and returns it to the caller.
So what you need for requestDataSynchronous to do is:
call requestData
wait till onDataUpdate is called (in a different thread)
get the data onDataUpdate received
return it to the caller
Of the above, #2 and #3 have to be done by some mode of inter-thread-communication. You can use wait/notifiy but it might be much simpler to use a BlockingQueue. onDataUpdate writes to it once data is available, and requestDataSynchronous reads from it, blocking on the read until onDataUpdate writes into it.
Using ExecutorService might make this even easier, but it will be useful to know what's going on.