I'm trying to get a basic implementation of a ZMQ publisher and subscriber working, but it's failing silently. I'm using JeroMQ 0.5.2 (the current version) and Java 8.
Consider the following official test file:
https://github.com/zeromq/jeromq/blob/master/src/test/java/org/zeromq/PubSubTest.java
I've copied the first test (testRaceConditionIssue322) in its entirety into a new main class and run it. The publisher binds to the port, and claims to send every message, but the receiver does not receive a single message. Adding logs indicates that the subscriber believes itself to be subscribed before the publisher sends messages.
I've tried this on two computers, as well as with different code, and it's the same net result each time. What gives?
Ok, I figured it out. Two things converged in an unfortunate way.
The test I linked was, possibly on purpose, starting the subscriber before the publisher. For some reason, the subscriber reported a successful connection, even though the publisher had not yet opened the port. It did not make a connection, and did not receive the messages subsequently sent. When I made sure the publisher was bound and listening for connections, and then the subscriber connected, and then the publisher published messages, it worked how I as expecting.
The OTHER code I was using, as a subscriber, had a line in it I didn't notice - socket.hasReceiveMore(). It was expecting two strings in one message, but I was sending two strings separately. This meant part of the receiver code never executed - it received the strings I was sending, but discarded them as partial messages. When I sent my first string with the flag publisher.send(msg, ZMQ.SNDMORE); (and the second without), it worked as I expected.
Related
Our app uses Spring Boot and JMS messages with Tibco. We have two production servers running and processing messages concurrently. Servers are listening to the same one queue. Each server has 10 concurrent listeners. I do not want the very same message gets processed by both servers at the same time. Nothing prevents our queue of having duplicate messages, like we can have two copies of the message A in the queue. If messages in the queue are: A, A, B, C, D, then if first A gets delivered to server1 and second A gets delivered to server2, and both servers process A at the same time, then they are chances of creating duplicate entities. I want to find a way to send all A messages to only one server. I can't use Message Selector b/c we have the same code base running on both servers. This is what I'm considering:
Based on the message, set properties in the headers. Once the message got delivered to the process() method, depending on which server is processing the message, either discard, simply return the message or process the message and acknowledge it. The problem with this solution is that since we need to dynamicacaly find out which server is processing the message, the server name needs to be hardcoded, meaning if the server moves, the code breaks!
Other solution - that might work - is the Destination field.
https://docs.spring.io/spring/docs/4.0.x/spring-framework-reference/html/jms.html
Destinations, like ConnectionFactories, are JMS administered objects
that can be stored and retrieved in JNDI. When configuring a Spring
application context you can use the JNDI factory class
JndiObjectFactoryBean / to perform dependency
injection on your object’s references to JMS destinations.
It's something I never done before. Is there anyway, to configure the Destination that it picks up the right server to route the message to? Meaning, if message1 is supposed to be delivered to server1, then it does not even gets delivered to server2 and remains in the queue until server1 consumes it?
What are other ways to implement this?
EDIT:
I still do not know what’s the best way to send certain messages to only one server for processing, however, accepted the response given to use database as validation, b/c this is what we consider to avoid creating duplicate entities when processing the data.
I think the idea of using the JMS Destination is a non-starter as there is nothing in the JMS specification which guarantees any kind of link between the destination and a broker. The destination is just an encapsulation for the provider-specific queue/topic name.
The bottom line here is that you either need to prevent the duplicate messages in the first place or have some way to coordinate the consumers to deal with the duplicates after they've been pulled off the queue. I think you could do either of these using an external system like a database, e.g.:
When producing the message check the database for an indication that the message was sent already. If no indication is found then write a record to the database (will need to use a primary key to prevent duplicates) and send the message. Otherwise don't send the message.
When consuming the message check the database for an indication that the message is being (or was) consumed already. If no indication is found then write a record to the database (will need to use a primary key to prevent duplicates) and process the message. Otherwise just acknowledge the message without processing it.
I suggest an alternative to "post DB sync".
Keep the servers and listeners as-is, and broadcast all+ the the processed messages on a topic. For servers just starting, you can use "durable subscribers" to not miss any messages.
If you broadcast each start and end of processing for messages A, B, C, etc AND consider adding a little pause (in milli), you should avoid collisions. It's the main risk of course.
It's not clear to me if you should validate for duplicate processing at the beginning or end of a message processing... it depends on your needs.
If this whole idea is not acceptable, DB validation might be the only option, but as stated in comments above, I fear for scaling.
I've seen a lot of code and examples on how to send a message from the server/producer to the client, but I'm lost as to how to send a message from the client to the server. Does it follow the same format?
For example, say my server has 5 topics: A, B, C, D, E. My client is listening to only topic C.
So my server creates a queue and sends messages to subscribers/clients.
My client gets messages that are on topic C. It performs some computation, and I want to send a message back to the publisher (say an int for example). What are the required steps?
Do I need to set a destination/topic for the producer to listen to?
Does my producer automatically listen to all clients (once they connect)?
I think you may need to do a bit of reading around the various communication models available e.g. this on JMS Models.
You have started by using Topics, but your statements hint at a One-to-One relationship between sender and receiver. For this queues may be more appropriate. Topics are generally for publishing information where the publisher doesn't know (nor care) about the subscribers - and there can be many.
On the other hand Queuing is intended to deliver to a specific target. With this you can still do "fire-and-forget" (where the sender puts the message out to a queue and doesn't wait for the expected consumer to process it)
Or you can add response handling to that - essentially by adding a "parallel" response queue (where the initial sender will processes the response messages back from the consumer - usually in a different thread).
ActiveMQ and JMS have "reply-to" functionality - where the sender indicates on send the queue name where it wants to see responses come back to - and the consumer complies with that. The reply-to queue can be fixed or a temporary one that the sender creates up-front.
All depends on what model you are going for.
The question is already posted, Mqtt How a client can get to know that another client is connected or not
and How to Find Connected MQTT Client Details
In my case, if client X is already subscribed in a channel A, client Y can't subcribe to the channel A, until X unsubcribes. I can only have one client subscribed in the channel
Can I also use the idea of retained messages and LWT?
If yes, I don't know exactly from where should I start. It would be good to start with a simple example to see how the retained messages and LWT work. So far, I just have experience in publishing and subscribing but no more.
Could you please, tell me some advises may be some links or examples or any useful information so I can have a starting point.
MQTT is all about having multiple clients subscribing to the same topics, it's part of the whole pub/sub pattern and sharing information. So there is nothing baked into the protocol that will do what you want.
You may be able to implement something like the following:
If have a topic say foo/bar and you only want one subscriber you could publish a retained message with a payload of the client-id of the subscriber to lock/foo/bar. You could then publish a "free" to this lock topic when you disconnected and set up a LWT to do the same in case the client dies.
The problem with this is that everything is asynchronous so it opens up lots of timing windows for race conditions. e.g. say client-1 and client-2 both want to subscribe to foo/bar, they would both need to first subscribe to lock/foo/bar to check it's state. They both do this at very nearly the same time, they then have to wait for some time to see what message they get back ("free" or a client-id). They would both get "free" so would both assume that they can publish their client-ids. client-1 published first shortly followed by client-2 and then they both subscribe to foo/bar.
I'm developing an Andoid App that is made of a Service running in Background and some Activities connected to that Service. The Service runs on it's own Process.
My Service mainly has 3 classes: ServiceMain, ServiceWorker,Message.
ServiceMain has all the functions that are used by the Activities like logIn,logOut,send ... and so on.
Message represents a message that is sent to our server or recieved. Simply a String and a bool where the String is the message and the bool is a flag saying if a response from server is needed.
ServiceWorker is a subclass of Thread and does all the sending an recieving of messages using Sockets.
ServiceMain contains 2 Queues:
Queue<Message> Sendingqueue= new ConcurrentLinkedQueue<Message>();
Queue<Message> Recievequeue = new ConcurrentLinkedQueue<Message>();
If the logIn method is called a ServiceWorker is created and started. In it's constructor it gets references to both queues and holds them.
private final Queue<Message> Sendingqueue;
private final Queue<Message> Recievequeue;
ServiceMain then creates some messages (M1,M2 for example) and adds them to the Sendingqueue.
The ServiceWorker builds the connection to our server and then runs into a loop where it looks for messages in Sendingqueue, sends them and doing some othe stuff like recieving ....
Hope the scenario is clear now.
Within ServiceWorker something strange happens on Sendingqueue:
Let's say ServiceMain added two messages, M1 and M2 to Sendingqueue while ServiceWorker is doing something time consuming or is not connected to our server.
Sendingqueue now contains two messages.
If the ServiceWorker next time gets the length of the Queue it sees 2 items. Ok so far.
Then it calls peek() (the message is removed only if it was successfull sent) on the Sendingqueue and should get M1 because it was added first.
But it gets M2.
The Sendingqueue seems to be reverted.
What's going wrong here ? What can I do to avoid this?
Thanks for any constructive reply.
Detlef
ConcurrentLinkedQueue doesn't make any guarantees about order, but the order of elements shouldn't change if you are adding to the end and taking from the start (or visa versa) thsi should work. You could run into a problem if you add and remove from the start or end as this will mean you are processing the newest rather than the oldest each time.
If you had a large powerful server, I would still suggest this approach is overkill. Instead of having a background thread to perform the processing, I would use the main thread.
Note: The socket is already an input and output queue on the client and on the server, so adding a third layer of queuing may be redundant in a large system and inefficient in a smaller device.
I am making an application that will work much like a real time chat. A user will be constantly writing on lets say a text area and messages will be send to other users. On the communications class I have set up a receiver. When a message from someone reaches the client, the receive method will be invoked and will get the message. What I can't understand is how the code will be executed. What happens if, while the user is typing/sending a message the receive message is invoked ? What do I need to do in order for this to work properly ?
Hope the question is clear enough.
ps : Im still in the design phase thats why I haven't tested it to see what happens.
Also atm I only use a second thread to receive messages which calls the receive method.
There should not be a problem at all.
When a message from someone reaches the client, the receive method
will be invoked and will get the message. What I can't understand is
how the code will be executed?
You should have a Receiver class that will encapsulate a socket (from which your receive data) and keep a set of listeners (see Observer pattern). A GUI can be one of the listeners. When a message is received via the socket, you need to notify all listeners by forwarding the data received. This way, you have a clean and nice way to notify the GUI about new messages arrivals.
What happens if, while the user is typing/sending a message the
receive message is invoked ?
This depends on the type of IP protocol you are using but in general your don't have to worry about this although I suggest you protect your sockets using lock mechanisms.
What do I need to do in order for this to work properly ?
Here is a nice example that can give you some inspiration :)
EDIT: As for your question regarding execution flow, sending and receiving are two different and uncorrelated operations that can happen at the same time. This can be achieved by implementing send and receive operations in two different threads. Here is an article on socket communications and multithreading.
You should either do what traditional Java EE app servers have done, which is assign a separate thread for processing each incoming message, or try a Java NIO solution along the lines of Netty.