I am working in a desktop application, where application is deployed in both windows and mac platforms. As part of the application, it should communicate with native layer. Currently the communication between native layer and Java layer is done using sockets. Recently some one in the team suggested to use zeroMQ. Can any one of you guys please clarify my doubts.
How zeroMQ better than sockets
Is it possible to install zeroMQ library as part the Desktop client installation
I gone through the link 'https://github.com/zeromq/clrzmq4', it given libraries specific to amd64 and i386 processor family. Do I need to build it separately from the source code for different processors?
Do I still require .dll files to use zeroMQ in Java?
Do I require Visual studio to build zeroMQ libraries in windows (Since my native layer written in C#, my C# application communicate with zeroMQ socket socket written java)?
How zeroMQ better than sockets
http://zeromq.org/topics:omq-is-just-sockets
Is it possible to install zeroMQ library as part the Desktop client installation?
Yes, you need to build the libraries depends on the processor and embed them in your application.
Do I need to build it separately from the source code for different processors?
Yes, you need to build the libraries from source. zeroMQ is processor centric.
Do I still require .dll files to use zeroMQ in Java?
Yes, Following link may help you
Exception in thread "main" java.lang.UnsatisfiedLinkError: ... \jzmq.dll: Can't find dependent libraries
Do I require Visual studio to build zeroMQ libraries in windows?
Yes
This link may help you to get basic examples.
Regarding ZeroMQ in a desktop application on Windows talking to another process on the same machine, bear in mind that zmq_ipc is not supported (see zmq_ipc(7)). Or at least, that's the last I heard. This is because it's fundamentally impossible to implement anything like quite like select() or epoll() for named pipes in Windows. Just use zmq_tcp instead.
The same basic problem plagued the development of the select() implementation in Cygwin and its derivatives. They got round the problem by starting a thread for every non-socket file descriptor (i.e. named pipes, serial ports, etc) being selected, with each thread polling the HANDLE for whether any data had arrived (or whatever events were being set in select()). Not very efficient. Yeurk.
Proactor vs Reactor
Windows is proactor (which can do only proactor), everything else (*nix, VxWorks) is reactor (which can also be used to implement a proactor). The development of the boost.asio library for C++ was influenced by this, and is a proactor design as a result so that it could be run on Windows. RabbitMQ is proactor too.
ZeroMQ with zmq_poll() is reactor.
Proactor - you pro-actively start up an asynchronous routine to handle whatever turns up in the future.
Reactor - you react to the arrival of whatever has turned up by starting a synchronous call to whatever routine you wish to handle it knowing that it will complete very quickly because the data is already there.
The difference is key. In a proactor design, once you have started up that asynchronous routine to read a message, you cannot (easily) stop it or change it until it has done its thing. That is very annoying if you change your mind, for example as a result of reading some message from somewhere else.
Small caveat - Windows does support select() for network sockets (thus reactor programming is possible with network sockets, but only network sockets), and is the only reason why ZMQ is supported to any extent whatsoever on Windows.
Mixing ZMQ with Desktop Application Event Callbacks
Anyway, proactor means that Windows and C# fundamentally expects everything to be served by callbacks. This basically means you won't be using the zmq_poll() call to tell you if new messages have arrived if you also have callbacks handling GUI events. Instead you'd most likely be making asynchronous calls to zmq_revcmsg(). Trying to mix zmq_poll() in with callbacks is asking for trouble (you'd be blending proactor and reactor).
Message Formats
ZeroMQ and sockets both transfer bytes (as discrete messages with ZeroMQ, as a byte stream with sockets). One still has the challenge of deciding what the bytes mean to applications.
I can recommend using something like Google Protocol Buffers to serialise messages for transport by ZeroMQ. It's available for both C# and Java, but it doesn't demarcate message boundaries. Fortunately, ZeroMQ does. [Using GPB over a socket stream can be painful, you have to demarcate the message boundaries oneself]. So you can serialise a message to a buffer, hand the buffer over to ZeroMQ as a message, the recipient receives the message and knows for absolute certain that there is one single solitary GPB within. If you like you can use GPB's "oneof" to smuggle arbitrary message types across, which can be very liberating. You can accomplish the same with other serialisation technologies too of course, my personal favourite being ASN.1.
Related
I have the following situation:
I have 2 JVM processes (really 2 java processes running separately, not 2 threads) running on a local machine. Let's call them ProcessA an ProcessB.
I want them to communicate (exchange data) with one another (e.g. ProcessA sends a message to ProcessB to do something).
Now, I work around this issue by writing a temporary file and these process periodically scan this file to get message. I think this solution is not so good.
What would be a better alternative to achieve what I want?
Multiple options for IPC:
Socket-Based (Bare-Bones) Networking
not necessarily hard, but:
might be verbose for not much,
might offer more surface for bugs, as you write more code.
you could rely on existing frameworks, like Netty
RMI
Technically, that's also network communication, but that's transparent for you.
Fully-fledged Message Passing Architectures
usually built on either RMI or network communications as well, but with support for complicated conversations and workflows
might be too heavy-weight for something simple
frameworks like ActiveMQ or JBoss Messaging
Java Management Extensions (JMX)
more meant for JVM management and monitoring, but could help to implement what you want if you mostly want to have one process query another for data, or send it some request for an action, if they aren't too complex
also works over RMI (amongst other possible protocols)
not so simple to wrap your head around at first, but actually rather simple to use
File-sharing / File-locking
that's what you're doing right now
it's doable, but comes with a lot of problems to handle
Signals
You can simply send signals to your other project
However, it's fairly limited and requires you to implement a translation layer (it is doable, though, but a rather crazy idea to toy with than anything serious.
Without more details, a bare-bone network-based IPC approach seems the best, as it's the:
most extensible (in terms of adding new features and workflows to your
most lightweight (in terms of memory footprint for your app)
most simple (in terms of design)
most educative (in terms of learning how to implement IPC). (as you mentioned "socket is hard" in a comment, and it really is not and should be something you work on)
That being said, based on your example (simply requesting the other process to do an action), JMX could also be good enough for you.
I've added a library on github called Mappedbus (http://github.com/caplogic/mappedbus) which enable two (or many more) Java processes/JVMs to communicate by exchanging messages. The library uses a memory mapped file and makes use of fetch-and-add and volatile read/writes to synchronize the different readers and writers. I've measured the throughput between two processes using this library to 40 million messages/s with an average latency of 25 ns for reading/writing a single message.
What you are looking for is inter-process communication. Java provides a simple IPC framework in the form of Java RMI API. There are several other mechanisms for inter-process communication such as pipes, sockets, message queues (these are all concepts, obviously, so there are frameworks that implement these).
I think in your case Java RMI or a simple custom socket implementation should suffice.
Sockets with DataInput(Output)Stream, to send java objects back and forth. This is easier than using disk file, and much easier than Netty.
I tend to use jGroup to form local clusters between processes. It works for nodes (aka processes) on the same machine, within the same JVM or even across different servers.
Once you understand the basics it is easy working with it and having the options to actually run two or more processes in the same JVM makes it easy to test those processes easily.
The overhead and latency is minimal if both are on the same machine (usually only a TCP rountrip of about >100ns per action).
socket may be a better choice, I think.
Back in 2004 I implement code which do the job with sockets. Until then, many times I search for a better solution, because socket approach triggers firewall and my clients worry. There is no better solution until now. Client must serialize your data, send and server must receive and unserialize.
It is easy.
I'm working on a project, which I am not at liberty to discuss the core, but I have reached a stumbling block. I need data to be transferred from C++ to some other language, preferably Java or Python, in realtime (~10ms latency).
We have a sensor that HAS to be parsed in C++. We are planning on doing a data read/output through bluetooth, most likely Java or C# (I don't quite know C#, but it seems similar to Java). C++ will not fit the bill, since I do not feel advanced enough to use it for what we need. The sensor parsing is already finished. The data transferring will be happening on the same machine.
Here are the methods I've pondered:
We tried using MatLab with whatever the Mex stuff is (I don't do MatLab) to access functions from our C++ program, to retrieve the data as an array. Matlab will be too slow (we read somewhere that the TX/RX will be limited to 1-20 Hz.)
Writing the data to a text, or other equivalent raw data, file constantly, and opening it with the other language as necessary.
I attempted to look this up, but nothing of use showed in the results.
It seems like you are looking for a IPC (Inter-process communication). The easiest ones to implement are the socket and Pipes. I have added links which will help you decide which one to use. I have implemented named pipes and socket to transfer data at every 1ms. I did these implementations in python, c and pascal.
Here is the descriptions of all the IPC types
What's the difference between pipes and sockets
We had same issue where we had to share sensor data between one Java app to other multiple apps including Java,Python and R.
First we tried Socket connections but socket communication were not fault tolerant. Restarting or failure in one app affected other.
Then we tried RMI calls between them but again we were unhappy due to scalability.
We wanted system to be reliable, scalable, distributed and fault tolerant. So, finally we started using RabbitMQ where we created one producer and multiple consumers. It worked well for 2 years. you may consider using Apache Kafka.
You have options like Socket pipes, RMI calls, RabbitMQ, Kafka, Redis based on your system requirements now and in near future.
I'd like to somehow get control over active sockets on my computer with java/scala. For example, if a program has stablished a tcp connection, i want to be able to retrieve and listen/write onto this socket.
So, the first question is: is that possible?
And the second: how?
You can do this by writing a custom Socket factory. This is fairly complicated and I don't suggest you do this unless you are trying a to hack a program you have no control over (again a very bad idea)
Instead I suggest you monitor your own usage. When you read/write from a Socket you also keep any information you need. i.e Its your program so change it to do what you need.
I found a java wrapper of lipcap http://jnetpcap.com/ that let me do the sort of things i was after. From the web:
jNetPcap is an open-source java library. It contains:
A Java wrapper for nearly all libpcap library native calls
Decodes captured packets in real-time
Provides a large library of network protocols (core protocols)
Users can easily add their own protocol definitions using java SDK
jNetPcap uses a mixture of native and java implementation for optimum packet decoding performance
can I call Java from Node.js via JNI? Are there any examples?
You should try the node-java npm module which is a well-written wrapper over JNI.
node-jave doesn't appear to (yet) have broad adoption, but playing with it, I've been impressed with how straightforward and robust it has been.
It's as simple as:
var list = java.newInstanceSync("java.util.ArrayList");
list.addSync("item1");
list.addSync("item2");
console.log(list.getSync(1)); // prints "item2"
You can do just about anything with your embedded JVM - create objects, call methods, access fields, etc.
There is a slight impedance mismatch between Node and Java, so if you are going to interact with something complicated, I'd recommend writing most of your interactions in Java and exposing a simpler interface across the Node/Java barrier. It just makes for easier debugging that way.
--- Dave
p.s., RealWorldUseCase(tm): I worked at a place that had a pretty complex (and spaghetti-coded) protocol between multiple browser clients and a Java-based service. I wrote a pretty sweet test-harness which used jsdom to host N simulated browsers and used node-java as a wrapper around the Java service code. It was trivial to shim out the transport interfaces, both in JS for the clients, and in Java for the service, so whenever any of these things sends a message, I capture that and stick it in a queue for probabilistic delivery to the intended target (ie, I virtualized the network). In this way, I could run a full-on simulation of multiple clients interacting with and through a Java service, and run the whole thing inside a single process without any wire communication. And then I could do fun stuff like deliberately reorder message deliveries to make sure the code was resilient to timing bugs. And when a bug was discovered, I had the message orderings logged and could reproduce them to repro the bug. Oh, and the whole thing set up and ran a pretty complex scenario with a few thousand lines of logging and finished in under 1 second per run. 2-weeks well spent. Fun stuff.
RealWorld Use Case #2: selenium-inproc - a module that wraps the SeleniumRC JAR file providing a node interface to browser automation testing w/ Selenium without having to run yet another localhost service.
That looks tricky. Node.JS runs on the Google Chrome JavaScript engine V8. What you will have to do is to create a V8 C++ binding (v8 c++ Crash Course shows an example) that starts a JVM and does all the JNI handling.
I think you might be better off letting a JavaServer and Node.js communicate via the network (someone wrote an example for using RabbitMQ for Java/Node.js message based communication). Here, JSON would be a great data exchange format (if you trust your Java server produces proper JSON you can just eval() it in Node).
I think what you are looking for is a native extension to use as a bridge. Although I don't have an example of what you are saying, I do have an example on how to call a C++ extension using Node.JS
https://github.com/jrgleason/NodeJSArduinoLEDController
I am not aware of all the details of Node.js but i am assuming that your mentioning of JNI is actually the Java Native Interface. One can only use JNI from Java, so imho it does not make sense to access Java from JNI if you are not already in java.
It would appear that this is the wrong approach, and you need to search the Node.js doco for their integration chapter...
I wonder if it is possible at all. but even if it is possible I imagine it is hard to implement and I am certain that nobody has done that yet.
how about using a named pipe to communicate between processes(java and node.js) ?
In delphi, I am trying to call a function from an external Java program. Is there any way to do it ?
The standard process to call native code is via JNI. A search on JNI and Delphi will reveal multiple pages that detail how this is done, like this and this
What is more desirable (setting up some out of process server (like Peter already detailed, so I skipped that) or using JNI to call a library depends on how often (and how realtime) you need this to be, and on allowable installation/configuration complexity
If it is a running Java application you will need to expose access to that function. There are a myriad of solutions possible.
If it is only 1 function or very limited functionality, then listening on the humble socket or named pipe is a solution which is currently undervalued and kind of forgotten.
On the next step of integration I would look at asynchronous message passing. It is easy to embed an activemq server or similar or start it in a separate process. This has a number of advantages like that the request are easily synchronized in the Java process by simply using one listening thread, that the behavior is well defined when the Java program is not available or the Delphi one. It is very easy to manage and you get the instrumentation for free.
An embedded Jetty webserver is an easy, reliable solution and implement a servlet to do your bidding. Again a lot of the complexity is now handled by using ubiquitous and standard protocols.
Then there are the synchronous RPC methods like COM, Corba, SOAP which I personally find much too complex, error-prone and maintenance unfriendly to use for ad-hoc communication between processes. If you want to build a complete infrastructure of stuff talking to each other it might be worth it, but not to get 2 programs talking.