We Keep Coding

How to use zeroMQ in Desktop application

I am working in a desktop application, where application is deployed in both windows and mac platforms. As part of the application, it should communicate with native layer. Currently the communication between native layer and Java layer is done using sockets. Recently some one in the team suggested to use zeroMQ. Can any one of you guys please clarify my doubts.
How zeroMQ better than sockets
Is it possible to install zeroMQ library as part the Desktop client installation
I gone through the link 'https://github.com/zeromq/clrzmq4', it given libraries specific to amd64 and i386 processor family. Do I need to build it separately from the source code for different processors?
Do I still require .dll files to use zeroMQ in Java?
Do I require Visual studio to build zeroMQ libraries in windows (Since my native layer written in C#, my C# application communicate with zeroMQ socket socket written java)?

How zeroMQ better than sockets
http://zeromq.org/topics:omq-is-just-sockets
Is it possible to install zeroMQ library as part the Desktop client installation?
Yes, you need to build the libraries depends on the processor and embed them in your application.
Do I need to build it separately from the source code for different processors?
Yes, you need to build the libraries from source. zeroMQ is processor centric.
Do I still require .dll files to use zeroMQ in Java?
Yes, Following link may help you
Exception in thread "main" java.lang.UnsatisfiedLinkError: ... \jzmq.dll: Can't find dependent libraries
Do I require Visual studio to build zeroMQ libraries in windows?
Yes
This link may help you to get basic examples.

Regarding ZeroMQ in a desktop application on Windows talking to another process on the same machine, bear in mind that zmq_ipc is not supported (see zmq_ipc(7)). Or at least, that's the last I heard. This is because it's fundamentally impossible to implement anything like quite like select() or epoll() for named pipes in Windows. Just use zmq_tcp instead.
The same basic problem plagued the development of the select() implementation in Cygwin and its derivatives. They got round the problem by starting a thread for every non-socket file descriptor (i.e. named pipes, serial ports, etc) being selected, with each thread polling the HANDLE for whether any data had arrived (or whatever events were being set in select()). Not very efficient. Yeurk.
Proactor vs Reactor
Windows is proactor (which can do only proactor), everything else (*nix, VxWorks) is reactor (which can also be used to implement a proactor). The development of the boost.asio library for C++ was influenced by this, and is a proactor design as a result so that it could be run on Windows. RabbitMQ is proactor too.
ZeroMQ with zmq_poll() is reactor.
Proactor - you pro-actively start up an asynchronous routine to handle whatever turns up in the future.
Reactor - you react to the arrival of whatever has turned up by starting a synchronous call to whatever routine you wish to handle it knowing that it will complete very quickly because the data is already there.
The difference is key. In a proactor design, once you have started up that asynchronous routine to read a message, you cannot (easily) stop it or change it until it has done its thing. That is very annoying if you change your mind, for example as a result of reading some message from somewhere else.
Small caveat - Windows does support select() for network sockets (thus reactor programming is possible with network sockets, but only network sockets), and is the only reason why ZMQ is supported to any extent whatsoever on Windows.
Mixing ZMQ with Desktop Application Event Callbacks
Anyway, proactor means that Windows and C# fundamentally expects everything to be served by callbacks. This basically means you won't be using the zmq_poll() call to tell you if new messages have arrived if you also have callbacks handling GUI events. Instead you'd most likely be making asynchronous calls to zmq_revcmsg(). Trying to mix zmq_poll() in with callbacks is asking for trouble (you'd be blending proactor and reactor).
Message Formats
ZeroMQ and sockets both transfer bytes (as discrete messages with ZeroMQ, as a byte stream with sockets). One still has the challenge of deciding what the bytes mean to applications.
I can recommend using something like Google Protocol Buffers to serialise messages for transport by ZeroMQ. It's available for both C# and Java, but it doesn't demarcate message boundaries. Fortunately, ZeroMQ does. [Using GPB over a socket stream can be painful, you have to demarcate the message boundaries oneself]. So you can serialise a message to a buffer, hand the buffer over to ZeroMQ as a message, the recipient receives the message and knows for absolute certain that there is one single solitary GPB within. If you like you can use GPB's "oneof" to smuggle arbitrary message types across, which can be very liberating. You can accomplish the same with other serialisation technologies too of course, my personal favourite being ASN.1.

Packet Capturing/Latency time in java

I am looking for HTTPS packet capturing code in java . But i am getting procedure for HTTP. As per our requirement we need HTTPS packet capturing code. Can anybody give me the link for that or would tell me where can I find it?.

jNetPcap might be what you're looking for: it is a java wrapper around the native libpcap library that sarnold mentions. From the basic capture example provided, it seems that it would be fairly straightforward to write a PcapPacketHandler to time the arrival of packets on an HTTPS stream.
It will, however, add native dependencies to your project, which will complicate packaging up your software.

I would suggest starting with libpcap (or its sibling, winpcap on Windows), and then interpreting the saved captures; the libpcap team has done a fantastic job making a very reliable and fast mechanism to interface with the kernel's raw sockets support to run as efficiently as possible. (The Linux kernel provides a BPF-like interface for specifying which packets to offload to userspace; see Documentation/networking/filter.txt for Linux details.)

FIX communication model - messaging or socket

My situation is: A c++ program needs to talk with a Java program using FIX protocol.
My solution:
- Messaging: C++ program publishes a text in FIX format which Java progrma can consume and parse with quickfix/j.
- Socket: Setup a FIX server in Java program, then C++ program as a client can connect to this socket and write byte stream into it using quickfix. Java program uses quickfix/j to parse the byte stream.
My questions:
1. Is there any compatiblity problem for socket solution, i.e. ,the byte stream coded with quickfix can be fully decoded by quickfix/j?
2. Which one is better? Cons and pros.
Thanks in advance.

FIX messaging would be an easier solution, rather than implementing sockets. There are socket communications already embedded in the quickfix libraries. It is no use to reimplement then unless you are doing something very different. The engine is meant to decipher FIX messages. And if you want to modify any of the socket communications for the libraries, you can change the libraries itself. You have the source code anyways.
If you try implementing sockets you may have to write wrappers around the sockets to parse messages from C++ along to Java and vice versa.
You have the C++ version of quickfix library. Use that as a initiator to send FIX messages across to the Java acceptor. You probably wouldn't have to worry about writing a C++ server to send FIX messages in a bytestream. Let the underlying library do the work of doing the communication rather than yourself.

FIX might be easier. But if you choose sockets, make sure to convert data send/recv from/on the C++ program to/from network byte order. (See reference for: htons(), htonl() ntohs(), and ntohl()). Java always uses network byte order so you don't have to do any conversion there.

FIX is a text based protocol, i.e. you don't have to worry about byte order. At the wire level, all you're doing is sending buffers of characters. So if you're writing in a C++ program to a java (quickfixj) based client/server, as long as you adhere to the FIX protocol, you'll have no issues.
Then again, as DumbCoder pointed out above, if you're not overly concerned about performance, you could use quickfix (the C++ version!)

Fix engines are written in a way they can communicate with other party fix engine.
Language or platform of the two parties does not matter.
You caan simply use JAVA version for one and C++ version for the other party.

Efficient file transfer from Java server to multiple C++ clients?

I need to transfer files fast over the Internet from a Java server to C++ clients, where often many clients would need the same files. I was looking at say transferTo() in Java which sounds like it would be a decently optimized function to send files. However, I'm not sure when I use transferTo() how to best receive that in C++ (i.e. is it just a raw data transfer, how do I determine when the file is over on the client side, etc.). I need this to work on both Windows and Linux. Also, other than transferTo(), would there be some way to be more efficient, especially by taking advantage of the fact that many clients will usually need the same files? I'm not sure how to do say multicast etc. Also, I'm using application-level security rather than a VPN, and on the Java server, encrypting with AES and using MAC digital signing, so I'm also looking for a cross-platform library recommendation to deal with the crypto on the C++ side with minimal pain.
I'm very proficient in C++ but have no previous experience with network programming, so please consider than in any suggestions.
Thanks.

An embedded webserver? http-transfers are efficient enough for you?
The simplest embeddable Java webserver I remember seeing is http://acme.com/java/software/Acme.Serve.Serve.html. We use embedded Jetty 6 in production at work, but that takes more elbow grease.
If your clients doesn't know where to find your webserver in the first place, consider announcing using Zeroconf. http://jmdns.sourceforge.net/

For scalability reasons, Thorbjørns suggestion of using http seems like a very good idea as it would allow you to easily set up http proxies for caching, use standard load balancing tools and so forth.
If you are looking to transfer more than just a blob of data, you might want to have a look at googles protocol buffers. They allow for very easy and fast encoding/decoding on the java and c++ end.

Consider chunking the file and sending via UDP datagram. C++ can re-compile as it receives it. Have you considered implementing/embedding an existing P2P protocol implementation?

If you need effecient transfer to many clients then your bottleneck is the server.
For this please look at the bit-torrent protocol as it distributes the transfer between the clients.

How do I read and write raw ip packets from java on a mac?

What would be the easiest way to be able to send and receive raw network packets. Do I have to write my own JNI wrapping of some c API, and in that case what API am I looking for?
EDIT: I want to be able to do what wireshark does, i.e. record all incomming packets on an interface, and in addition be able to send back my own created packets. And I want to do it on a mac.

If you start with the idea that you need something like a packet sniffer, you'll want to look at http://netresearch.ics.uci.edu/kfujii/jpcap/doc/.

My best bet so far seems to be the BPF api and to write a thin JNI wrapper

Raw Socket for Java is a request for JDK for a looong long time. See the request here. There's a long discussion there where you can look for workarounds and solutions. I once needed this for a simple PING operation, but I can't remember how I resolved this. Sorry :)

You can't access raw sockets from pure Java, so you will need some sort of layer between your Java code and the network interfaces.
Also note that access to raw sockets is normally only available to "root" processes, since otherwise any user could both a) sniff all traffic, and b) generate spoofed packets.
Rather than write your whole program so that it needs to run as "root", you might consider having the packet capture and generation done in a standalone program with some sort of IPC (RMI, named pipe, TCP socket, etc) to exchange the data with your Java app.

TINI is a java ethernet controller, which may have libraries and classes for directly accessing data from ethernet frames to TCP streams. You may be able to find something in there that implements your needed classes. If not, there should be pointers or user groups that will give you a head start.