Background Info
I'm writing a custom UDP protocol which is targeted for general-purpose use, but will probably be used in a game development setting. My protocol handles all the issues of UDP, reliability, ordering and fragmentation are all handled. I'm using UDP because of the flexibility as I can send some packets unreliable and others reliable.
The Problem
I want my protocol to be encrypted, and I'm also very concerned about MITM. I've read several questions of people who also want to encrypt their UDP protocols and most people recommend DTLS. However the problem with DTLS is that it seems that no one uses it. I can't find decent guides or documentation on how to set up a client/server program for my chosen language (Java). It looks like the only option is bouncycastle, however considering the fact that their client/server test programs won't work with each other, it's probably not a good idea.
I then decided to write the low-level packet receiving and sending code in C and use OpenSSL for the DTLS implementation. I would then call my C code using JNA. However, once again I could not find decent guides or tutorials on how to do DTLS. I could only find two ones which were somewhat helpful, the first one just went over the general C functions to call in which order. I got the impression that your application had to do the client verification yourself and since I have no idea how to do that it wasn't very helpful. The second one was just a raw client/server program which worked when ran but with closer inspection it seems to disable client verification.
The actual question
How would I go about creating my own encrypted transport system over UDP? I read a little about DHKE, but I don't know how to write a good implementation of it in Java using UDP and closer inspection seems that it doesn't prevent MITM. For my specific application I actually probably don't need a key exchange, the clients can actually have the key pair with the server installed beforehand. Would going this route work well? (I would probably just encrypt each packet body with the pair and send it to the server/client)
Related
I'm developing a multi-player turn based game for Android - a poker game.
I'm almost finished with the clinet game-play. Now I need to implement the multi-player thing.
I thought about using Web Services but I have no experience with that and I undertsand that socket-communication is faster.
I have some experience with writing socket-communication in Java:
Using ObjectOutputStream and ObjectInputStream to exchange data (game-states and turn actions in this case).
But I have a concern with this approach - the server-side is platform-dependent.
If I want to have an iPhone developer create the app for iPhone, he could not do this because the server expects Java code. Am I right?
And another question please:
Can I implement normal-socket communication and then later easily change it to secure-socket communication? It's just that I need to learn how to use SSL and I don't want to go into it right now.
Thanks!
This is an old question but relevant for me and hopefully others so here goes. I'm currently working on a project that involves multiple computers on different platforms using Java sockets. I'm using sockets because eventually I can use any language as long as I adhere to the protocol I wrote. I too have some predefined integer constants that control what data is expected in what order. For example, before I start some particular function, I send a particular integer, followed by the expected data. Anything that doesn't come in in the expected order results in an error. So as long as the language/runtime offers the ability to write integers and strings over a TCP socket, it should work. I think that covers the vast majority of major programming languages.
As for SSL sockets, I'm doing that also :) I started out using non-SSL sockets and then refactored to make it work with SSL sockets. I was surprised at how easy it is. I would recommend this git repo for sample code: https://github.com/jawi/ssl-socket-demo.
Eventually I plan to have the user who uses my app generate the key store and self-signed certificate locally when they first run the app as part of its initial setup. As long as they import that same key store into the app on every other node that they use in their network, they'll be able to talk to each other. This will ensure that the communication is encrypted in a way that is unique to their network. Of course this will require that they keep the key store file securely stored in their local file systems :)
I am currently decide what kind of communication method/network protocol I am going to use for a new project.
What I can tell you about this project is that:
- It is Android/java based, using X amount of Android devices
- These devices should be able to send strings to each other over a local network. We are talking about small strings here. Small as in less than 100 characters.
- The amount of packages/transmissions being sent can vary "A LOT". I can't say how much unfortunately, but the network protocol needs to be as scalable as possible.
I have researched different kinds of possible solutions and is now deciding wether to use "Sockets" or "RMI"
As I have understood about RMI:
It is easier than Java sockets to implement and maintain (smaller amount of code)
It is "a bit slower" than sockets, as it is a new "layer" build on top of Sockets
There may be some scalability issues (if this is true, how "serious" is it?) as it creates a lot of new sockets, resulting in Exceptions.
Obviously the system needs to run as smooth as possible, but the main objective is to make it scalable so it can handle more Android devices.
EDIT: The system the system is not "peer-to-peer". All of the android devices should be able to be configured as the server.
None of your concerns are the real issue, in my view.
RMI has a pre-defined protocol, raw sockets do not.
If you use raw sockets, you have to do all the work to define what messages and protocols are exchanged by client and server.
There are so many good existing protocols (RMI, HTTP, etc.) that I'd wonder why you feel the need to invent your own again.
Android devices communicating over HTTP - tell me why it won't be fast or scalable enough. HTTP is good enough for the Internet - why not you and your solution?
I would suggest you to expose some kind of webservice (SOAP or REST) in your application server. For example, people frequently expose their data to mobile devices as a REST webservice API returning some kind of JSON format in order to make it easier to marshal it again in the client device.
This way you take profit of the underlying implementation of HTTP communication in every application server; any other way, you would have to write your own worker thread pool using nio primitive operations in order to achieve performance... Not a thing to be done in a real production environment - maybe in an academic one?
When you work with Bluetooth or WiFi Direct in Android, at the end of all of the handshaking and such, you wind up with sockets.
With TCP/IP, we have a zillion-plus-one libraries that layer on top of sockets, for high-level protocols: HTTP, XMPP, IMAP, etc. Courtesy of these libraries, we can deal with more domain-specific abstractions of an operation (e.g., "download this file"), with low-level socket plumbing handled by the library.
Question: Are there equivalents, for any high-level protocol, that are known to work (or are likely to work) with the sockets produced via Android's Bluetooth and/or WiFi Direct layers?
Right now, I'm not fussy about the specific protocol -- I'm just looking for examples of this sort of protocol layer, to make using these sorts of connectivity options easier for developers.
For example, it looks like I could create a fork or add-on for OkHTTP that uses an alternative source for sockets, and I could probably create a Java HTTP server that does the same. Given those, app developers would write HTTP apps that talked over Bluetooth or WiFi Direct (and, on the client side at least, the coding should be fairly "natural" in feel, once the connectivity-specific pairing and handshaking has gone on).
IOW, going back to dealing with raw sockets feels so two decades ago... :-)
Thanks!
UPDATE
Based on Kristopher Micinski's comment on the ZeroMQ answer, I figured some clarification might be in order.
It's easier to say what I don't want: I don't want to touch sockets, after creating them. Something else at a higher level should handle those for me, plus handle what I'd consider a "protocol" to be (e.g., determining when some communications operation has ended, beyond a socket closing).
Mostly, this is for my book. Most book examples for low-level socket stuff are unrealistic, such as "we open a socket to the server and immediately start blasting the bytes representing some image to be uploaded, then close the socket when we're done". While the examples work, you'd never write something like that in real life:
If you're really working at the socket level, you'd be implementing some protocol that has the hopes of addressing authentication, error handling, etc., even if you're rolling the protocol yourself
Few developers work directly with sockets today for Internet operations
Now, it'd be cool if the protocol offered by the layer were something developers were used to (e.g., HTTP) or had heard of even if they haven't used it (e.g., XMPP). And I'll settle for simple scenarios (e.g., N-way support is cool but not necessary). In this respect, based on preliminary research (conducted by a sleep-deprived brain), ZeroMQ isn't a bad option. It lacks a bit of "brand recognition" compared to, say, an XMPP stack that could work with arbitrary sockets. But off the cuff it seems to meet what else I'm looking for.
I recognize that these stacks will have limitations imposed by the underlying transport (e.g., Bluetooth works well for N-way only for small values of N). And I certainly don't want to portray -- here or in my book -- that whatever solution I portray is the be-all and end-all of socket based communication.
I just want something that has a prayer of being more realistic for actual use. Bonus points if it is something that I can grok, as I have always used higher-level protocols for TCP/IP communications, and so I'm short on experience with direct socket manipulation.
I found ZeroMQ to be useful for managing socket connection. They have a support in multiple languages which includes JAVA. You may use this to manage the sockets once you establish the connection over wifi-direct or BT.
I know it's a somewhat old question and already answered but I would like to contribute.
I did this app: https://play.google.com/store/apps/details?id=com.budius.WiFiShoot and although the WiFi Direct connection n handshake is somewhat broken and it's what causes most of my unhappy users, I'm handling all the communication using the excellent https://github.com/EsotericSoftware/kryonet
and my code is pretty much what you see on their examples, create kryo, register classes, open server, connect client to server IP and shoot objects across with the file information and later I shoot the actual files using this code https://code.google.com/p/kryonet/source/browse/trunk/kryonet/test/com/esotericsoftware/kryonet/InputStreamSenderTest.java
hope it helps.
When implementing a client/server solutions, one of the questions you always need to answer is about protocol.
In simple cases, it's possible that packets are always of the same type, so the protocol could even have no logic at all: client connects to the server and sever just says some fact, the client disconnects and that's it.
In more complex cases, some packets are can only be sent in some specific cases. For instance, imagine an abstract server that requires authorization: clients have to authorized before sending or getting any useful data. In this case, the concept of session appears.
Session is an concept that describes the state of client/server dialog: both client and sever expect something from eachother, while there are also things that both of them don't expect.
Then, going even deeper, pretend that protocol is quite complicated and it's implementation should be easily extendable. I believe, that the theoretically right solution here is using a finite state machine. Are there any Java frameworks/libraries that allow this state machine to be easily implemented? Or probably, any more protocol-specific solutions?
What I'm expecting is a framework that allows me to define states and transitions between them.
Update: the question is not about easiest client/server solution implementation, the question is about implementing custom protocol. So, please, don't recommend using web services.
I remember using Unimod FSM for finite state machines a few years ago, although for serious work I always preferred to implement the finite state machines directly.
I need to transfer files fast over the Internet from a Java server to C++ clients, where often many clients would need the same files. I was looking at say transferTo() in Java which sounds like it would be a decently optimized function to send files. However, I'm not sure when I use transferTo() how to best receive that in C++ (i.e. is it just a raw data transfer, how do I determine when the file is over on the client side, etc.). I need this to work on both Windows and Linux. Also, other than transferTo(), would there be some way to be more efficient, especially by taking advantage of the fact that many clients will usually need the same files? I'm not sure how to do say multicast etc. Also, I'm using application-level security rather than a VPN, and on the Java server, encrypting with AES and using MAC digital signing, so I'm also looking for a cross-platform library recommendation to deal with the crypto on the C++ side with minimal pain.
I'm very proficient in C++ but have no previous experience with network programming, so please consider than in any suggestions.
Thanks.
An embedded webserver? http-transfers are efficient enough for you?
The simplest embeddable Java webserver I remember seeing is http://acme.com/java/software/Acme.Serve.Serve.html. We use embedded Jetty 6 in production at work, but that takes more elbow grease.
If your clients doesn't know where to find your webserver in the first place, consider announcing using Zeroconf. http://jmdns.sourceforge.net/
For scalability reasons, Thorbjørns suggestion of using http seems like a very good idea as it would allow you to easily set up http proxies for caching, use standard load balancing tools and so forth.
If you are looking to transfer more than just a blob of data, you might want to have a look at googles protocol buffers. They allow for very easy and fast encoding/decoding on the java and c++ end.
Consider chunking the file and sending via UDP datagram. C++ can re-compile as it receives it. Have you considered implementing/embedding an existing P2P protocol implementation?
If you need effecient transfer to many clients then your bottleneck is the server.
For this please look at the bit-torrent protocol as it distributes the transfer between the clients.