We are building a new client along with a caching proxy component for an existing online game. The current setup looks like:
Server
| \
TCP TCP
| \
Client1 Client2
The new setup will be:
Server
| |
TCP TCP
\ \
Proxy <-- (New!)
| \
0MQ 0MQ
| \
Client1 Client2 <-- (New!)
The proxy component will sit close to the existing game server and speak the old line based TCP protocol to it, while speaking a new Google Protocol Buffers-based protocol over 0MQ to the new client. In addition to forwarding and translating messages between the server and its clients, the proxy will also cache (protobuf) messages destined for clients.
The server responds to client requests, but may also send unsolicited messages to specific clients, or to all clients. Since the proxy will maintain a cache of messages destined for a certain client, it needs to keep some state associated with each client.
I've been reading quite a bit in the ZeroMQ Guide, but I'm still not sure which of its "patterns" or socket combinations would suit this setup best. I'm hoping there's some ZeroMQ wizard out there who can give some advice on what the best way to go here is.
The question is not language specific, but the proxy component will be written in Java and the client in C#.
Many thanks in advance!
I think the Asynchronous Client Server pattern will get you close. Each of your clients creates a DEALER socket which connects to the ROUTER socket in your proxy. Review the example Java source. You could base your proxy on the ServerTask class, using the frontend/ROUTER socket and omitting the backend/dealer socket.
Your server will be able to respond to client requests and send unsolicited messages back to any client. You will need to cache the socket id of each client after the client has sent at least one message.
Your TCP connection to the game server is conceptually similar to the backend DEALER sockets between the server and workers - however since your connection is classic TCP and and not ZeroMQ you cannot use ZeroMQ sockets and will need to do something custom. And your custom solution requires that you not violate ZeroMQ's concurrency rules, when you integrate with classic TCP sockets.
Here is one way - use a set of PAIR sockets (lets call them A and B) and a BlockingQueue to act as a bridge between ZeroMQ and your TCP sockets. And create 3 threads - TCPRead, TCPWrite and ZeroMQ.
TCPRead reads from the TCP connection to the game server. It converts them to protobuf messages and forwards them through the A socket.
TCPWrite polls the BlockingQueue for messages. When it receives a message it converts it and sends it to the game server via TCP connection.
Finally, most of the work is in the ZeroMQ thread. It uses ZeroMQ polling and is constantly polling it's router socket (for all messages from all clients) and the B socket (for messages from the game server). Sending messages to a client is straightforward and is handled by sending the message to the router socket, prefixing the message with the socket id of the client. Sending messages to the server is done by adding a message to the BlockingQueue. It will be picked up by TCPWrite and forwarded to the game server.
Once you get this basic design working, you can go nuts and add workers etc as described in the various patterns.
Hope this helps.
You can also use 0MQ ROUTER sockets in raw mode, meaning you could write the proxy with two ROUTER sockets. I'd stay away from PAIR sockets except for parent-to-child thread pipes.
It takes a while to plough through the ZeroMQ documentation, excellent as it is. I did and came to the conclusion that Asynchronous Client-Server was close to what I initially wanted.
But it wasn't quite. So I then proceeded to build.
What I actually wanted is based on what I'd learnt from the various examples. A full description of what I ended up with is in here as is the code which may be of some use, for reference or as an actual library.
Related
I've an application that's using Apache mina library for communicating based on TCP. The apache mina library provides a callback with IOBuffer that contains data coming over the network, however often times the data is received out of order or redundantly. I skimmed through the TCP protocol and it says that the protocol always ensures delivery of the data in correct order. The company that provided the APIs for their server claim that they are using TCP/IP for sending the response back however before sending the response back their server doesn't care about confirming if the client (in this case my application/apache mina library) is connected to the server. So the server just fires off the message and moves on.
If I'm not mistaken, that's the UDP protocol's behavior. My question is, if the server is using TCP for sending the response back:
Why do I get out of order data (it's rare but happens one in a while)?
How can a machine that's using TCP protocol just fire and forget about the data without making sure the receiver device is connected to it before sending the data?
Is this really TCP or UDP or some variation of TCP protocol?
Apache Mina does asynchronous messaging over the top of various transports including TCP.
Since Mina is asynchronous, out-of-order delivery should be expected.
Why do I get out of order data (it's rare but happens one in a while)?
One possible explanation is that multiple TCP streams are being used. Data delivered using one TCP stream will be delivered in order, but if multiple streams are used, data in one stream could "overtake" data on another stream, in the TCP stacks on the sending or receiving end, on the network, or in the client side library.
How can a machine that's using TCP protocol just fire and forget about the data without making sure the receiver device is connected to it before sending the data?
Because ... reliable delivery is not a basic attribute of Mina.
If you are using Mina to talk to a service with a particular application protocol, then that protocol will determine will determine whether "sending the data before checking the receiver is connected" is allowed / will work or not. For example, it won't for an HTTP response, because an HTTP response is sent on a connection that was previously established to send the request.
Actually, it seems that there are a variety of ways to use Mina. Some involve an application protocol; e.g. see HttpClientCodec and HttpServerCodec. Others don't.
Is this really TCP or UDP or some variation of TCP protocol?
If they say that TCP is being used as transport, then it is. However, Mina is neither TCP or UDP. It is Mina. It hides the details of the transport.
Bottom line, if you want the reliability / in-order delivery properties of TCP/IP, you should probably use them directly. Mina is providing higher performance than conventional TCP/IP over a synchronous socket by relaxing the normal properties of a (single) stream-based transport.
I have 2 components, A and B, on different LANs with a Linux server between which acts as gateway for both LANs.
A sends TCP traffic to B using a gateway deployed on the public Internet. Therefore, no SYN is sent from A to B via the Linux server. However, C sends TCP traffic back to A using a local TCP connection (the gateway is not involved). By sniffing the traffic, I can see that C is not establishing a regular TCP connection (SYN) but starts by sending (SYN+ACK).
I'm using iptables on the Linux server to redirect locally the TCP traffic sent by C to A. Traffic is redirected to local port 9000.
However, I would also like to intercept this traffic, therefore I've built a simple Netty proxy which listens on port 9000. For some reason, Netty doesn't receive or ignore this traffic.
I wonder if Netty is able to manage the case of a SYN-ACK without SYN.
Any idea?
The question is whether the TCP protcool implementation stack in the kernel will manage it.
Netty won't even see it, whether managed by TCP/IP or not.
Netty isn't an implementation of TCP/IP. It is an API layered over the Java API, which is layered over the BSD Sockets API, and none of those is an implementation of TCP/IP either. It can't see any further into the network than calling listen() and accept() in this case.
I'm currently developing a Java WebSocket Client Application and I have to make sure that every message from the server is received by the client. Is it possible that I lose some messages (once they are sent from the server) due to a connection interruption? WebSocket is based on TCP so this shouldn't happen right?
It can happen. TCP guarantees the order of packets, but it does not mean that all packets sent from a server reach a client even when an unrecoverable trouble happens in an underlying network. Imagine someone pulls out your LAN cable or switches off your WiFi access point at the worst timing while your application is communicating with your server. TCP does not overcome such a trouble.
To ensure that every WebSocket message sent from your server reaches your client, you have to implement some kind of SYN/ACK in the application layer.
TCP is a guaranteed protocol - packets will be received in the correct order by the higher application levels at the far end (this is as opposed to UDP which is a send and hope protocol).
Generally speaking TCP should be be used for connections where all the data must arrive correctly at the far end. UDP is used where a missing packet can be dropped without significant issue (e.g. streaming services, NTP updates)
In my game, to counter missed web socket messages, I added an int/long ID for each message. When the client detects that something is wrong in the sequence of IDs it receives, the client will request for new data from the server to be able to recover properly.
TCP has something called Control Flow- which means it provides reliable, ordered, and error-checked delivery.
In other words TCP is a protocol that checks constantly whether the data arrived.
This protocol has different mechanisms to ensure that.
You can see the difference between TCP and UDP (which has no control flow) in the link below.
Difference between tcp and udp
I am looking to build an instant messenger in Java.
Clients will connect to the server to log in.
They will start a conversation with one or more other clients.
They will then post messages to the server that will relay the messages to all the clients.
The client needs to be continually updated when users post messages or log in.
so the way I see it, the client needs to run a server itself in a separate thread so that the main server can send stuff to it. Otherwise the client will have to the poll the main server every xyz seconds to get the latest updates. And that would need a separate thread anayway, as that would be purely for getting updates whereas the 'main' thread would be used for when the client initiates actions such as posting messages/inviting others to conversations etc...
So anyone recommendations on how to write this instant messenger? Does it sound like a good idea to make the connection a 'two-way' connection where both the client and server act as servers? Or is polling a better option? Anyone know how the IRC protocol does this?
There's no real advantage of having 2 connections unless they can be handled independently (for example receiving / sending a file usually done in a separate connection). A connection itself is already a two-way communication channel so it can be used to both send and receive messages, events etc. You don't need to poll server since client is able to maintain persistent connection and just wait for data to appear (optionally sending periodic PING-like message to ensure connection is alive).
IRC uses a single connection to server to exchange text commands. For example one of the main commands:
PRIVMSG <msgtarget> <message>
This command can be originated either by client or by server. Client sends PRIVMSG to notify that it wants to deliver message to one or more destination (in IRC this either user(s) or channel(s)). Server's task here is to properly broadcast this message to appropriate clients.
If you're using raw InputOutput streams then yes this is a good way of doing it. You create one thread on the clientside that acts in a similar fashion as the server thread - waits for any incoming updates and when it does it updates the client. I wouldn't call it a server though. So you'd ideally have 2 TCP/UDP connections one for requests made by the client and one to notify the client of server changes.
This solution in an enterprise environment would probably be done through some kind of messaging framework such as Spring Integration but dig deep enough and it will essentially be a similar way to how you mentioned.
Do you need a fully custom protocol or would it be sufficient to use the XMPP? There are several open source libraries implementing XMPP.
http://xmpp.org/xmpp-software/libraries/
e.g. http://www.igniterealtime.org/projects/smack/
For me, to develop instant messaging service, I will use websocket protocol instead of normal java socket because the normal socket can not work well with HTTP protocol and moreover some network providers and firewalls banned custom ports. If you develop it in normal socket, your service could not be accessed by web clients.
Did you plan to develop the instant messaging service yourself? How about using other protocols such as Jabber?
I am writing this game in Java and have problems with networking architecture.
I decided I will UDP packets. I am just at the beginning, but the problem I am facing is that it seems to be that server have to respond from exactly same IP/Port to client (which is behind router which uses NAT) as client connected that server.
For example I have client A behind router. Client A has IP (local) 192.168.8.100 and it connects server B from port 1234. Server is on 11.11.11.11:2345.
When client A connects to server B it uses 192.168.8.100:1234 but router converts that to (for example) 22.22.22.22:6789.
Now, when server wants to send packets to that client it has to be from 11.11.11.11:2345.
I would like to send data from another port like 11.11.11.11:2222, but this does not seem to work, at least not with my router.
I want to use different port because I want to have two threads one for listening and one for sending data, and each thread would have it's own DatagramSocket. But, as i said once client A connects to server on port 2345, I can not send data from port 2222.
Does anyone know how is this handled? I am doing it in Java, but it's not really a language specific problem.
UPDATE
After #Perception commented I have some more questions regarding his comments:
OK, so if I understand this correctly, if I have server which is hosting 1000 games, each with 2 players, all sending/receiving will have to be done through the same DatagramSocket.
As I understand DatagramSocket is thread safe so I guess I can have one thread doing:
datagramSocket.receive();
while at the same time second thread is doing
datagramSocket.send(.....);
Correct?
Also, two threads can send data at the same time through the same DatagramSocket? Is sending in any way serialized, meaning that second send() starts only after previous send() is finished or is data being sent at the same time?
gorann, I'm not sure if I'm understanding you correctly, but it sounds like you're trying to control the port on which the server communicates with the client. There's no way to control this, and for good reasons.
This is one of the trickier differences between TCP and UDP.
When a new TCP session is initiated, the server side call to accept() gives you a new socket and the OS handles multiplexing the various sessions for you. With UDP, you need to handle the multiplexing yourself. But you need to do so in a way that works with NATs and other firewalls.
The way NAT works is that when it sees an outgoing packet, it creates a temporary rule allow packets to return along the same port pair. Data returning from a port that the client has not yet sent to will likely be blocked.
This gives you two choices:
You could do all of your communication through a single port. This is not a bad option, it just means that you need a way to identify client sessions and route them to the appropriate thread.
You could create a separate port and instruct the client to send to that one instead. Have the server listen on a fixed port. The client sends a message to there, the server then sets up a new session port and sends that number back to the client using the server's listen port. The client then sends a message to the session port, which causes the NAT to open up that port and allow return traffic. Now the client and server thread have their own private port pair.
Option 1 is a bit more work because it requires data to be exchanged between threads, but it scales up better. Option 1 is easier and more CPU efficient because each session thread can be independent, but there are a finite number of ports available.
Either way, I recommend that you have the client include a semi-unique session id in each packet so that the server has more than just the client address and port number to verify who belongs to each session.