I'm currently developing a simple P2P network as an exercise. Each node in the network sends heartbeats to a subset of the other nodes to be able to detect nodes that have left the network. Beside the heartbeat packets I send packets when new nodes join/leave the network, when they want to locate a resource (small text files), etc. All packets are UDP packets.
Whenever I receive a packet I start a new thread that handles that specific packet. I am however concerned with the amount of threads I start during one applications lifetime which adds up to quite a lot (Especially because of the heartbeats). (There is also the risk of deadlocks and the like I would like to avoid).
I thought about having a queue or something where I put all incoming packets and have a single thread handling all packets one at a time from that queue (something like the producer-consumer pattern). I would like the packets to be handled rapidly so the sender doesn't think the packet is lost.
What is the best way to handle a lot of different incoming packets without having to start a new thread for each of them? Should I go with what I have, the producer-consuming or something different?
How long does it take to your application to process one packet?
For the ping ones it is probably faster to just process them as they are received, you can put the others in a shared data structure such as a particular blocking queue, so when the queue is empty the worker threads wait for new jobs, and when a new jobs is added, a thread is awaken and will do the job.
Probably starting one thread per packet makes you consume more time on starting and stopping the threads than in actually doing the job.
If the things to do in response of a packet aren't so time consuming for all the type of packets, it might be the case that the extra time spent with the locks of the queue and scheduling threads makes your program slower rather than faster.
In any case use thread pool and start the workers in the beginning. If you want you could increase or reduce the number of working threads dynamically depending on the load of the past minutes.
I would use an event driven architecture. Creating a new thread for every packet is not scalable, so this will work to a certain amount of workload, but there is a point it won't work anymore. You could compare that to e.g. a chat program like the Facebook chat where messages are the packets.
An event driven architecture would be scalable and IMHO exactly what your looking for. Just do some googling, there libraries for many programming languages, so just pick the right one for you (I like to do that in Erlang, Scala, C or Python).
edit: ok, didn't see the java tag. But the language does not matter.
Take a look at this link for example:
http://www.nightmare.com/medusa/async_sockets.html
I find it a quite good one to get the idea of event driven programming.
Related
I have an issue that is driving me crazy! Both design-wise and tech-wise.
I have a need to listen to a LOT of multicast addresses. They are divided into 3 groups per item that I am monitoring/collecting. I have gone down the road of having one process spin-up 100 threads. Each thread uses 2 ports, and three addresses/groups. (2 of the groups are on same port) I am using MulticastChannel for each port, and using SELECT to monitor for data. (I have used datagram but found NIO MulticastChannel much better).
Anyway, I am seeing issues where I can subscribe to about a thousand of these threads, and data hums along nicely. Problem is, after a while I will have some of them stop receiving data. I have confirmed with the system (CentOS) that I am still subscribed to these addresses, but data just stops. I have monitors in my threads that monitor data drops and out-of-order via the RTP headers. When I detect that a thread has stopped getting data, I do a DROP/JOIN, and data then resumes.
I am thinking that a router in my path is dropping my subscription.
I am at my wits end writing code to stabilize this process.
Has anyone ever sent IGMP joins out the network to keep the data flowing? Is this possible, or even reasonable.
BTW: The computer is a HP DL380 Gen-9 with a 10G fiber connection to a 6509 switch.
Any pointers on where to look would really help.
Please do not ask for any code examples.
The joinGroup() operation already sends out IGMP requests on the network. It shouldn't be necessary to send them out yourself, and it isn't possible in pure Java anyway.
You could economize on sockets and threads. A socket can join up to about 20 groups on most operating systems, and if you're using NIO and selectors there's no need for more than one thread anyway.
I have used datagram but found NIO MulticastChannel much better).
I don't know what this means. If you're referring to DatagramSocket, you can't use it for receiving multicasts, so the sentence is pointless. If you aren't, the sentence is meaningless.
I have a Java program with multiple sockets that occasionally have data that need to be read and processed, but there is an indeterminate amount of time which there is no data to be read. I need a good way to constantly check if there is data in the sockets, and process the data. Assigning one thread per socket is not a good idea since there could be too many sockets and use too much memory.
Currently, I have a couple threads, each one assigned to service its own list of sockets. If there was nothing to read in any of the sockets, then sleep one second, then loop. If there was something to read in any of the sockets, just loop without waiting and iterate through the sockets again.
The reason I do this is because I don't want to use up too much resources if there is nothing to read, and the one second delay is not a problem. The only down side is that there is no flexibility for sockets to jump threads, so the worst case scenario is that a single thread is overloaded with work, while the other threads are doing nothing.
Another idea I've had: create a thread pool, and queue up all the sockets to be serviced, and re-add them when they are serviced, but there is no good way to know if none of the sockets need servicing and the threads can take a break to free up CPU cycles.
Is there a good way to assign threads tasks, but not overload computer resources if there is nothing to do?
Ideally an event is triggered each time there is data available in a socket, but as far as I know, there is no way to do this, and I must poll the sockets.
To reiterate, I do not want a one to one relationship between socket and thread.
there could be too many sockets and use too much memory.
You can achieve 1,000 to 10,000 this way. Memory is much cheaper than it was when NIO was introduced 12 years ago and threads are more efficient and scalable than they used to be.
I have a couple threads, each one assigned to service its own list of sockets. If there was nothing to read in any of the sockets, then sleep one second, then loop.
I use a pause which busy waits for a short period and yeilds and finally sleeps for an escalating period of time.
You can use Selectors, but these are not simple to use correctly. In this situation I would use a library like netty or at the very least read the code it uses.
The only down side is that there is no flexibility for sockets to jump threads, so the worst case scenario is that a single thread is overloaded with work, while the other threads are doing nothing.
This is where using a thread per socket is better.
I must poll the sockets.
You can use Selectors, but these are single threaded and switch sockets between selectors is not simple.
I would reconsider using more threads for simplicity.
One EventHandler(DatabaseConsumer) of the Disruptor calls stored procedures in database, which is so slow that it blocks the Disruptor for some time.
Since I need the Disruptor keep running without blocking. I am thinking adding an extra queue so that EventHandler could serve as Producer and another new-created thread could serve as Consumer to handle database's work, which could be asynchronous without affecting the Disruptor
Here is some constrain:
The object that Disruptor passed to the EventHandler is around 30KB and the number of this object is about 400k. In theory, the total size of the objects that needs to be handled is around 30KBX400K =12GB. So the extra queue should be enough for them.
Since performance matters, GC pause should be avoided.
The heap size of the Java program is only 2GB.
I'm thinking text file as a option. EventHandler(Producer) writes the object to the file and Consumer reads from them and call stored procedure. The problem is how to handle the situation that it reach to the end of the file and how to know the new coming line.
Anyone who has solve this situation before? Any advice?
The short answer is size your disruptor to cope with the size of your bursts not your entire volume, bare in mind the disruptor can just contain a reference to the 30kb object, the entire object does not need to be in the ring buffer.
With any form of buffering before your database will require the memory for buffering the disruptor offers you the option of back pressure on the rest of the system when the database has fallen too far behind. That is to say you can slow the inputs to the disruptor down.
The other option for spooling to files is to look at Java Chronicle which uses memory mapped files to persist things to disk.
The much more complicated answer is take advantage of the batching effects of the disruptor so that your DB can catch up. I.e. using a EventHandler which collects events a batch of events together and submits them to the database as one unit.
This practice allows the EventHandler to become more efficient as things back up thus increasing throughput.
Short answer: don't use disruptor. Use a distributed MQ with retransmission support.
Long answer: If you have fast producers with slow consumers you will need some sort of retransmission mechanism. I don't think you can escape from that unless you can tolerate nasty blocks (i.e. huge latencies) in your system. That's when distributed MQs (Messaging Queues) come to play. Disruptor is not a distributed MQ, but you could try to implement something similar. The idea is:
All messages are sequenced and processed in order by the consumer
If the queue gets full, messages are dropped
If the consumer detects a message gap it will request a retransmission of the lost messages, buffering the future messages until it receives the gap
With that approach the consumer can be as slow as it wants because it can always request the retransmission of any message it lost at any time. What we are missing here is the retransmission entity. In a distributed MQ that will be a separate and independent node persisting all messages to disk, so it can replay back any message to any other node at any time. Since you are not talking about an MQ here, but about disruptor, then you will have to somehow implement that retransmission mechanism yourself on another thread. This is a very interesting problem without an easy answer or recipe. I would use multiple disruptor queues so your consumer could do something like:
Read from the main channel (i.e. main disruptor queue)
If you detect a sequence gap, go to another disruptor queue connected to the replayer thread. You will actually need two queues there, one to request the missing messages and another one to receive them.
The replayer thread would have another disruptor queue from where it is receiving all messages and persisting it to disk.
You are left to make sure your replayer thread can write messages fast enough to disk. If it cannot then there is no escape besides blocking the whole system. Fortunately disk i/o can be done very fast if you know what you are doing.
You can forget all I said if you can just afford to block the producers if the consumers are slow. But if the producers are getting messages from the network, blocking them will eventually give you packet drops (UDP) and probably an IOException (TCP).
As you can see this is a very interesting question with a very complicated answer. At Coral Blocks we have experience developing distributed MQs like that on top of CoralReactor. You can take a look in some of the articles we have on our website.
I have a server for a client-server game (ideally the basis for small MMO) and I am trying to determine the best way to organize everything. Here is an overview of what I have:
[server start]
load/create game state
start game loop on new thread
start listening for udp packets on new thread
while not closing
listen for new tcp connection
create new tcp client
start clients tcp listener on new thread
save game state
exit
[game loop]
sleep n milliseconds // Should I sleep here or not?
update game state
send relevant udp packet updates to client
every second
remove timed out clients
[listen for udp]
on receive, send to correct tcp client to process
[listen for tcp] (1 for each client)
manage tcp packets
Is this a fair design for managing the game state, tcp connections, and send/receive udp packets for state updates? Any comments or problems?
I am most interested on the best way to do the game loop. I know I will have issues if I have a large number of clients because I am spawning a new thread for each new client.
That looks like a reasonable start to the design. It wouldn't be bad to go beyond 10 clients (per your comment) in terms of scaling the number of threads. As long as the threads are mostly waiting and not actually processing something you can easily have thousands of them before things start to break down. I recall hitting some limits with a similar design over 5 years ago, and it was around 7000 threads.
Looks like a good design. If I were you I would use an existing nio framework like netty.
Just google java nio frameworks and you'll find several frameworks with examples and documentation.
Update
Thanks, but I prefer to do it all on my own. This is only for a side project of mine, and much of its purpose is
imho you'll learn more by using an existing framework and focus on the game design than doing everything by yourself from start.
Next time you'll know how do design the game and that makes it easier to design the IO framework too.
I'd say that (especially) in Java, threads are more for convenience than for performance (there are only oh-so-many cores, and I guess you'd like to keep some control on which threads have priority).
You haven't said anything about the volume of message exchange and number of clients, but you might want to consider more natural from the perspective of having a single thread handling the network IO, and having it dispatch incoming requests in a tight loop, and mapping the ACTUAL work (i.e. requests that you can't handle in the tight loop) to a thread pool. IIRC, in such a setup the limiting factor will be the maximum number of TCP connections you can wait for input on simultaneously in a single thread.
For increased fun, compare this with a solution in a language with has more light-weight threads, like in Erlang.
Of course, as #WhiteFang34 said, you won't need such a solution until you do some serious simulation/use of your code. Related techniques (and frameworks like Netty where you could find inspiration) might also be found in this question here.
There is a certain cost to thread creation, mostly the per-thread stack.
From what I read about Java NIO and non-blocking [Server]SocketChannels, it should be possible to write a TCP server that sustains several connections using only one thread - I'd make a Selector that waits for all relevant channels in the server's loop.
Is that right, or am I missing some important detail? What problems can I encounter?
(Background: The TCP communication would be for a small multiplayer game, so max. 10-20 simultaneous connections. Messages will be sent about every few seconds.)
Yes, you are right. The problems you can encounter is when the duration of processing is too long. In this case, you'd have to wrap the processing inside another thread, such that it will not interfere with the networking thread, and prevent noticeable delay.
Another detail; Channels are all about "moving" data. If your data you'd wish to send is ready, then you can move this data to a networking channel. The copying/buffering/etc. is all done by the NIO implementation, then.
Your single-threaded "networking thread" is only steering the connection, but not throttling it (read: weird analogy with a car).
The basic multithreaded approach is easier to design and implement than a single threaded NIO. Performance gain isn't noticeable in a small multiplayer game server/client, especially if a message is only sent every few seconds.
Brian Agnew said:
This all works well when the server-side processing
for each client is negligible. However a multi-threaded
approach will scale much better.
I beg to disagree. A one-client-one-thread approach will exhaust memory much faster than if you handle multiple clients per thread as you won't need a full stack per client. See the C10K paper for more on the topic: http://www.kegel.com/c10k.html
Anyway, if there won't be more than 20 clients, just use whatever is easiest to code and debug.
Yes you can. See this example for an illustration on how to do this.
The important section is this:
for (;;) { // Loop forever, processing client connections
// Wait for a client to connect
SocketChannel client = server.accept();
// Build response string, wrap, and encode to bytes (elided)
client.write(response);
client.close();
}
This all works well when the server-side processing for each client is negligible. However a multi-threaded approach will scale much better.