I'm just looking for some insight into program design, since I'm not that experienced with threads.
Currently programming a little client. You can specify if that client is the "server" or "client" and it will create the protocols in the correct manner.
I do a little handshake between them (a syn, ack, synack sort of thing) and then start the network thread.
I have the main thread, who has an infinite while loop and does two things.
Adds a message to a "toSend" queue.
Pulls in all the messages from the "received" queue and prints them.
The network thread has an infinite while loop that does two things.
Pulls in all the messages from the "toSend" queue and sends them.
Adds a message to the "received" queue (that was fetched).
The queues are currently ArrayBlockingQueue<String>(1000)
My question is the following:
Is this towards a good design of a network thread? Are there any gotchas I need to handle?
Currently I've had an issue where one of the threads (the main thread) is filling the "toSend" queue even before the network thread has a chance to send even one message. I've "handled" this by making the main thread do some work that blocks (I/O).
It seems like you are re-inventing the wheel and are struggling with issues others have solved (and most probably with an optimized solution) before, please take a look at Netty:
The Netty project is an effort to provide an asynchronous event-driven network application framework and tools for rapid development of maintainable high performance & high scalability protocol servers & clients.
There is an example for an broadcast UDP/IP client and server.
Personally I think that handling UDP networking is best done using a single threaded, event driven programming style, rather than using threads. It is stateless anyhow, so it makes little sense trying to track the state in a thread.
Things like timeouts are easily handled with timers adding events to the event queue.
This effectively sidesteps all the synchronisation issues.
I have been working on UDP/TCP real time systems for more than 4 years now.
Your question has a lot of sense. In my humble opinion, the Netty framework is a good way to go but a bit heavy.
I am using the exact same design that you mentioned and have seen no issues with it so far.
Regards,
The 'toSend' queue is pointless. Just have everybody send directly via the socket.
Related
I am developing server-client communication based system and I am trying to determine the most optimal way to handle multiple clients. What is important I really don't want to use any third-party libraries.
In many places in the Internet I saw this resolved by creating a separate thread for each connections, but I don't think it is the best way when I assume there will be a huge number of connections (maybe I'm wrong). So, solution that I'm thinking of is
Creating queue of events and handling them by workers - the defined pool of threads (where there is a constant number n of workers). This solution seems to be pretty slow, but I can not imagine how big difference will be in case of handling huge amount of clients.
I've been thinking also about load-balancing via multiinstantiatig the server (on different physical machines) but it is only a nice add-on to any solution, not the solution itself.
I am aware that Java is not really async-friendly, but maybe I lack some knowledge and there is nice solution. I'll be grateful for any sugestions.
Additional info:
I assume really big number of connections
Every connection will last for a long time (days, maybe weeks)
Program will need to send some data to specified client quite frequently
Each client will send data to server about once a 3 seconds
To avoid discussion (as SO is not a place for them):
One client - one thread
Many clients - constant number of threads and events pool
Any async-like solution, that I'm not aware of
Anything else?
I'd suggest starting off with the simple architecture of one thread per connection. Modern JVMs on sufficiently sized systems can support thousands of threads. You might be pleasantly surprised at how well even this simple scheme works. If you need 300k connections, though, I doubt that one thread per connection will work. (But I've been wrong before.) You might have to fiddle with the thread stack size and OS resource limits.
A queueing system will help decouple the connections from the threads handling the work, but it will add to the amount of work done per message received from each client. This will also add to latency (but I'm not sure how important that is). If you have 300k connections, you'll probably want to have a pool of threads reading from the connections, and you'll also want to have more than one queue through which the work flows. If you have 300k clients sending data once every 3 seconds, that's 100k ops/sec, which is a lot of data to shove through a single queue. That might turn into a bottleneck.
Another approach probably worth investigating is to have a pool of worker threads, but instead of each worker reading data from a queue written by connection reader threads, have each worker handle a bunch of sockets directly. Use a NIO Selector to have each thread wait on multiple sockets. Say, have 100 threads each handling 3,000 sockets. Or perhaps have 1,000 threads each handling 300 sockets. This depends on the amount of data and the amount of work necessary to process each incoming message. You'll have to experiment. This will probably be considerably simpler than using asynchronous I/O.
Java 7 has true asynchronous IO under the NIO package I've heard. I don't know much about it other than its difficult to work with.
Basic IO in java is blocking. This means using a fixed number of threads to support many clients is likely not possible with basic IO as you could have all threads tied up in blocking calls reading from clients who aren't sending data.
I suggest you look in asynchronous IO with Grizzly/Netty, if you change your mind on 3rd party libraries.
If you haven't changed your mind, look into NIO yourself.
Is it better to kick off one thread that will handle one client and another for each other one that connects but tells them the server is busy, or should I just stick with a singlethreaded approach where the same thread accepts and processes the client so others can't connect to it? (if that's the case of course)
Edit: I should note that there won't be 239482340 people connecting to it. Generally only one person will be connecting to the server, but I want my app to deal with another person trying to connect without falling over.
Sticking to a single thread is better for the server since the resource consumption is very low. However, this is probably annoying for the client since it doesn't know if there is something wrong with the server or if it is just busy.
Having threads that tell additional users that the server is busy takes up more resources but is better for the client.
In your particular case, either approach should be fine. I guess it really depends on the clients and what they want... :-/
Two major strategies are commonly used to build this kind of systems:
A solution based on a multithread strategy, assigning a different thread or process to the each incoming request. This model is used in many commercial servers
The event-driven model, is based on the use of non-blocking I/O operations to respond simultaneously to a number of requests coming from different clients. This is a growing approach.
I am developing an application that entails a single server, and a large number of clients. I'm utilizing the Java socket programming APIs to accomplish this task. At the moment, I am considering restructuring the entire design of my application because I simply do not think that it is structured in the most efficient way, and would appreciate some guidance towards an optimal path.
Current Implementation
I have one ServerSocket located on port 5000, and the thread that contains the socket simply runs continuously and accepts any connection. It then starts up a new server thread (based on a synchronized table of available ports) that handles communication with that client, and then blocks for ServerSocket.accept() again.
The threads that are spawned from this main thread also contain a ServerSocket and are used as a means to handle multiple connections at once.
Now, the client thread simply connects to port 5000, receives the next available port as a reply, then disconnects from port 5000 (by calling Socket.close()), and reconnects to the port that the server said was available.
My Question
Is this the most optimal way (or better yet, is it even reasonable?) to handle multiple clients on a single server? Or should I simply open ServerSocket's on all available ports and just listen constantly? Perhaps something that I have not yet considered?
Addendum
I'm trying to think in terms of very large client-server applications such as MMORPGs or some chat application to get a feeling for my implementation's feasibility. For example, I try to ask myself: "Although this might work, would it be a good solution if this application had a large user-base?". That being said, it would be easier for me to understand the optimal nature of a solution if I could see how it would work on a large scale, with say, millions of users.
I don't understand why you would need to use a new ServerSocket each time the main one accepts a connection. Why don't you simply use the socket returned by accept() (as explained in the Java tutorial)?
Also, instead of starting a new thread for each client, you should use a thread pool. This would avoid constant creations of new threads, and would avoid starting too many threads and bring your server to its knees.
This architecture is not the best one to handle a huge number of users, though. Using asynchronous IO would probably be a better solution if you really need such a scalability, but I don't have much experience with that.
When thinking of server architecture, the first question is to estimate how much memory and precessing power is required for single connection. The second is the number of simultaneous connections. After multiplication, we can decide if single machine is sufficient or we need a cluster.
Then we decide if we can afford a thread (some 128..512 KBytes) for a connection. If we can, then classic one-thread per-connection is OK. If we cannot, then async architecture based on NIO or NIO2 is more suitable.
After the basic decisions are done, we can select appropriate libraries and frameworks. Doing everything from scratch is more interesting, but would take so much time that the result may be interesting to nobody at the moment it is achieved.
I agree with your following suggestion because the single server on port 5000 is a bottleneck:
Or should I simply open ServerSocket's on all available ports and just
listen constantly?
I prefer the pool of serversocket.
Use JMS (in my case its ActiveMq) achieve you target. You can have load balancing and fail over easily
I need some advice in building a Java server that handles multiple clients at the same time. The clients need to remain connected for fairly long periods of time. I'm currently using blocking IO and spawning a thread to read from each client that connects to the server, but this is obviously not scalable.
I've found a few options, including using Selector or Executor with fixed size thread pools. I am not too familiar with either one, so which would be the best solution here? Thanks!
It depends on your definition of scalable. The system you have described with a single thread per connection is scalable up to hundreds may be even a couple of thousand concurrent connections, it will hit a wall at some point.
Your question says that your clients connect and stay connected for an extended period of time, it would be possible to have a single IO thread to handle the reading and writing, but have the processing of the request dispatched to another thread using an Executor.
There are frameworks/servers that are already written to handle this sort of event driven design. Have a look at:
Netty recently used by twitter in there query server
Jetty (not to be confused with Netty) capable of NIO and very scalable, might be to HTTP focused
MINA
Grizzly
It's worth noting that the world is full of failed startups & software products that had really scalable architecture. Scaling is a nice problem to have, better to have the problem than not to have it and no customers.
using multiple threads is scalable. Apache for example does this, and some sites using it get many visitors. However, another approach would indeed be using selector, though I have no experience using it.
After all, this seems like a question, which religion is the best.
there's a lot of framework for this kind of job, examples
Netty
Apache MINA
Independently of scalability every server application has it's limits. By using blocking IO, one of your limits will be the number of threads that the VM can spawn because the approach you take is "one-thread-per-client". With NIO (of which Selector is one of the classes), the approach is "one-thread-per-request" which will run out of threads much latter.
Horizontal scalability ( http://en.wikipedia.org/wiki/Scalability#Scale_horizontally_vs._vertically ) of your app will not depend on either of these choices.
I can create multiple threads for supporting multi-client feature in socket programming; that's working fine. But if 10,000 clients want to be connected, my server cannot create so many threads.
How can I manage the threads so that I can listen to all these clients simultaneously?
Also, if in this case the server wants to send something to a particular client, then how is it possible?
You should investigate Java's NIO ("New I/O") library for non-blocking network programming. NIO was designed to solve precisely the server scalability problem you are facing!
Introductory article about NIO: Building Highly Scalable Servers with Java NIO
Excerpts from O'Reilly's Java NIO book
Highly scalable socket programming in Java requires the selectable channels provided in the "New I/O", or NIO packages. By using non-blocking IO, a single thread can service many sockets, tending only to those sockets that are ready.
One of the more scalable open-source NIO applications is the Grizzly component of the Glassfish application server. Jean-Francois Arcand has written a number of informative, in-depth blog posts about his work on the project, and covers many subtle pitfalls in writing this kind of software with NIO.
If the concept of non-blocking IO is new to you, using existing software like Grizzly, or at least using it as a starting point for your adaptation, might be very helpful.
The benefits of NIO are debatable. See Paul Tyma's blog entries here and here.
A thread-per-connection threading model (Blocking Socket I/O) will not scale too well. Here's an introduction to Java NIO which will allow you to use non-blocking socket calls in java:
http://today.java.net/cs/user/print/a/350
As the article states, there are plenty of frameworks available so you don't have to roll your own.
As previously mentioned, 10.000 clients is not easy. For java, NIO (possibly augmented with a separate threadpool to handle each request without blocking the NIO thread) is usual way to handle a large amount of clients.
As mentioned, depending on implementation, threads might actually scale, but it depends a lot on how much interaction there is between client connections. Massive threads are more likely to work if there is little synchronization between the threads.
That said, NIO is notoriously difficult to get 100% right the first time you implement it.
I'd recommend either trying out, or at least looking at the source for the Naga NIO lib at naga.googlecode.com. The codebase for the lib is small compared to most other NIO frameworks. You should be able to quickly implement a test to see if you can get 10.000 clients up and running.
(The Naga source also happens to be free to modify or copy without attributing the original author)
This is not a simple question, but for a very in depth (sorry, not in java though) answer see this: http://www.kegel.com/c10k.html
EDIT
Even with nio, this is still a difficult problem. 10000 connections is a tremendous resource burden on the machine, even if you are using non-blocking sockets. This is why large web sites have server farms and load balancers.
Why don't you process only a certain amount of requests at a time.
Let's say you want to process a maximum of 50 requests at a time (for not creating too many threads)
You create a threadpool of 50 threads.
You put all the requests in a Queue (accept connections, keep sockets open), and each thread, when it is done, gets the next request then process it.
This should scale more easily.
Also, if the need arise, it will be easier to do load balancing, since you could share your queues for multiple servers
Personally I would rather use create a custom I/O non blocking setup, for example using one thread to accept clients and using one other thread to process them (checking if any input is available and writing data to the output if necessary).
You'll have to figure out why your application is failing at 10,000 threads.
Is there a hard limit to the number of threads in the JVM or the OS? If so, can it be lifted?
Are you running out of memory? Try configuring a smaller stack size per thread, and/or add more memory to the server.
Something else? Fix it.
Only once you have determined the source of the problem will you be able to fix it. In theory 10,000 threads should be OK but at that level of concurrency it requires some extra tuning of the JVM and operating system if you want it to work out.
You can also consider NIO but I think it can work fine with threads as well.