java tcp server to many connections - java

I wrote a simple TCP server to transfare some user Data to it and save it in an simple MySQL table. If i now run more than 2000 clients after each other it stops working. While running i get some IO error java.io.EOFException you may also see the misstake i made for that. But the most importand is that i get this
IO error java.net.SocketException: Connection reset
Exception in thread "main" java.lang.OutOfMemoryError: unable to create new native thread
at java.lang.Thread.start0(Native Method)
at java.lang.Thread.start(Unknown Source)
at Server.main(Server.java:49)
Enough Memory schould be there but the threads are still running and i dont see where i made the misstake that they dont get terminated. So i got up to 3900 threads running than.
So here is the part of the Server:
try {
// create new socket
ServerSocket sock = new ServerSocket(port);
textArea.setText(textArea.getText() + "Server started\n");
while (true) {
// accept the connection
Socket newsock = sock.accept();
// handle the action
Thread t = new ThreadHandler(newsock);
newsock.setSoTimeout(2000); // adding client timeout
t.start();
}
} catch (Exception e) {
guess really simple. Here is how i handle the socket:
class ThreadHandler extends Thread {
private Socket socket;
private MySQLConnection sqlConnection;
ThreadHandler(Socket s) {
socket = s;
sqlConnection = new MySQLConnection();
}
public void run() {
try {
DataOutputStream out = new DataOutputStream(
socket.getOutputStream());
DataInputStream in = new DataInputStream(new BufferedInputStream(
socket.getInputStream()));
Server.textArea.append((new Date()) + "\nClient connected IP: " + socket.getInetAddress().toString()+"\n");
int firstLine = in.readInt(); // get first line for switch
switch (firstLine) {
case 0:
// getting the whole objekt for the database in own lines!
String name2 = in.readUTF();
int level2 = in.readInt();
int kp2 = in.readInt();
String skill = in.readUTF();
LeadboardElement element2 = new LeadboardElement();
element2.setName(name2);
element2.setLevel(level2);
element2.setKillPoints(kp2);
element2.setSkill(skill);
sqlConnection.saveChaToLeadboard(element2);
break;
//case 1 return the top10
###.... shorten here the rest of the cases
out.close();
in.close();
//close this socket
socket.close();
Server.textArea.append("Client disconnected IP: " + socket.getInetAddress().toString()+ "\n" + (new Date())
+ "\n----------------------------------------------------\n");
// autoscrolldown
Server.textArea.setCaretPosition(Server.textArea.getDocument()
.getLength());
} catch (Exception e) {
System.out.println("IO error " + e);
try {
socket.close();
} catch (IOException e1) {
e1.printStackTrace();
}
}finally{
try {
socket.close();
} catch (IOException e) {
e.printStackTrace();
}
}
the saveChaToLeadboard simply gets the name level kp and skill and uses a preparedStatement so save it to my MySQL Table.
I hope you can help me i just dont see the misstake of it. I think i need to Join it somewhere but if i put a join at the end of it (after socket.close()) it still does the same.
Here the save to database methode:
public void saveChaToLeadboard(LeadboardElement element) {
try {
// load driver
Class.forName("com.mysql.jdbc.Driver");
connection = DriverManager.getConnection(this.databaseURL
+ DATABASE_NAME, this.user, this.password);
// insert values into the prep statement
preparedStatement = connection
.prepareStatement(PREP_INSERT_STATEMENT);
preparedStatement.setString(1, element.getName());
preparedStatement.setInt(2, element.getLevel());
preparedStatement.setInt(3, element.getKillPoints());
if(!element.getSkill().equalsIgnoreCase("")){
preparedStatement.setString(4, element.getSkill());
}else{
preparedStatement.setString(4, null);
}
// execute
preparedStatement.executeUpdate();
connection.close();
} catch (Exception e) {
Server.textArea.append(e.getMessage() + "\n");
Server.textArea.setCaretPosition(Server.textArea.getDocument()
.getLength());
try {
connection.close();
} catch (SQLException e1) {
e1.printStackTrace();
}
}
Thanks alot!
Regards

Your run() method is mangled, but I suspect that part of the problem is that you are not always closing network sockets and streams. In particular, I suspect that you are not closing them if there is an exception while reading, or processing the data you read. You should always close sockets and streams in a finally block (or the Java 7 equivalent).
Another potential problem is that some of the connections may be stalling due to the other end not sending data. To deal with that, you would need to set a read timeout on the socket ... so that connections to slow / stuck clients can be closed.
Finally, it is probably unrealistic to even try to process 2000+ connections in parallel with a thread per connection. That's a LOT of resources1. I recommend you use a thread pool with a fixed upper limit in the low hundreds, and stop accepting new connections if all threads are in use.
1 - Each thread stack occupies at least 64K of memory on a HotSpot JVM, and possibly as much of 1Mb. Then there are the Heap resources that the thread directly or indirectly refers to, and OS resources needed to maintain the state of the threads and the sockets. For 2000 threads, that's probably multiple Gb of memory.

IMHO 2000 threads is on the high side for a single process and 2000 database connections definately is.
Regardless of whether or not you're hitting limits with 2000 incoming connections, your approach simply will not scale.
To acheive scalability you need to look at using resource pools - this means:
a pool of reader threads reading from the sockets queuing the data for processing.
a pool of worker threads processing the data queued by the reader threads.
a pool of database connections used by the worker threads - this connection pool could be adjusted so that each worker thread has it's own connection but the important thing is that you don't continually open and close database connections.
Look at the concurreny API for the thread pools and the NIO API for the IO.
This arrangement will allow you to tune your server to acheive the desired throughput.

Related

How to keep Java Client Socket alive when network connection drops?

I often have to restart my client application when my internet connection goes down for a few seconds. Do I have to create new connection on every disconnect or is there a way to keep client socket alive while "waiting?" for connection to be reestablished? I am not sure when exactly the disconnect happens. Is it when we try to write back to the outputstream as readLine() might be able to survive the disconnect? Or is the "&& !kkSocket.isClosed()" check redundant and closes the while loop? Thanks in advance.
try {
kkSocket = new Socket("12.345.67.899", 1234);
out = new PrintWriter(kkSocket.getOutputStream(), true);
} catch (IOException e) {
e.printStackTrace();
return;
}
try (
BufferedReader in = new BufferedReader(
new InputStreamReader(kkSocket.getInputStream()));
) {
String fromServer;
while ((fromServer = in.readLine()) != null && !kkSocket.isClosed()) {
doSomething(fromServer);
out.println("Back to server");
}
} catch (IOException e) {
Thread.currentThread().interrupt();
} catch (InterruptedException e) {
e.printStackTrace();
}
It is not possible to make a client socket "wait" for the connection to be reestablished. What you can do instead is creating a loop that will retry to connect to the server upon disconnect, for example based on a timer (every X seconds). This way you avoid having to manually restart your application.
Note that you would have to implement some way of exiting this retry loop to exit your program. (ex: throw a keyboard event, exit after X consecutive unsuccessful attempts etc...)

Java Server having many clients connect without bottlenecking

So what I'm trying to do is have a socket that receives input from the client, put the client into the queue and then return a message to each client in the queue when my algorithm returns true.
This queue should support a few hundred clients at once but at the same time not bottle neck the server so it can actually do what its supposed to do.
This is what i have so far:
private static final int PORT = 25566;
private static final int THREADS = 4;
private ExecutorService service;
public void init() throws IOException, IllegalStateException {
ServerSocket serverSocket;
serverSocket = new ServerSocket(PORT);
service = Executors.newCachedThreadPool();
Socket socket;
while(true) {
socket = serverSocket.accept();
System.out.println
("Connection established with " + socket.getInetAddress().toString());
service.execute(() -> {
Scanner scanner = null;
PrintWriter output = null;
String line = null;
try {
scanner = new Scanner(new InputStreamReader(socket.getInputStream()));
output = new PrintWriter(socket.getOutputStream());
} catch(IOException e) {
e.printStackTrace();
}
try {
if (scanner == null || output == null)
throw new IllegalStateException("Scanner/PrintWriter is " + "null!");
line = scanner.nextLine();
while (line.compareTo("QUIT") != 0) {
/* This is where input comes in, queue for the algorithm,
algorithm happens then returns appropriate values */
output.flush();
line = scanner.nextLine();
}
} finally {
try {
System.out.println
("Closing connection with " + socket.getInetAddress().toString());
if(scanner != null) {
scanner.close();
}
if(output != null) {
output.close();
}
socket.close();
} catch(IOException e) {
e.printStackTrace();
}
}
});
}
}
Now what I think will happen with this, is if the queues do reach high enough levels, my thread pool will completely bottleneck the server as all of the threads are being put to use on handling the clients in the queue and there won't be enough processing for the algorithm.
EDIT: After a bunch of testing, I think it will work out if in the algorithm it returns the value then disconnects, not waiting for user response but having the users client reconnect after certain conditions are met.
Your bottleneck is unlikely to be processing power unless you are machine limited. What's more likely to happen is that all the threads in your thread pool are consumed and end up waiting on input from the clients. Your design can only handle as many clients at once as there are threads in the pool.
For a few hundred clients, you could consider simply creating a thread for each client. The limiting resource for the number of threads that can be supported is typically memory for the stack that each thread requires, not processing power; for a modern machine with ample memory, a thousand threads is not a problem, based on personal experience. There may be an operating system parameter limiting the number of threads which you may have to adjust.
If you need to handle a very large number of clients, you can set up your code to poll sockets for available input and do the processing only for those sockets that have input to be processed.

Sampler gives High CPU load on socket.receive(packet)

I'm, trying to minimize the footprint of a Java program.
The JVisualVM Sampler tells me that the socket.receive() takes a lot of CPU time, even if no data is coming in.
The socket is blocking and not connected to any other machine.
Is this just a problem of the sampler or a problem of the JVM or my Code?
// constructor
if (this.isRunning && !this.socket.isClosed())
return;
this.socket = new DatagramSocket(port);
this.isRunning = true;
super.start();
...
// receive loop
while (this.isRunning && !thread.isInterrupted()) {
try {
byte[] buf = new byte[256];
// receive request
DatagramPacket packet = new DatagramPacket(buf, buf.length);
this.socket.receive(packet);
if (isRunning) {
tokenize(packet, true);
}
} catch (SocketException se) {
LOG.warn("Socket closed from outside");
} catch (IOException e) {
e.printStackTrace();
this.isRunning = false;
}
}
Receive( ) is a blocking operation so the thread that executes it, will be blocked.
JVisualVM is saying that the thread is spending much time there (which is normal as nothing will be done until a new connection arrives).
You can also check the thread monitor to see the % time that the thread is blocked (i.e. in a I/O operation).
Ps. You might want to create worker threads in order to process the received packets. Otherwise your service will be able to handle just one request at a time.

Thread sleep in connection

I have a network client which tries 3 times in a loop to connect to server. During this time I use sleep thread. Is there any way to replace Thread.sleep(700); with some code which skip the waiting period right after the client is connected.
NClient pc;
if (pc == null)
{
try
{
Thread.sleep(700);
}
catch (InterruptedException x)
{
//TODO
}
if (pc != null)
{
outPrint.println("Connected");
break;
}
}
I would like to improve the user experience by reducing the waiting period in which the connection negotiation is in progress. What are the options in Java to do this?
The answer to this question depends on the implementation of NClient. Typically, I'd use a connect timeout for this. The example below indicates how to do this with a Socket. I don't know what NClient is, so I can't give you an NClient example unfortunately.
Create a method that attempts to connect - up to 3 times
Socket connectToServer() {
Socket socket = new Socket();
final int connectTimeoutMs = 700;
for (int i=0; i<3; i++) {
try {
// the call to connect blocks the current thread for a maximum of 700ms if it can't connect
socket.connect(new InetSocketAddress("localhost", 8080), connectTimeoutMs);
} catch (IOException e) {
// failed to successfully connect within 700 milliseconds
e.printStackTrace();
}
}
return socket;
}
Use the above as follows
Socket socket = connectToServer();
if (socket.isConnected()) {
// do stuff with the valid socket!
}
In short, use a connect timeout!

How to add heartbeat messaging on top of this Java code( for KnockKnockClient/Server)?

I'm studying the following basic Java socket code( source ). It's a Knock-Knock-Joke client/server app.
In the Client, we set up the socket as usual:
try {
kkSocket = new Socket("localhost", 4444);
out = new PrintWriter(kkSocket.getOutputStream(), true);
in = new BufferedReader(new InputStreamReader(kkSocket.getInputStream()));
} catch( UnknownHostException uhe ){ /*...more error catching */
And then later, we just read and write to Server:
BufferedReader stdIn = new BufferedReader(new InputStreamReader(System.in));
String fromServer;
String fromUser;
while ((fromServer = in.readLine()) != null) {
System.out.println("Server: " + fromServer);
if (fromServer.equals("bye."))
break;
fromUser = stdIn.readLine();
if (fromUser != null){
System.out.println("Client: " + fromUser);
out.println(fromUser);
}
And on the server, we have the corresponding code, to get the joke punch-line.
KnockKnockProtocol kkp = new KnockKnockProtocol();
outputLine = kkp.processInput(null);
out.println(outputLine);
while ((inputLine = in.readLine()) != null) {
outputLine = kkp.processInput(inputLine);
out.println(outputLine);
if (outputLine.equals("Bye."))
break;
I want to attach a heartbeat to the whole thing, which will print out to the console whenever it detects that the other side died. Because what happens now if I kill the other side is an exception - like this one below:
So if I am running both KnockKnockClient and KnockKnockServer, then I shut down KnockKnockServer, what should happen is that on the Client I see this outputted:
>The system has detected that KnockKnockServer was aborted
I'm looking for any tips. So far I've mainly been trying to run a daemon thread that periodially creates new connections to the other side. But I'm confused about what condition to check for(but I think it's just a boolean value?). Is that the right approach? I just found out online there's a library called JGroups for multicast networking - would that be a better way? I'm looking for any tips.
My server-code so far(sorry it's messy)
&
Client-side
thanks
But the exception you are getting is exactly this! It's telling you that the other side just died. Just catch the exception and print to the console, that "The system has detected that KnockKnockServer was aborted".
You are using TCP connection and TCP has built-in heartbeat (keepalive) mechanism that will do this for you. Just set setKeepAlive() on the socket. That being said - It is possible to control keepalive frequency per each connection, but I do not know how to do that in java.
http://tldp.org/HOWTO/TCP-Keepalive-HOWTO/overview.html
https://stackoverflow.com/a/1480259/706650
you have a Synchronous communication. for having the heartbeat message, use an asynchronous communication. there will be 2 threads. one will read from the socket and another will keep writing to the socket. If you use asynchronous communication, the server will be sending a message every 10 seconds. the client thread will be reading messages from the server and if there is no message, it means the server is down. in your case, the server either sends back the message to client(if client has some message) or send an automatic reply.your server code can be modified like this.
Create a server thread that will keep sending messages to client every 10 seconds.
public class receiver extends Thread{
public static bool hearbeatmessage=true;
Socket clientSocket=new Socket();
PrintWriter out=new PrintWriter();
public receiver(Socket clientsocket){
clientSocket=clientsocket;
out = new PrintWriter(clientSocket.getOutputStream(), true);
}
public void run(){
while(true)
{
if(heartbeatmessage){
thread.sleep(10000);
out.println("heartbeat");
}
}
}
}
In your server code:
KnockKnockProtocol kkp = new KnockKnockProtocol();
outputLine = kkp.processInput(null);
out.println(outputLine);
receiver r=new reciver(clientSocket);
r.run(); /*it will start sending hearbeat messages to clients */
while ((inputLine = in.readLine()) != null) {
outputLine = kkp.processInput(inputLine);
reciver.hearbeatMessage=false; /* since you are going to send a message to client now, sending the heartbeat message is not necessary */
out.println(outputLine);
reciver.hearbeatMessage=true; /*start the loop again*/
if (outputLine.equals("Bye."))
break;
The client code will also be modified, a thread will keep reading messages from the socket and if it has not received message for more than 11 seconds(1 second extra), it will declare the server is not available.
Hope this helps. There might be some flaw in the logic too. Let me know.
The following are best practices which we apply on a daily base when interfacing with hardware (using sockets).
Good practice 1 : SoTimeout
This property enables a read timeout. The goal of this is to avoid the issue that Tom had. He wrote something in the line of : "you will need to wait till the next client message arrives". Well, this offers a solution to that problem. And it's also the key to implementing a heartbeat and many other checks.
By default, the InputStream#read() method will wait forever, until a message arrives. The setSoTimeout(int timeout) changes this behaviour. It will apply a timeout now. When it timeouts it will throw the SocketTimeoutException. Just catch the exception, check a couple of things and continue reading (repeat). So basically, you put your reading method in a loop (and probably even in a dedicated thread).
// example: wait for 200 ms
connection.setSoTimeout(200);
You can use these interruptions (caused by the timeout) to validate the status: E.g. how long has it been since I received my last message.
Here is an example to implement the loop:
while (active)
{
try
{
// some function that parses the message
// this method uses the InputStream#read() method internally.
code = readData();
if (code == null) continue;
lastRead = System.currentTimeMillis();
// the heartbeat message itself should be ignored, has no functional meaning.
if (MSG_HEARTBEAT.equals(code)) continue;
//TODO FORWARD MESSAGE TO ACTION LISTENERS
}
catch (SocketTimeoutException ste)
{
// in a typical situation the soTimeout should be about 200ms
// the heartbeat interval is usually a couple of seconds.
// and the heartbeat timeout interval a couple of seconds more.
if ((heartbeatTimeoutInterval > 0) &&
((System.currentTimeMillis() - lastRead) > heartbeatTimeoutInterval))
{
// no reply to heartbeat received.
// end the loop and perform a reconnect.
break;
}
// simple read timeout
}
}
Another use of this timeout: It can be used to cleanly stop your session by setting active = false. Use the timeout to check if this field is true. If that's the case, then break the loop. Without the SoTimeout logic this would not be possible. You would either be forced to do a socket.close() or to wait for the next client message (which clearly makes no sense).
Good practice 2 : Built-in Keep-Alive
connection.setKeepAlive(true);
Well basically this is pretty much what your heart-beat logic does. It automatically sends a signal after a period of inactivity and checks for a reply. The keep-alive interval is operating system dependent though, and has some shortcomings.
Good practice 3 : Tcp No-Delay
Use the following setting when you are often interfacing small commands that need to be handled quickly.
try
{
connection.setTcpNoDelay(true);
}
catch (SocketException e)
{
}
I think you are over complicating things.
From the client side:
If the client gets an IOException for the connection reset, then this means the server is dead. Instead of printing the stack trace just do what ever you need to do once you know that the server is down. You already know the server is down due to the exception.
From the server side:
Either start a timer and if you don't get a request for a time more than the interval assume that the client is down.
OR start a background server thread at the client (making the client and server peers) and have the server send a "dummy" hearbeat request (server now acts as a client). If you get exception the client is down.
Figured I'd take a crack at this... I started with the KnockKnockServer and KnockKnockClient that are on the Java site (in your original question).
I didn't add any threading, or heartbeats; I simply changed the KnockKnockClient to the following:
try { // added try-catch-finally block
while ((fromServer = in.readLine()) != null) {
System.out.println("Server: " + fromServer);
if (fromServer.equals("Bye."))
break;
fromUser = stdIn.readLine();
if (fromUser != null) {
System.out.println("Client: " + fromUser);
out.println(fromUser);
}
}
} catch (java.net.SocketException e) { // catch java.net.SocketException
// print the message you were looking for
System.out.println("The system has detected that KnockKnockServer was aborted");
} finally {
// this code will be executed if a different exception is thrown,
// or if everything goes as planned (ensure no resource leaks)
out.close();
in.close();
stdIn.close();
kkSocket.close();
}
This seems to do what you want (even though I modified the original Java website example, rather than your code - hopefully you'll be able to see where it plugs in). I tested it with the case you described (shut down the server while the client is connected).
The downside to this is that, while the client is waiting for user input, you don't see that the server has died; you have to enter client input, and then you'll see that the server has died. If this is not the behavior you want, please post a comment (perhaps that was the whole point of the question - it just seemed like you might have been going down a longer road than you needed in order to get to where you wanted to be).
Here's a slight modification to the client. It doesn't use an explicit heartbeat, but as long as you keep reading from the server, you'll immediately detect the disconnect anyway.
This is because readLine will immediately detect any read errors.
// I'm using an anonymous class here, so we need
// to have the reader final.
final BufferedReader reader = in;
// Decouple reads from user input using a separate thread:
new Thread()
{
public void run()
{
try
{
String fromServer;
while ((fromServer = reader.readLine()) != null)
{
System.out.println("Server: " + fromServer);
if (fromServer.equals("Bye."))
{
System.exit(0);
}
}
}
catch (IOException e) {}
// When we get an exception or readLine returns null,
// that will be because the server disconnected or
// because we did. The line-break makes output look better if we
// were in the middle of writing something.
System.out.println("\nServer disconnected.");
System.exit(0);
}
}.start();
// Now we can just read from user input and send to server independently:
while (true)
{
String fromUser = stdIn.readLine();
if (fromUser != null)
{
System.out.println("Client: " + fromUser);
out.println(fromUser);
}
}
In this case, we allow client writes even when we're waiting for reply from the server. For a more stable application, we'd want to lock the input while we're waiting for a reply by adding a semaphore controlling when we start reading.
These are the modifications we would make to control the input:
final BufferedReader reader = in;
// Set up a shared semaphore to control client input.
final Semaphore semaphore = new Semaphore(1);
// Remove the first permit.
semaphore.acquireUninterruptibly();
new Thread()
... code omitted ...
System.out.println("Server: " + fromServer);
// Release the current permit.
semaphore.release();
if (fromServer.equals("Bye."))
... code omitted ...
while (true)
{
semaphore.acquireUninterruptibly();
String fromUser = stdIn.readLine();
... rest of the code as in the original ...
I think #Bala's answer is correct on server side. I'd like to give a supplementary on client side.
On client side, you should:
use an variable to keep the timestamp of the last message from server;
start a thread which runs periodically(every 1 second, e.g.) to compare current timestamp and the last message timestamp, if it is longer than desired timeout(10 seconds, e.g.), a disconnection should be reported.
Following are some code snippet:
The TimeoutChecker class(thread):
static class TimeoutChecker implements Runnable {
// timeout is set to 10 seconds
final long timeout = TimeUnit.SECONDS.toMillis(10);
// note the use of volatile to make sure the update to this variable thread-safe
volatile long lastMessageTimestamp;
public TimeoutChecker(long ts) {
this.lastMessageTimestamp = ts;
}
#Override
public void run() {
if ((System.currentTimeMillis() - lastMessageTimestamp) > timeout) {
System.out.println("timeout!");
}
}
}
Start the TimeoutChecker after connection is established:
try {
kkSocket = new Socket("localhost", 4444);
// create TimeoutChecker with current timestamp.
TimeoutChecker checker = new TimeoutChecker(System.currentTimeMillis());
// schedule the task to run on every 1 second.
ses.scheduleAtFixedRate(, 1, 1,
TimeUnit.SECONDS);
out = new PrintWriter(kkSocket.getOutputStream(), true);
in = new BufferedReader(new InputStreamReader(kkSocket.getInputStream()));
} catch( UnknownHostException uhe ){ /*...more error catching */
The ses is a ScheduledExecutorService:
ScheduledExecutorService ses = Executors.newScheduledThreadPool(1);
And remember to update the timestamp when receiving messages from server:
BufferedReader stdIn = new BufferedReader(new InputStreamReader(System.in));
String fromServer;
String fromUser;
while ((fromServer = in.readLine()) != null) {
// update the message timestamp
checker.lastMessageTimestamp = System.currentTimeMillis();
System.out.println("Server: " + fromServer);
if (fromServer.equals("bye."))
break;
Adel,was looking at your code http://pastebin.com/53vYaECK
Can you try the following solution. not sure whether it will work.
instead of creating a bufferedreader with the inputstream once,
we can create an instance of BufferedReader eachtime.
when the kkSocket.getInputStream is null, it comes out of the while loop and set completeLoop to false, so that we exit the while loop.
it has 2 while loops and the objects are created each time.
if the connection is open but does not have data in it inputstream will not be null,
BufferedReader.readLine would be null.
bool completeLoop=true;
while(completeLoop) {
while((inputstream is=kkSocket.getInputStream())!=null) /*if this is null it means the socket is closed*/
{
BufferedReader in = new BufferedReader( new InputStreamReader(is));
while ((fromServer = in.readLine()) != null) {
System.out.println("Server: " + fromServer);
if (fromServer.equals("Bye."))
break;
fromUser = stdIn.readLine();
if (fromUser != null) {
System.out.println("Client: " + fromUser);
out.println(fromUser);
}
}
}
completeLoop=false;
System.out.println('The connection is closed');
}

Categories