we are using an Apache Sever as a front server an a Tomcat server for the backend. The frontend client is a java swing application. The protocol is hessian.
Sometimes we got a lot of small requests. When doing a "nestat -a" there are a lot of TIME_WAIT connections, which are blocking the server to open new connections. Only the connections to the tomcat seem to stay. The connections to the apache seem to be closed.
We are using a rewrite rule to forward the requests to the tomcat
RewriteEngine On
RewriteCond %{REQUEST_URI} .*\.servlet.*$
RewriteRule ^/(.*)$ http://localhost:8080/$1 [P]
Any ideas?
UPDATE:
Thanks for your advice,
but it still doesn't work. Every stream is closed and there are still these TIME_WAIT's:
if (conn != null) {
try {
IOUtils.closeQuietly(conn.getInputStream());
} catch (IOException e) {
// do nothing
}
try {
IOUtils.closeQuietly(conn.getOutputStream());
} catch (Exception ex) {
// do nothing
}
try {
IOUtils.closeQuietly(((HttpURLConnection) conn).getErrorStream());
} catch (Exception ex) {
// do nothing
}
}
if (conn instanceof HttpURLConnection) {
((HttpURLConnection) conn).disconnect();
}
Most likely you don't close the Input/Output Stream in the Swing application when making the request.
From here:
If the result is an InputStream, it is very important that the
InputStream.close() be put in an finally block, because Hessian will
not close the underlying HTTP stream until all the data is read and
the input stream is closed.
TIME_WAIT is a normal state of a recently closed connection, it will remain in it for a kernel-defined timeout. You did everything you can from the application's side.
If you have connections flicker too fast and experience a shortage of available ports, you could tweak your system's recycling of them. Huge amount of TIME_WAIT connections gives an overview for Linux as well as basic theory, Tuning Windows for TCP/IP performance lists corresponding Windows parameters with much breifer explanation.
It's not Tomcat, it's your operating system.
The OS is leaving connection in TIME_WAIT in order to avoid interference caused by too quick port reuse. Imagine packets from the old connection arriving late while a new connection is open in the same port.
Related
This question already has answers here:
Java socket API: How to tell if a connection has been closed?
(9 answers)
Closed 5 years ago.
When I'm using e.g. PuTTY and my connection gets lost (or when I do a manual ipconfig /release on Windows), it responds directly and notifies my connection was lost.
I want to create a Java program which monitors my Internet connection (to some reliable server), to log the date/times when my internet fails.
I tried use the Socket.isConnected() method but that will just forever return "true". How can I do this in Java?
Well, the best way to tell if your connection is interrupted is to try to read/write from the socket. If the operation fails, then you have lost your connection sometime.
So, all you need to do is to try reading at some interval, and if the read fails try reconnecting.
The important events for you will be when a read fails - you lost connection, and when a new socket is connected - you regained connection.
That way you can keep track of up time and down time.
Even though TCP/IP is "connection oriented" protocol, normally no data is sent over an idle connection. You can have a socket open for a year without a single bit sent over it by the IP stack. In order to notice that a connection is lost, you have to send some data on the application level.(*) You can try this out by unplugging the phone cable from your ADSL modem. All connections in your PC should stay up, unless the applications have some kind of application level keepalive mechanism.
So the only way to notice lost connection is to open TCP connection to some server and read some data from it. Maybe the most simple way could be to connect to some FTP server and fetch a small file - or directory listing - once in a while. I have never seen a generic server which was really meant to be used for this case, and owners of the FTP server may not like clients doing this.
(*) There is also a mechanism called TCP keepalive but in many OS's you have to activate it for all applications, and it is not really practical to use if you want to notice loss of connection quickly
If the client disconnects properly, a read() will return -1, readLine() returns null, readXXX() for any other X throws EOFException. The only reliable way to detect a lost TCP connection is to write to it. Eventually this will throw an IOException 'connection reset', but it takes at least two writes due to buffering.
Why not use the isReachable() method of the java.net.InetAddress class?
How this works is JVM implementation specific but:
A typical implementation will use ICMP ECHO REQUESTs if the privilege can be obtained, otherwise it will try to establish a TCP connection on port 7 (Echo) of the destination host.
If you want to keep a connection open continually so you can see when that fails you could connect to server running the ECHO protocol yourself rather than having isReachable() do it for you and read and write data and wait for it to fail.
You might want to try looking at the socket timeout interval. With a short timeout (I believe the default is 'infinite timeout') then you might be able to trap an exception or something when the host becomes unreachable.
Okay so I finally got it working with
try
{
Socket s = new Socket("stackoverflow.com",80);
DataOutputStream os = new DataOutputStream(s.getOutputStream());
DataInputStream is = new DataInputStream(s.getInputStream());
while (true)
{
os.writeBytes("GET /index.html HTTP/1.0\n\n");
is.available();
Thread.sleep(1000);
}
}
catch (IOException e)
{
System.out.println("connection probably lost");
e.printStackTrace();
}
Not as clean as I hoped but it's not working if I leave out the os.writeBytes().
You could ping a machine every number of seconds, and this would be pretty accurate. Be careful that you don't DOS it.
Another alternative would be run a small server on a remote machine and keep a connection to it.
Its probably simpler to connect to yahoo/google or somewhere like this.
URL yahoo = new URL("http://www.yahoo.com/");
URLConnection yc = yahoo.openConnection();
int dataLen = yc.getContentLength() ;
Neil
The isConnected()method inside Socket.java class is a little misleading. It does not tell you if the socket is currently connected to a remote host (like if it is unclosed). Instead, it tells you whether the socket has ever been connected to a remote host. If the socket was able to connect to the remote host at all, this method returns true, even after that socket has been closed. To tell if a socket is currently open, you need to check that isConnected() returns true and isClosed() returns false.
For example:
boolean connected = socket.isConnected() && !socket.isClosed();
I currently have a simple instant messaging program which is utilizing Java's Socket and ServerSocket classes. It is functioning as intended but when I attempt to close the connection it is not using the 4 way handshake TCP teardown to close the connection. Instead it is closing the connection abruptly with an RST packet.
The way in which I am closing the connection is sending a string from the client to the server which the server will recognize as the command to close the connection. I then use the ServerSocket.close() method on the server and the Socket.close() method on the client.
What is the correct way and/or order of events to properly close a TCP connection utilizing these classes?
Client side disconnect code:
//Disconnects from remote server
//Returns true on success, false on failure
public boolean disconnect(){
try{
this.clientOut.println("0x000000");
this.clientRemoteSocket.close();
this.isConnected = false;
return true;
}catch(Exception e){
return false;
}
}
Server side disconnect code:
//Check to see if the client wants to close the connection
//If yes, then close the connection and break out of the while loop
if(incoming.equals("0x000000")){
serverLocalSocket.close();
break;
}
EDIT:
The code works perfectly fine. I'm just trying to learn socket programming in Java and know that a proper TCP teardown process is to include a 4 way handshake. A FIN packet to the remote host, then an ACK packet from the remote host back. Then a FIN packet from the remote host, then an ACK packet to the remote host. When monitoring the traffic via Wireshark I am not getting that. Instead I am getting a FIN to the remote server, then a RST/ACK back from the server.
This image depicts a proper TCP 4 way teardown process.
So far everything I've found suggest that all one needs is a call to close() or to just let Java's Try-with-resources statement handle the clean up. I can't see Java implementing functionality which does not comply with the standard TCP specifications though. It is very possible I may be calling certain lines in an incorrect order or something of the sort, I'm just unaware of it.
If you are resetting your own connection on close, either:
You haven't read all the pending incoming data that was sent by the peer, or
You had already written to the connection which had previously already been closed by the peer.
In both cases, an application protocol error.
The great part about TCP is if you close your socket, your partner will automatically know and throw an error on reading.
So all you have to do in the client is:
clientRemoteSocket.close();
And with the server, just add an error case to your normal reading of data:
try {
// Read from the socket:
incoming = socketInputStream.read();
// Handle the data here
} catch (IOException e) {
// Client has disconnected
}
There might be a more specfic exception you can catch, I'm not sure, it's been a while. But that should work. Good luck!
I'm trying to see what IP's on a list are actually online. I'm currently using
public boolean IsOnline(String ip) {
try (Socket s = new Socket(ip, 80)) {
return true;
} catch (IOException ex) {
}
return false;
}
To check if the server is online. But it takes about 30 seconds for each IP to get checked. Is there a way to get faster results like another kind of method?
Not sure, but would INetAddress.isReachable suffice for your needs? You can't do ICMP messages in Java.
You may use the INetAddress.isReachable
".. A typical implementation will use ICMP ECHO REQUESTs if the
privilege can be obtained, otherwise it will try to establish a TCP
connection on port 7 (Echo) of the destination host..".
The only reason I can imagine, why this takes 30 seconds is, that nobody listens to port 80 and your socket connection timeout is 30 seconds. Please check for this by adding ex.printStackTrace() within the catch clause.
Please take a read here: Why are empty catch blocks a bad idea?
How you should "ping" a server, depends on what ports it is listening to. Normally INetAdresse.isReachable() works good. If your administrator blocks ICMP requests you may use the ssh port 22, to probe for the servers existence.
I've created an MMO for the Android phone and use a Java server with TCP/IP sockets. Everything generally works fine, but after about a day of clients logging on and off my network becomes extremely laggy -- even if there aren't clients connected. NETSTAT shows no lingering connections, but there is obviously something terribly wrong going on.
If I do a full reboot everything magically is fine again, but this isn't a tenable solution for the long-term. This is what my disconnect method looks like (on both ends):
public final void disconnect()
{
Alive = false;
Log.write("Disconnecting " + _socket.getRemoteSocketAddress());
try
{
_socket.shutdownInput();
}
catch (final Exception e)
{
Log.write(e);
}
try
{
_socket.shutdownOutput();
}
catch (final Exception e)
{
Log.write(e);
}
try
{
_input.close();
}
catch (final Exception e)
{
Log.write(e);
}
try
{
_output.close();
}
catch (final Exception e)
{
Log.write(e);
}
try
{
_socket.close();
}
catch (final Exception e)
{
Log.write(e);
}
}
_input and _output are BufferedInputStream and BufferedOutputStream spawned from the socket. According to documentation calling shutdownInput() and shutdownOutput() shouldn't be necessary, but I'm throwing everything I possibly can at this.
I instantiate the sockets with default settings -- I'm not touching soLinger, KeepAlive, noDelay or anything like that. I do not have any timeouts set on send/receive. I've tried using WireShark but it reveals nothing unusual, just like NETSTAT.
I'm pretty desperate for answers on this. I've put a lot of effort into this project and am frustrated with what appears to be a serious hidden flaw in Java's default TCP implementation.
Get rid of shutdownInput() and shutdownOutput() and all the closes except the close for the BufferedOutputStream, and a subsequent close on the socket itself in a finally block as a belt & braces. You are shutting down and closing everything else before the output stream, which prevents it from flushing. Closing the output stream flushes it and closes the socket. That's all you need.
OP here, unable to comment on original post.
Restarting the server process does not appear to resolve the issue. The network remains very "laggy" even several minutes after shutting down the server entirely.
By "laggy" I mean the connection becomes extremely slow with both up and down traffic. Trying to load websites, or upload to my FTP, is painfully slow like I'm on a 14.4k modem (I'm on a 15mbs fiber). Internet Speed Tests don't even work when it is in this state -- I get an error about not finding the file, when the websites eventually load up.
All of this instantly clears up after a reboot, and only after a reboot.
I modified my disconnect method as EJP suggested, but the problem persists.
Server runs on a Windows 7 installation, latest version of Java / Java SDK. The server has 16gb of RAM, although it's possible I'm not allocating it properly for the JVM to use fully. No stray threads or processes appear to be present. I'll see what JVISUALVM says. – jysend 13 mins ago
Nothing unusual in JVISUALVM -- 10mb heap, 50% CPU use, 3160 objects (expected), 27 live threads out of 437 started. Server has been running for about 18 hours; loading up CNN's front page takes about a minute, and the normal speed test I use (first hit googling Speed Test) won't even load the page. NETSTAT shows no lingering connections. Ran all up to date antivirus. Server has run 24/7 in the past without any issues -- it is only when I started running this Java server on it that this started to happen.
This question already has answers here:
Java socket API: How to tell if a connection has been closed?
(9 answers)
Closed 5 years ago.
When I'm using e.g. PuTTY and my connection gets lost (or when I do a manual ipconfig /release on Windows), it responds directly and notifies my connection was lost.
I want to create a Java program which monitors my Internet connection (to some reliable server), to log the date/times when my internet fails.
I tried use the Socket.isConnected() method but that will just forever return "true". How can I do this in Java?
Well, the best way to tell if your connection is interrupted is to try to read/write from the socket. If the operation fails, then you have lost your connection sometime.
So, all you need to do is to try reading at some interval, and if the read fails try reconnecting.
The important events for you will be when a read fails - you lost connection, and when a new socket is connected - you regained connection.
That way you can keep track of up time and down time.
Even though TCP/IP is "connection oriented" protocol, normally no data is sent over an idle connection. You can have a socket open for a year without a single bit sent over it by the IP stack. In order to notice that a connection is lost, you have to send some data on the application level.(*) You can try this out by unplugging the phone cable from your ADSL modem. All connections in your PC should stay up, unless the applications have some kind of application level keepalive mechanism.
So the only way to notice lost connection is to open TCP connection to some server and read some data from it. Maybe the most simple way could be to connect to some FTP server and fetch a small file - or directory listing - once in a while. I have never seen a generic server which was really meant to be used for this case, and owners of the FTP server may not like clients doing this.
(*) There is also a mechanism called TCP keepalive but in many OS's you have to activate it for all applications, and it is not really practical to use if you want to notice loss of connection quickly
If the client disconnects properly, a read() will return -1, readLine() returns null, readXXX() for any other X throws EOFException. The only reliable way to detect a lost TCP connection is to write to it. Eventually this will throw an IOException 'connection reset', but it takes at least two writes due to buffering.
Why not use the isReachable() method of the java.net.InetAddress class?
How this works is JVM implementation specific but:
A typical implementation will use ICMP ECHO REQUESTs if the privilege can be obtained, otherwise it will try to establish a TCP connection on port 7 (Echo) of the destination host.
If you want to keep a connection open continually so you can see when that fails you could connect to server running the ECHO protocol yourself rather than having isReachable() do it for you and read and write data and wait for it to fail.
You might want to try looking at the socket timeout interval. With a short timeout (I believe the default is 'infinite timeout') then you might be able to trap an exception or something when the host becomes unreachable.
Okay so I finally got it working with
try
{
Socket s = new Socket("stackoverflow.com",80);
DataOutputStream os = new DataOutputStream(s.getOutputStream());
DataInputStream is = new DataInputStream(s.getInputStream());
while (true)
{
os.writeBytes("GET /index.html HTTP/1.0\n\n");
is.available();
Thread.sleep(1000);
}
}
catch (IOException e)
{
System.out.println("connection probably lost");
e.printStackTrace();
}
Not as clean as I hoped but it's not working if I leave out the os.writeBytes().
You could ping a machine every number of seconds, and this would be pretty accurate. Be careful that you don't DOS it.
Another alternative would be run a small server on a remote machine and keep a connection to it.
Its probably simpler to connect to yahoo/google or somewhere like this.
URL yahoo = new URL("http://www.yahoo.com/");
URLConnection yc = yahoo.openConnection();
int dataLen = yc.getContentLength() ;
Neil
The isConnected()method inside Socket.java class is a little misleading. It does not tell you if the socket is currently connected to a remote host (like if it is unclosed). Instead, it tells you whether the socket has ever been connected to a remote host. If the socket was able to connect to the remote host at all, this method returns true, even after that socket has been closed. To tell if a socket is currently open, you need to check that isConnected() returns true and isClosed() returns false.
For example:
boolean connected = socket.isConnected() && !socket.isClosed();